Text reasoning method and device, electronic equipment, storage medium and program product

Abstract

The application discloses a text reasoning method and device, electronic equipment, storage medium and program product, relates to the technical field of language model reasoning, and comprises the following steps: determining a target calculation sub-tree matched with the original text semantics, mapping the model calculation process into a tree topology structure, and performing the tasks of leaf nodes, intermediate nodes and root nodes in parallel based on the layer number corresponding to the topology depth. The leaf nodes can simultaneously extract the hidden state of the original text, and the intermediate nodes can simultaneously carry out semantic aggregation on the output of different leaf nodes. The application introduces a historical high-frequency hidden state as a reference state. When the task execution unit of the intermediate node performs semantic aggregation, the hidden state output by the leaf node is aggregated with the reference state. The root node only needs to perform feature conversion on the aggregated state to obtain the core feature vector, thereby avoiding full transmission of the intermediate state between layers in the traditional parallel mode and reducing the redundancy of the intermediate state transmission.

Description

Technical Field

[0001] This application relates to the field of server hardware management technology, and in particular to a text reasoning method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, the number of parameters in Large Language Models (LLMs) has reached hundreds of billions, leading to a surge in demand for applications such as dialogue generation. Training and inference with hundreds of billions of LLMs presents enormous computational and storage pressures, making it difficult for a single device to handle. The industry has proposed three mainstream parallel technologies: data parallelism, static model parallelism, and pipelined model parallelism. Data parallelism replicates the complete LLM across multiple devices, dividing the data into different batches and synchronizing the results. Static model parallelism partitions the LLM across multiple devices by layer or tensor dimension, dividing the computation among them. Pipeline model parallelism combines the first two technologies, splitting the input into micro-batches to achieve pipelined overlapping computation, improving efficiency. However, static model parallelism suffers from poor adaptability due to its fixed partitioning granularity, requires strict synchronization between layers, and experiences linearly increasing latency as the number of nodes increases. Furthermore, the full transmission of intermediate hidden states results in excessive bandwidth consumption. Pipeline model parallelism, on the other hand, suffers from drawbacks such as the lack of optimization for LLM semantic features during intermediate state transmission and high communication pressure, making it unsuitable for applications with hundreds of billions of LLMs and requiring urgent improvement. Summary of the Invention

[0003] This application provides a text reasoning method, apparatus, electronic device, storage medium, and program product to at least solve the problems of strong inter-layer synchronization dependency, low expansion efficiency, and redundant intermediate state transmission in the related art of static model parallelism and pipelined model parallelism.

[0004] This application provides a text reasoning method, including: acquiring the original text and determining a target computation subtree that semantically matches the original text; determining the layer number of task execution units at each node based on the tree topology depth of the target computation subtree; inputting the original text into the target computation subtree and obtaining the core feature vector output by the target computation subtree; the target computation subtree is used to determine the hidden state of the original text based on the layer number of task execution units at each node, the task execution units at the leaf nodes perform semantic aggregation on the hidden state and the reference state to obtain the aggregated state, and the task execution units at the root node perform feature transformation on the aggregated state to obtain the core feature vector; wherein, the reference state is the historically frequently occurring hidden state; and converting the core feature vector into natural language text corresponding to the original text.

[0005] This application also provides a text reasoning apparatus, including: The acquisition module is used to acquire the original text and determine the target computation subtree that semantically matches the original text; The determination module is used to calculate the tree topology depth of the subtree based on the target and determine the layer of the task execution unit of each node; The inference module is used to input the original text into the target computation subtree and obtain the core feature vector output by the target computation subtree. The target computation subtree is used to determine the hidden state of the original text by the task execution unit of the leaf node based on the number of layers of the task execution unit of each node. The task execution unit of the intermediate node performs semantic aggregation on the hidden state and the reference state to obtain the aggregated state. The task execution unit of the root node performs feature transformation on the aggregated state to obtain the core feature vector. The reference state is the hidden state that appears frequently in history. The conversion module is used to convert the core feature vectors into the corresponding natural language text of the original text.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of the above-described text reasoning method when executing the computer program.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described text reasoning method.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described text reasoning method.

[0009] This application maps the model computation process into a tree-like topology by determining the target computation subtree that semantically matches the original text. The task execution units of leaf nodes, intermediate nodes, and root nodes can be executed in parallel based on the layer number corresponding to the topology depth, which can break the strict synchronization dependency between linear layers and reduce the impact of latency growth caused by node expansion. Leaf nodes can simultaneously extract the hidden state of the original text, and intermediate nodes can synchronously perform semantic aggregation on the outputs of different leaf nodes. By introducing historically frequently occurring hidden states as reference states, when the task execution units of intermediate nodes perform semantic aggregation, they aggregate the hidden states output by the leaf nodes with the reference states. The root node only needs to perform feature transformation on the aggregated states to obtain the core feature vector, which can replace the full transmission of traditional intermediate hidden states, reduce redundant communication data, alleviate bandwidth occupation pressure, and improve expansion efficiency and processing performance. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic diagram of the specific hardware architecture on which the execution of a text reasoning method provided in this application depends; Figure 2 A flowchart illustrating a text reasoning method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a text reasoning device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided as an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] To more clearly illustrate the embodiments of this application, the technical terms used in the embodiments will be briefly introduced below: Large Language Models (LLMs) are a class of deep learning models based on the Transformer architecture, trained on massive amounts of text data, and possessing a large parameter scale (typically ranging from hundreds of millions to trillions). Their core capabilities include understanding natural language semantics and generating grammatically and logically consistent text content. They also support diverse natural language processing tasks such as question answering, translation, summarization, and code writing, and can model and process complex language scenarios by learning the contextual relationships within language.

[0015] A Dynamic Partition Controller (DPC) is a control module used for managing computing or storage resources. Its core function is to dynamically adjust the size, number, and allocation strategy of resource partitions based on changes in real-time system load, task priority, and resource requirements. In large model training or inference scenarios, DPC can flexibly partition the computing nodes, memory space, network bandwidth, and other resources of a computing cluster to achieve an optimal balance between resource utilization and task execution efficiency.

[0016] Locality-Sensitive Hashing (LSH) is a hash algorithm that has the characteristic that "similar inputs are more likely to map to the same hash value," unlike traditional hashing which aims to "avoid collisions as much as possible." Its core principle is to construct a family of hash functions that satisfy locality sensitivity, mapping high-dimensional data to a low-dimensional hash space, such that data points that are closer together in the original space have a significantly higher probability of colliding in the hash space than data points that are farther apart.

[0017] The Tensor Processing Unit (TPU) is an application-specific integrated circuit (ASIC) proposed and developed by Google. It is specifically designed for hardware-level optimization of tensor operations (such as matrix multiplication and convolution) in deep learning tasks. The TPU adopts a highly parallel computing architecture and features high-bandwidth memory and low power consumption, enabling it to efficiently process large-scale tensor data during the training and inference of deep learning models.

[0018] A Graphics Processing Unit (GPU) is a processor specifically designed for graphics rendering and parallel computing. Originally designed to accelerate the generation and display of computer graphics, its architecture contains numerous lightweight computing cores capable of executing massive amounts of parallel tasks simultaneously. It has since been widely adopted in deep learning to accelerate parallel computational tasks such as matrix operations and convolution operations during model training and inference, significantly improving the efficiency of large-scale data processing through its high degree of parallelism.

[0019] A feedforward network (FFN) is an artificial neural network in which signals propagate unidirectionally from the input layer to the output layer, and there are no recurrent or feedback connections in its structure. Its basic components are an input layer, several hidden layers, and an output layer. Neurons in each layer are fully connected only to neurons in adjacent layers. Through a combination of linear transformations and nonlinear activation functions, it extracts and maps features from the input data. In the Transformer architecture, the feedforward network is the core component of the encoder and decoder, responsible for performing deep transformations on the features output by the attention mechanism.

[0020] Key-Value Cache (KV) is an optimization technique for the inference stage of large language models. Its core principle is to store the key and value tensors calculated during model generation in a high-speed cache. When generating subsequent tokens, the historical key-value data in the cache can be directly reused, eliminating the need to repeatedly calculate the key-value matrix operations in the attention mechanism. This technique effectively reduces the computational load and latency in scenarios involving the generation of large models and long texts, thereby improving inference efficiency.

[0021] Proximal Policy Optimization (PPO) is a reinforcement learning algorithm based on policy gradient, designed to address the problems of uncontrollable policy update step size and training instability in traditional policy gradient algorithms. Its core idea is to achieve larger policy update steps while ensuring monotonic policy improvement by limiting the difference between the new and old policies (i.e., "proximal constraint"). A pruning loss function is also introduced to prevent excessively large policy update magnitudes. PPO offers advantages such as stable training, high sample utilization, and strong hyperparameter robustness.

[0022] Remote Direct Memory Access (RDMA) is a technology that enables direct transfer of memory data between different nodes in a network without intervention from the host central processing unit (CPU). RDMA reduces CPU usage and data copying frequency by offloading data transfer tasks to dedicated network adapters, achieving low-latency, high-bandwidth inter-memory data exchange. This technology is a core network technology for high-performance computing clusters, distributed storage systems, and large model training clusters, supporting the efficient transmission of massive amounts of data between nodes.

[0023] A Multi-Layer Perceptron (MLP) is a feedforward neural network containing at least one hidden layer. It consists of an input layer, hidden layers, and an output layer, all fully connected. Its working principle is as follows: the input layer receives data, the hidden layers extract features using linear transformations and non-linear activation functions, and the output layer outputs the final prediction result. The MLP is one of the fundamental models in deep learning, used for supervised learning tasks such as classification and regression, and is also a core component of the feedforward network in the Transformer architecture.

[0024] Byte Pair Encoding (BPE) is a sub-word segmentation algorithm for text segmentation, with the core goal of achieving a balance between vocabulary size and segmentation granularity. Its basic process is as follows: First, a single character is used as the initial vocabulary. Then, the frequency of adjacent character pairs in the corpus is counted. The most frequent character pairs are merged into new sub-words and added to the vocabulary. This process is repeated until the preset vocabulary size or number of iterations is reached.

[0025] InfiniBand Architecture (InfiniBand) is a high-performance, low-latency interconnect technology standard designed for high-performance computing clusters and data center server clusters. This architecture employs a switch-based full-duplex communication mode, supports Remote Direct Memory Access (RDMA), multicast, and other functions, providing high-bandwidth, low-latency inter-node communication capabilities. It can meet the efficient transmission needs of massive amounts of data between nodes in large-scale distributed computing tasks (such as large model training). The rapid development of artificial intelligence technology has driven the number of parameters in Large Language Models (LLMs) from tens of billions to hundreds of billions, with broad application prospects in dialogue generation, reasoning, question answering, and other fields, leading to a surge in application demand. However, training and inference with models containing hundreds of billions of parameters faces enormous computational and storage pressures, which a single device cannot meet. Therefore, the industry has proposed three mainstream parallel processing technologies: data parallelism, static model parallelism, and pipelined model parallelism.

[0026] Data parallelism is based on the principle of replicating the complete model across multiple devices, allocating different batches of data, synchronizing gradients or results, and then aggregating the global output (e.g., GPT-3 is deployed on 8 GPUs, each processing 1 / 8 of the batch of data, and synchronizing results through the AllReduce protocol). Static model parallelism involves splitting the model by layer or tensor dimension and deploying it on multiple devices, with each device responsible for a portion of the layer computation, and transmitting intermediate results sequentially (e.g., GPT-3's 96-layer Transformer is split into 8 segments, which are processed sequentially by 8 GPUs). Pipeline model parallelism combines the advantages of the previous two approaches, splitting the data into micro-batches and achieving pipelined operations through overlapping computation (e.g., both the model and the micro-batches are split into 4 parts for parallel processing).

[0027] While these three technologies alleviate computational and storage pressures, they still have significant drawbacks: Asynchronous pipeline architectures require a single device to store the complete model, and the storage requirements for hundreds of billions of models far exceed the memory requirements of mainstream GPUs, resulting in high hardware costs. Furthermore, adding devices only increases throughput but does not reduce latency per request, making them unsuitable for real-time inference; Static model parallel partitioning has a fixed granularity, making it unable to adapt to changes in input length and fluctuations in hardware load. Strict inter-layer synchronization dependencies lead to low scalability, and full transmission of intermediate states consumes high bandwidth; Pipeline model parallelism has idle "bubbles" on devices, resulting in low resource utilization. The order of micro-batches cannot be disrupted, which can easily cause blocking, and full transmission of intermediate states leads to high communication pressure.

[0028] In summary, the existing technologies cannot simultaneously meet the requirements of processing efficiency, real-time performance, and other aspects, making it difficult to satisfy the high-efficiency and stable processing needs of practical applications involving large language models with hundreds of billions of entries. There is an urgent need for new parallel processing solutions.

[0029] To address all or part of the aforementioned technical problems, this application abandons the rigid execution logic of "layer-by-layer serial synchronization" in traditional parallel modes. By determining the target computation subtree that matches the semantics of the original text, the model computation process is mapped into a tree-like topology. The task execution units of leaf nodes, intermediate nodes, and root nodes can be executed in parallel based on the number of layers corresponding to the topology depth. Leaf nodes can simultaneously extract the hidden state of the original text, and intermediate nodes can synchronously perform semantic aggregation on the outputs of different leaf nodes. There is no need to wait for all lower-level nodes to complete the computation before starting the upper-level task, thus breaking the synchronization dependency bottleneck of "inter-layer blocking" in traditional parallel modes.

[0030] The number of layers of task execution units at each node is dynamically determined based on the tree topology depth of the target computation subtree, rather than using the "fixed layer partitioning" of static model parallelism or the "fixed stage partitioning" of pipeline model parallelism. When it is necessary to expand the model capacity, it can be achieved by increasing the number of nodes in the subtree or adjusting the topology depth without reconstructing the overall model architecture, and without extending the synchronization link due to the addition of new layers, thus effectively improving the expansion efficiency.

[0031] This application introduces historically frequently occurring hidden states as reference states. When the task execution unit of the intermediate node performs semantic aggregation, it does not need to transmit the hidden states of all leaf nodes. Instead, it compares and aggregates the hidden states output by the leaf nodes with the reference states. It only needs to transmit the aggregation information of the hidden states and the reference states. At the same time, the root node only needs to perform feature transformation on the aggregated states to obtain the core feature vector. This avoids the full transmission of intermediate states between layers in the traditional parallel mode and reduces the redundancy of intermediate state transmission.

[0032] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] The specific application environment architecture or specific hardware architecture on which the execution of the text reasoning method depends is described here.

[0034] like Figure 1 As shown, the specific hardware architecture on which the text reasoning method relies for execution consists of three main types of components: heterogeneous computing nodes, high-speed interconnection networks, and edge-cloud collaborative deployment units. Each component has a clear division of labor and works together to support the efficient operation of the entire reasoning process.

[0035] Heterogeneous computing nodes are the core computing carriers, comprising two key hardware types: GPUs and TPUs. GPUs primarily handle attention-intensive computation segments, where attention computation accounts for over 60%. Leveraging their thread-level parallelism, GPUs can simultaneously process computational tasks from multiple attention heads, reducing latency. For example, they can parallelly process query, key-value matrix operations from 16 attention heads, efficiently completing multi-head attention computation by concatenating the results and mapping them to the output space. TPUs, on the other hand, focus on FFN-intensive computation segments, where feedforward network computation accounts for over 70%. The TPU's Matrix Multiply Unit (MXU) can perform large-dimensional matrix multiplication operations in a single operation, achieving a matrix multiplication efficiency up to 1.8 times that of GPUs. It can quickly process linear transformations and activation operations in feedforward networks, adapting to the high-efficiency computational needs of high-dimensional features.

[0036] High-speed interconnection networks are crucial for ensuring distributed node collaboration, primarily built upon the RDMA protocol and the InfiniBand architecture. The RDMA protocol enables sub-second parameter migration. When the DPC dynamic partitioning strategy requires adjustments to the computation segment configuration, this protocol can rapidly migrate key parameters such as the weight matrix to the target node, with migration latency controlled within 100ms. This ensures timely computation segment reorganization and avoids slowing down inference due to parameter transmission. The InfiniBand architecture provides high-speed bandwidth support, reducing bandwidth waste during the transmission of intermediate hidden states between distributed nodes. It is particularly well-suited for the large data transmission requirements of long-sequence tasks, alleviating communication redundancy issues and providing stable network guarantees for parallel computing at the same level and cross-level feature aggregation.

[0037] The edge-cloud collaborative deployment unit achieves layered adaptation of computing resources, encompassing edge devices and cloud clusters. Edge devices (such as edge GPUs) deploy leaf node computing segments, responsible for the initial semantic extraction task. The core of the leaf node computing segment is converting input text into basic semantic features. Deploying it at the edge reduces the distance of raw data transmission to the cloud, lowers transmission latency, and adapts to the response speed requirements of real-time inference scenarios. The cloud cluster deploys root node computing segments, undertaking the final feature transformation task. The root node needs to integrate the aggregated features from various intermediate nodes and perform complex calculations. The abundant computing resources in the cloud ensure the accuracy and efficiency of this process, while providing unified scheduling and management support for heterogeneous computing nodes, achieving collaborative optimization between initial edge processing and core cloud computing.

[0038] Embodiments of this application provide a text reasoning method, such as... Figure 2 As shown, the method includes the following steps: S201. Obtain the original text and determine the target computation subtree that semantically matches the original text.

[0039] In some embodiments, the original text is segmented to generate a word sequence, the word sequence is embedded and encoded to obtain an embedding matrix, the embedding matrix is ​​then mapped to a semantic feature vector, and the cosine similarity between the semantic feature vector and the semantic center of each computational subtree is calculated. Each computational subtree is a substructure of the pre-trained language model after tree-like segmentation.

[0040] Optionally, the original text is received, segmented using Byte-Pair Encoding (BPE) to generate a word sequence, and then converted into an embedding matrix using embedding encoding. A semantic feature vector is obtained through Local Sensitive Hash (LSH) semantic mapping. The cosine similarity between the semantic feature vector and the semantic center of each computational subtree is calculated. Based on these cosine similarities, the target number of words to be computed in the original text is determined. The computational subtree with the highest similarity can be used as the target computational subtree.

[0041] For example, word segmentation is performed on the original text input x to obtain a word sequence. , denoted as the length of the word sequence; assuming the original text is "high-power CPU and GPU accompanied by heat dissipation", after word segmentation, an ordered word sequence composed of "high-power", "CPU", "and", "GPU", "accompanied", "heat dissipation", etc. is obtained.

[0042] Subsequently, each word is mapped to a vector using an embedding layer. The embedding layer converts each word into a dense vector of fixed dimensions based on pre-trained word embedding parameters, and then, considering batch processing requirements, combines all word vectors into an embedding matrix. ,in The size of the text batch processed at one time. The hidden layer dimension is 1024. The embedding matrix preserves the semantic information of the terms and also adapts to the format requirements of subsequent parallel computation.

[0043] Next, semantic dimension compression and feature extraction are performed on the embedding matrix X using LSH. First, a K×d hash matrix is ​​initialized, where each element is a pre-trained feature extraction parameter, and a corresponding bias term is configured for each hash dimension. For each element in the embedding matrix X, an intermediate calculation result for that dimension is obtained by calculating the weighted sum of all d-dimensional features and the corresponding hash matrix, and then adding the bias term. The intermediate calculation result is then input into a sign function, which outputs ±1 depending on the sign of the result, ultimately generating a K-dimensional semantic feature vector. The formula is as follows: (1) In formula (1), This is an LSH hash matrix with dimensions of . , used to extract semantic features; For bias terms; This is a sign function, and its output is ±1. , For semantic dimensions, such as 64. Semantic feature vectors reduce data dimensionality while preserving core semantic information.

[0044] Then, calculate the semantic feature vector. semantic center of each computational subtree cosine similarity This similarity is obtained by dividing the vector dot product by the product of the magnitudes of the two vectors, and the value ranges from -1 to 1. The closer it is to 1, the higher the semantic fit. Here, the semantic center c... m Obtained through pre-training, it represents the semantic domain processed by the computational subtree. Number the computational subtrees. Based on the calculated cosine similarity, assign the original text to the computational subtree with the highest similarity, ensuring that semantically similar requests are processed within the same computational subtree, reducing semantic interaction and communication overhead across subtrees.

[0045] In the above embodiments, word segmentation transforms the original text into a structured sequence of lexical units, eliminating the ambiguity and redundancy of natural language. Embedding encoding further maps the lexical sequence into a high-dimensional embedding matrix, preserving the contextual semantic relationships of the lexical units. The mapping of semantic feature vectors achieves dimensionality upgrading from low-dimensional lexical representation to high-dimensional semantic representation. Combined with cosine similarity calculation of the matching degree between semantic feature vectors and the semantic centers of computational subtrees, the semantic relationships between the text and computational subtrees can be accurately captured, avoiding matching biases caused by insufficient semantic representation in traditional matching methods and improving the accuracy of target computational subtree selection. Each computational subtree is a substructure of the pre-trained language model after tree-like segmentation. Target computational subtrees that semantically match the original text can be directly selected through semantic similarity matching without calling the complete pre-trained language model for processing, reducing unnecessary computation. At the same time, the tree-like segmented model substructure supports parallel computation, further improving the processing speed of text reasoning.

[0046] S202. Calculate the tree topology depth of the subtree based on the target, and determine the number of layers of the task execution unit of each node.

[0047] In some embodiments, when performing step S202, the number of available computing nodes and the number of basic configuration nodes of the pre-trained language model are first obtained, and then the tree topology depth of the target computing subtree is calculated based on the number of available computing nodes and the number of basic configuration nodes; then, based on the tree topology depth, the first initial layer number of the task execution unit of the leaf node, the second initial layer number of the task execution unit of the intermediate node, and the third initial layer number of the task execution unit of the root node are calculated respectively.

[0048] The number of available computing nodes reflects the actual hardware capacity. The number of basic configuration nodes reflects the minimum functional requirements of the model. The task execution units of leaf nodes are used to perform initial feature extraction, the task execution units of intermediate nodes are used to perform cross-branch gating fusion, and the task execution units of the root node perform the final feature transformation.

[0049] The above embodiments use the number of available computing nodes and the number of basic configuration nodes as the basis for calculating the topology depth. This avoids insufficient hardware resources due to excessive topology depth or waste of resources due to excessively low topology depth, ensuring a high degree of matching between the structure of the target computing subtree and hardware resources and the basic configuration of the model, thus improving the overall utilization rate of computing resources. Based on the calculated tree-like topology depth, the initial layer number of the task execution units of leaf nodes, intermediate nodes, and root nodes is calculated separately, realizing dynamic allocation of the layer number of each type of node, rather than using a uniform layer division method. Differentiated layer configurations can be adapted according to the functional positioning of different nodes (leaf nodes are responsible for hidden state extraction, intermediate nodes are responsible for semantic aggregation, and root nodes are responsible for feature transformation), enhancing the adaptability of the target computing subtree to different text reasoning tasks. Layered calculation of the initial layer number of each type of node can clarify the computing task boundaries of each node, avoid functional overlap of task execution units of different nodes, and ensure clear task flow logic from leaf nodes to root nodes. This reduces computational blocking or redundancy caused by unreasonable layer division, improving the overall execution efficiency of the target computing subtree.

[0050] Optionally, in calculating the tree topology depth of the target computation subtree based on the number of available computation nodes and the number of basic configuration nodes, first calculate the ratio of the number of available computation nodes to the number of basic configuration nodes, then perform a logarithmic operation on the ratio to obtain the logarithmic result, and perform a round-down operation to truncate the decimal part to obtain the tree topology depth of the target computation subtree. The tree topology depth D can be calculated according to the following formula (2): (2) In formula (2), The depth of the tree topology is used to control the parallel granularity, such as... When the value is 3, the tree contains 4 leaf nodes, 2 intermediate nodes, and 1 root node; This refers to the actual number of available computing nodes, such as a 16-node cluster. Configure the number of nodes as the basic configuration, such as ; For the floor function, such as hour, =2.

[0051] The above-mentioned optional implementation method uses the ratio of available computing nodes to basic configuration nodes as the core calculation basis, which can intuitively reflect the redundancy of the actual hardware carrying capacity relative to the minimum functional requirements of the model, providing objective data support for determining the topology depth. The logarithmic operation can transform the linear growth of the node ratio into the logarithmic growth of the topology depth, avoiding excessive expansion of the topology depth due to the increase in the number of hardware nodes, ensuring that the depth expansion is always within a controllable range. At the same time, the floor operation further ensures that the topology depth is an integer, which fits the natural properties of the tree structure, so that the final determined topology depth can make full use of available hardware resources without exceeding the hardware carrying capacity limit due to excessive depth.

[0052] The leaf nodes, intermediate nodes, and root nodes are divided into layers based on the depth of the tree topology, forming task execution units (also known as computation segments).

[0053] The number of task execution unit layers in a leaf node is determined by the ratio of the total number of layers L to 2 raised to the power of (D-1). L is the total number of layers in the pre-trained language model. For example, when L=96 and D=2, L1 has 48 layers.

[0054] The task execution unit of the leaf node is used to extract the initial feature (hidden state) H1. First, a linear transformation is performed on the embedding matrix X, and then a bias term is superimposed for fine-tuning to obtain the transformation result. Then, the transformation result is processed according to the layer normalization function (LayerNorm) to obtain the hidden state of the original text, as shown in formula (3): (3) In formula (3), The leaf node layer weight matrix; dimension It shares weights with the Transformer layer to ensure model accuracy; For bias terms, For layer normalization function, The hidden states output by the leaf nodes provide basic semantic features for subsequent fusion calculations.

[0055] The number of task execution unit layers in intermediate nodes is determined by the ratio of the total number of layers L to 2 raised to the power of (D-2). The task execution unit of the intermediate node is used to receive the hidden state of the left / right child nodes (which may be leaf nodes or other intermediate nodes). , (dimension) The two states are fused through a gating mechanism, integrating the semantic features of different branches to achieve layer-by-layer feature aggregation.

[0056] The number of task execution unit layers in the root node is the total number of layers L minus the sum of the number of layers in the computation segments of the leaf nodes and intermediate nodes. The root node's task execution unit receives the aggregated state output from the upper-level intermediate nodes, transforms the features, and outputs the core feature vector. , dimension .

[0057] The process of dividing the task execution unit into layers described above involves dynamically adjusting the number of layers according to the topology depth D. As the depth increases, the number of layers for leaf nodes and intermediate nodes decreases exponentially, which matches the distribution pattern of a large number of lower-level nodes and a small number of upper-level nodes in a tree structure, thus avoiding computational load imbalance caused by a mismatch between the level and the number of layers. This layered approach enables fine-grained allocation of computing resources. Leaf nodes, as the front-end units for extracting the hidden state of the original text, need to handle a large number of basic semantic parsing tasks. By using the ratio of the total number of layers L to 2 raised to the power of (D-1), a relatively balanced configuration of basic layers is achieved, ensuring the efficiency of parallel processing of multiple leaf nodes. Intermediate nodes undertake semantic aggregation tasks. By using the ratio of the total number of layers L to 2 raised to the power of (D-2), the layer configuration is appropriately improved based on the leaf nodes, enabling them to handle more complex semantic fusion calculations. Root nodes, as the core unit for the final feature transformation, obtain the appropriate remaining layers through completion logic, allowing them to focus on the accurate extraction of core features. This differentiated allocation avoids resource waste caused by redundancy in the number of layers in some nodes and prevents computational bottlenecks caused by insufficient layers in key nodes, thereby improving the overall utilization of computing resources.

[0058] In some embodiments, after calculating the initial number of task execution units for each node, the method further includes: first obtaining the state space of each node when executing a task, then determining the target adjustment action for the number of task execution unit layers based on the state space and the near-end policy optimization engine, and reorganizing the number of task execution units for each node according to the target adjustment action.

[0059] The state space includes the hardware utilization of each node, memory pressure, cross-node latency, and computational complexity of the current task.

[0060] Specifically, after calculating the initial layer number of the task execution unit of each node, the state space S of each node during task execution is monitored. t This includes node hardware utilization, memory pressure, cross-node latency, and the L2 norm of hidden states. ,in, No. The hardware utilization of each node, such as a GPU utilization of 80%, indicates resource shortage and the need to split task execution units to alleviate the pressure. For the first The memory pressure of each node, such as memory utilization rate, ranges from 0-100%, reflecting the memory usage load of the node. For the first With the Inter-node network latency, measured in milliseconds, affects the efficiency of cross-node data transmission. For the first The L2 norm of the hidden state reflects the complexity of the current computational task.

[0061] Let the state space S t Input PPO strategy engine output adjustment action for the number of task execution unit layers ,in For the first The adjustment amount of the number of layers in the task execution unit is positive, which means increasing the number of layers within the segment, which can reduce the number of communications; negative means reducing the number of layers within the segment, which can improve parallelism, such as ±2 layers. For the first The number of parallel segments in a layer can be adjusted, such as ±1 segments, to adapt to different load requirements by increasing or decreasing the number of parallel segments.

[0062] The initial layer allocation is calculated based on a fixed formula of the total number of layers in the model and the topology depth, which is difficult to fully match the dynamic changes in actual task execution. However, by obtaining the state space of each node when it executes a task (which can cover real-time information such as computing load, resource utilization, and task processing progress), the actual operating state differences of each node can be accurately captured, avoiding the imbalance problem of "some nodes being overloaded and some nodes being idle" that may exist in the initial allocation. The near-end policy optimization engine then uses this real-time state data to generate the optimal layer adjustment action through reinforcement learning logic, realizing the dynamic reorganization of the number of layers of each node. This ensures that the layer configuration matches the actual carrying capacity of the nodes and the task processing requirements in real time, improving the overall balance of computing load.

[0063] Dynamic adjustments can also specifically address bottlenecks in task execution. When a node experiences computational delays due to excessive layers, adjustments can be made to reduce the number of layers in its task execution unit, distributing some tasks to nodes with lighter loads. Conversely, when node resources are redundant, the number of layers can be appropriately increased to fully utilize resources and avoid wasting computing power. This precise dynamic optimization shortens the overall task processing cycle and reduces inter-layer synchronization waiting time caused by load imbalance, further improving the collaborative execution efficiency of tree-structured computational subtrees.

[0064] Optionally, in the process of determining the target adjustment action of the task execution unit layer based on the state space and the near-end policy optimization engine, the state space is first input into the near-end policy optimization engine to obtain the predicted adjustment action, and then the reward function result corresponding to the predicted adjustment action is calculated. If the reward function result meets the preset adjustment conditions, the predicted adjustment action is taken as the target adjustment action.

[0065] The adjustment actions include adjusting the number of layers in the task execution unit and adjusting the number of parallel segments.

[0066] Action decisions depend on the reward function of the PPO algorithm, as shown in formula (4) below: (4) In formula (4), These are the weighting coefficients of the reward function, used to balance throughput, latency, and energy consumption. They can be adjusted according to the scenario, such as... , , ; For actual throughput, As a benchmark throughput, This represents the actual P99 latency, indicating that in all request latency data, 99% of the request latency is less than or equal to this value, and only 1% of the request latency is greater than or equal to this value. As a reference delay, Actual energy consumption This is the baseline energy consumption.

[0067] The reward function compares actual throughput, actual P99 latency, and actual energy consumption with their respective baseline values ​​to calculate the reward result. It then determines whether the reward function result meets adjustment conditions, such as a reward value exceeding a threshold or a load balancing improvement rate meeting a target. If met, the adjusted task execution unit parameters are migrated to the task execution units via Remote Direct Memory Access (RDMA) protocol, achieving segmented reorganization. If not met, the current strategy is maintained, and status changes continue to be monitored to ensure that the task execution unit configuration can respond to load fluctuations and maintain efficient operation.

[0068] Then, based on the target, the adjustment actions include the adjustment amount of the number of task execution units and the adjustment amount of parallel segments, and the tasks of each node's task execution units are re-divided to achieve load balancing of each node and improve communication efficiency.

[0069] The above optional implementation first inputs the real-time state space of each node into the near-end policy optimization engine, and uses the engine's reinforcement learning capability to generate predictive adjustment actions that fit the current operating state, avoiding blind adjustments without basis; then, the predictive adjustment actions are quantitatively verified through a reward function. The reward function transforms the abstract adjustment effect into a quantifiable numerical result. Only when the result of the reward function meets the preset adjustment conditions is the predictive adjustment action determined as the target adjustment action, effectively filtering out invalid adjustment actions that may lead to load imbalance and efficiency reduction, and improving the accuracy and rationality of layer adjustment decisions.

[0070] S203. Input the original text into the target computation subtree and obtain the core feature vector output by the target computation subtree.

[0071] The target computation subtree is based on the number of task execution units at each node. The hidden state of the original text is determined by the task execution units at the leaf nodes. The aggregated state is obtained by semantically aggregating the hidden state and the reference state through the task execution units at the intermediate nodes. Finally, the core feature vector is obtained by feature transformation of the aggregated state through the task execution unit at the root node. The reference state is a historically frequently occurring hidden state, derived from classic semantic scenarios that have appeared multiple times in past inference tasks. The reference state carries clear domain semantic features, such as the feature states generated by tasks related to "GPU computing power" and "CPU power consumption" in the server domain.

[0072] The leaf node's task execution unit extracts features from the input raw text and outputs the hidden state (H1).

[0073] The task execution unit of the intermediate node queries the reference state from the reference state dictionary, then calculates the cosine similarity between the hidden state and the reference state, determines the weight coefficient of the reference state based on the cosine similarity, and then concatenates the hidden state, the reference state and their weight coefficients to obtain the aggregate state.

[0074] The reference state dictionary is dynamically updated and is used to store historically frequently occurring hidden states. The dictionary has a default capacity of M (e.g., 1000). When a new frequently occurring hidden state appears, the contents of the reference state dictionary are updated according to rules to ensure that the dictionary always contains representative semantic features.

[0075] Specifically, the task execution unit of the intermediate node first obtains the hidden state H of the leaf node or other intermediate nodes. k At the same time, the reference state is queried from the reference state dictionary S. ,in, This refers to historically high-frequency hidden states, such as "GPU computing power" and "CPU power consumption" in the server field; Let the dictionary capacity be 1000. Then calculate the hidden state. With reference states in the dictionary The cosine similarity is calculated by dividing the dot product of the two sets of state vectors by the product of the magnitudes of the two sets of vectors. This measures the degree of fit between the two states in the semantic space. The higher the degree of fit, the closer the cosine similarity value is to 1. And generate weights using Softmax. As shown in formula (5), the cosine similarity of a single group is divided by the smoothing coefficient and then substituted into the natural exponent calculation to obtain the exponent mapping result corresponding to the reference state. The smoothing coefficient controls the weight distribution characteristics. If the coefficient value is too small, the numerical difference between the high matching reference states will widen. If the coefficient value is too large, the mapping results of each group will tend to be balanced. After traversing all reference states in the dictionary and completing the exponent mapping operation of each group of states, the total number of mapping results is summed to obtain the total exponent of the whole domain. The exponent mapping result of a single reference state is used as the numerator and the total exponent of the whole domain is used as the denominator for division. The result is the weight coefficient corresponding to the reference state. The sum of the weight coefficients of all reference states is always equal to one, which can quantify the proportion of the influence of each historical reference hidden state on the current hidden state. (5) In formula (5), For smoothing coefficients, The smaller the value, the more concentrated the weight is on reference states with high similarity; The larger the value, the more evenly weighted the distribution. The sum of the weight coefficients of all reference states is 1, accurately reflecting the degree of influence of each reference state on the current hidden state.

[0076] Subsequently, the hidden state H is fused using a multilayer perceptron (MLP). k Compared with the reference state, the output aggregate state is obtained. The semantic weighted aggregation is completed as shown in formula (6). The current hidden state to be processed and a single set of reference states are combined by vector concatenation. The concatenation operation completely retains the real-time semantic information of the current state and introduces the prior knowledge of the industry domain stored in the reference state. After concatenation, the feature dimension is doubled. The combined features after concatenation are sent to the multilayer perceptron fusion network to obtain the feature fusion result. The input dimension of the network is the extended dimension after concatenation, and the output dimension is restored to the original state dimension. The trainable parameters inside the network are optimized and converged by the model fine-tuning process to ensure that the fused features have unified semantic representation characteristics. The weight coefficients of the corresponding reference states calculated in the early stage are multiplied element-wise with the fusion result of the single set of features to obtain the fused features. The reference states with higher weight values ​​will have a higher contribution ratio in the final result. All historical reference states in the reference state dictionary are traversed. After repeating the concatenation, network fusion, and weighted multiplication process, all weighted fused features are accumulated one by one to finally output the aggregated state. The aggregated state carries both the current real-time semantic features and the prior experience of historical high-frequency states.

[0077] (6) In formula (6), This represents the concatenation of two states, preserving the real-time semantic details contained in the current hidden state, and incorporating the domain knowledge carried by the reference state. The dimensions of the concatenated state are expanded to... MLP is a fusion network whose parameters are trained through model fine-tuning to ensure semantic consistency of the fused states; input dimension Output dimension The higher the weight of the reference state, the greater the proportion of the fused features in the aggregated state, and the aggregated state takes into account both real-time semantics and historical experience.

[0078] The task execution unit of the intermediate node provided in this application retrieves historical high-frequency hidden states from the reference state dictionary. By calculating the cosine similarity between the current hidden state and each reference state, it can accurately measure the degree of fit between the two in the semantic space. Then, it uses the Softmax function to generate weight coefficients to focus the weights on high-similarity reference states, ensuring the priority integration of historical high-frequency semantic features during the aggregation process. Subsequently, the hidden state and the reference state are spliced ​​and fused through MLP, which not only preserves the real-time semantic details of the current hidden state, but also incorporates the domain knowledge carried by the reference state. This makes the final generated aggregated state take into account both real-time performance and historical experience, effectively improving the accuracy and richness of semantic aggregation.

[0079] The reference state dictionary stores historically frequent hidden states. When intermediate nodes perform semantic aggregation, they do not need to transmit the complete hidden states of all leaf nodes or other intermediate nodes. They only need to combine the reference state in the local dictionary with the current hidden state for fusion calculation, which greatly reduces the amount of intermediate state transmission between devices. At the same time, the dynamic update mechanism of the dictionary ensures that it always contains representative semantic features, avoiding aggregation deviations caused by outdated reference states. This reduces transmission redundancy while ensuring aggregation effect.

[0080] The reference state dictionary can store high-frequency hidden states related to areas such as "GPU computing power" and "CPU power consumption" in the server domain, enabling the model to quickly capture core semantic features within the domain. The dictionary's preset capacity and dynamic update rules avoid the reduced query efficiency caused by an excessively large dictionary while promptly incorporating new high-frequency hidden states. This ensures the model can integrate domain knowledge across different domain tasks, significantly enhancing its domain adaptability and generalization. Furthermore, the introduction of the smoothing coefficient κ makes weight allocation adjustable, allowing for adjustments to the weight concentration based on different task scenarios, further improving the flexibility and scenario adaptability of the technical solution.

[0081] The root node's task execution unit contains the remaining The layer performs the final feature transformation and outputs the core feature vector. , dimension .

[0082] In some embodiments, task execution units can be divided into two categories according to computation type: attention-intensive and FFN-intensive.

[0083] Attention-intensive tasks, which primarily involve multi-head attention computation (such as the Encoder / Decoder layers of a Transformer), require highly parallel hardware. These tasks, where attention computation accounts for over 60% (such as the first 32 layers of a Transformer), demand high parallel processing capabilities from the hardware, requiring hardware capable of handling multiple parallel tasks simultaneously.

[0084] FFN-intensive tasks primarily rely on feedforward network computation (such as the intermediate layers of a Transformer), requiring hardware with high matrix multiplication efficiency. FFN computation accounts for over 70% of these tasks (such as the last 64 layers of a Transformer), and the core of these tasks is matrix multiplication, necessitating hardware with efficient matrix multiplication processing capabilities.

[0085] Based on the task characteristics of each task execution unit, each task execution unit is assigned to a corresponding processor. Attention-intensive tasks are assigned to the GPU, leveraging the GPU's thread-level parallelism to accelerate attention computation, as shown in the following formula: (7) In formula (7), To query the key / value matrix, The number of attention heads (e.g., 16). To output the weight matrix, the GPU can process the computation of 16 attention heads in parallel. The GPU can process the computation of multiple attention heads simultaneously, for example, 16 attention heads can operate synchronously. By concatenating the computation results of each attention head and performing a linear transformation with the output weight matrix, the final attention computation result is obtained. This parallel processing method can reduce the attention computation latency by 50%.

[0086] The FFN-intensive segments are allocated to the TPU, and the matrix multiplication unit (MXU) of the TPU is used to accelerate FFN calculation, as shown in the following formula: (8) In formula (8), Here is the FFN weight matrix, with dimensions as follows: The TPU's Matrix Multiplication Unit (MXU) is a core component for efficiently processing matrix operations. It can perform large-dimensional matrix multiplications in a single operation (such as the multiplication of a d×4d weight matrix and a feature matrix). Its matrix multiplication efficiency can be increased to 1.8 times that of the GPU, perfectly meeting the computational requirements of FFN tasks.

[0087] In some embodiments, the core task of the leaf node computation segment is semantic extraction. This type of task has a relatively moderate computational load and is sensitive to data transmission latency. The task execution unit of the leaf node can be deployed on edge devices, allowing the data to complete preliminary semantic extraction at the edge, close to the user, reducing the latency of data transmission to the cloud. The root node computation segment is responsible for the final feature transformation and prediction, which requires stronger computing power to support complex calculations. The task execution unit of the root node can be deployed in a cloud cluster, leveraging the abundant hardware resources of the cloud to ensure computational accuracy and efficiency. Through this collaborative architecture where the edge handles preliminary processing and the cloud handles core computation, the computing power advantages of different deployment environments are fully utilized while reducing transmission latency, achieving the optimal balance of overall inference performance.

[0088] S204. Convert the core feature vectors into the natural language text corresponding to the original text.

[0089] In some embodiments, when performing step S204, the core feature vector is first linearly transformed to obtain a log-odds vector, which includes the original prediction score of each candidate word in the vocabulary; then the generation probability of each candidate word is calculated by temperature sampling, and the natural language text corresponding to the original text is determined based on the generation probability.

[0090] Specifically, the root node outputs the feature vector. (dimension) The output layer converts the logistic odds vector into a logits vector. ( The size of the vocabulary (e.g., 50257, covering all candidate words) is specified in the following formula: (9) In formula (9), To output the weight matrix (dimensions) ), This is the output bias term. The output weight matrix and bias term can be optimized through model training to make the generated log-odds vector more closely match the semantic distribution of the vocabulary. Each element of the log-odds vector z corresponds to the original prediction score of a candidate word in the vocabulary.

[0091] Candidate word probabilities are obtained through temperature sampling. The Logits vector is then processed. Temperature sampling is performed to control the diversity of generated text. The probability calculation logic of temperature sampling is based on the original prediction scores of all candidate words in the vocabulary, ensuring that the generation probability can accurately reflect the semantic fit between the candidate words and the core feature vector. First, the original prediction scores of all words output by the model are extracted to form a prediction score vector z. Each element in the Logits vector z (the original prediction score of the candidate word) is divided by the temperature parameter τ. The temperature parameter adjusts the concentration of the probability distribution of candidate words. The smaller the temperature parameter, the more obvious the difference in the probability proportion of high-scoring candidate words. The larger the temperature parameter, the smoother the probability distribution of each word. Then, the original prediction scores of all candidate words are converted into non-negative values ​​through exponential operation to obtain the exponential mapping result of each candidate word. All candidate words in the vocabulary are traversed to complete the exponential mapping operation of the scaling scores of all words. The global exponential sum is calculated by accumulating all mapping results. Finally, the generation probability of each candidate word is obtained by dividing the exponential mapping result of a single candidate word by the global exponential sum, as shown in the following formula: (10) In formula (10), Candidate words, The temperature parameter has different values, which affect the characteristics of the generated text. When the probability is highly concentrated on candidate words with high prediction scores, the generated text is highly deterministic but low in diversity, which is suitable for scenarios where high accuracy of results is required. When the probability distribution is relatively balanced, the text can ensure semantic coherence and also has a certain degree of diversity, making it suitable for common scenarios such as dialogue generation and reasoning question answering. At this time, the probability distribution is more gradual, each candidate word has an equal chance of being selected, the text diversity is high but the certainty is reduced, which is suitable for creative writing and other scenarios that require rich output; The probability of a candidate word is used to select the next generated word based on that probability.

[0092] In the above embodiments, the core feature vector is the key output after the root node performs feature transformation on the aggregation state. It carries the core semantic information of the original text. Through the output layer, it is linearly transformed into a log-odds vector, which can be directly mapped to the original prediction scores of candidate words in the vocabulary, ensuring that the semantic association between the feature vector and the candidate words is not lost. Subsequently, the generation probability of candidate words is calculated based on the log-odds vector, so that the final generated natural language text closely matches the core semantics of the original text, avoiding semantic deviation or logical breaks, and improving the semantic consistency between the generated text and the original text. The entire process is compatible with the preceding tree-like computation subtree structure. After being processed layer by layer by leaf nodes and intermediate nodes, the semantic information of the core feature vector can be fully transformed into natural language text in this step, ensuring the end-to-end adaptability of the technical solution from text input to result output, and improving the effectiveness of the overall text reasoning process.

[0093] In some embodiments, after performing a linear transformation on the core feature vector to obtain the log-odds vector, the method further includes: first determining whether the text prefix of the candidate word matches the cached result in a preset prefix tree; if the text prefix of the candidate word matches the target cached result, obtaining the natural language text corresponding to the original text based on the subsequent sequence and text prefix of the target cached result. This eliminates the need for word-by-word generation through sampling, reduces redundant calculations, and improves generation speed.

[0094] Among them, the preset prefix tree cache It stores frequently occurring generation sequences, which are common expressions in specific scenarios, such as consecutive word combinations like "heat dissipation" and "linkage" in server system design. When the generated prefix matches the cache, the cached result is directly called, reducing redundant calculations. Experiments show that the generation latency is 20ms when a match is found and 30ms when a match is found, resulting in a speedup of 1.5 times.

[0095] The system queries the cached results of the preset prefix tree and determines whether the text prefix of the candidate word matches. If the text prefix of the candidate word matches a cached result in the preset prefix tree, the cached result is used to generate subsequent text, reducing redundant calculations. If there is no match, subsequent text is generated word by word according to the probability obtained from temperature sampling. Finally, all generated words are integrated to output natural language text that conforms to semantic logic and is easy to understand.

[0096] The preset prefix tree in the above embodiment caches the prefix-subsequence sequence correspondence results accumulated in historical generation tasks. When the text prefix of a candidate word matches the cached result in the tree, the subsequent sequence corresponding to the target cached result can be directly called without recalculating the candidate word generation probability through temperature sampling and generating word by word, thus shortening the text generation cycle in short text or high-frequency prefix scenarios. The prefix tree caching mechanism can effectively avoid redundant calculations. For frequently occurring text prefixes, their corresponding subsequent sequences only need to be probability calculated and sequence determined during the first generation. When the same prefix is ​​encountered subsequently, the cached result can be directly reused without repeating linear transformation, temperature sampling, and other calculation steps, reducing the resource consumption for core feature vector processing and candidate word probability calculation. At the same time, the tree structure of the prefix tree has efficient prefix matching capabilities, which can quickly locate the target cached result, avoiding additional resource consumption during the cache query process and further optimizing the utilization efficiency of computing resources. The cached subsequent sequences in the prefix tree are validated results from historical generation tasks, possessing good semantic coherence and rationality. When a text prefix matches a target cached result, combining the text prefix with the cached subsequent sequence generates natural language text. This avoids semantic breaks caused by temperature parameter adjustments or probability distribution fluctuations during pure probability generation. Furthermore, the reuse of cached results ensures consistency of subsequent sequences corresponding to the same prefix across different generation tasks, improving the overall quality of the generated text. In addition, this mechanism complements the temperature sampling generation path. For text prefixes that do not match cached results, text can still be generated by calculating the generation probability through temperature sampling. This retains the flexibility of probability generation while optimizing efficiency in high-frequency scenarios through cache matching, enhancing the scenario adaptability and practicality of the technical solution.

[0097] In summary, the text reasoning method provided in this application first achieves accurate semantic alignment between the original text and the pre-trained language model's tree-like segmentation and subtree calculation through word segmentation, embedding encoding, semantic mapping, and cosine similarity matching, avoiding semantic bias in traditional matching methods. Intermediate nodes utilize a dynamically updated reference state dictionary, combined with cosine similarity calculation and Softmax weight generation, and fuse the current hidden state with historical high-frequency reference states through MLP, ensuring that the aggregated state takes into account both real-time semantic details and domain historical experience, thus improving the accuracy of semantic aggregation. Finally, text generation is achieved through linear transformation of the core feature vector and temperature sampling probability calculation. The adjustability of the temperature parameter τ allows for flexible control of the determinism and diversity of the generated text, meeting the needs of different scenarios.

[0098] The tree-like topology design of the computation subtree breaks away from the traditional parallel mode's layer-by-layer serial synchronization logic. Leaf nodes, intermediate nodes, and root nodes can execute tasks in parallel based on the topology depth, solving the problem of strong inter-layer synchronization dependencies. The tree-like topology depth is determined based on the number of available computing nodes and the number of basic configuration nodes. Then, an exponential formula is used to achieve differentiated allocation of the initial number of layers for each node. At the same time, a dynamic adjustment mechanism of state space awareness and near-end policy optimization engine is introduced. The rationality of the adjustment action is ensured through reward function verification, realizing the upgrade from static allocation to dynamic adaptation of the number of layers and solving the problem of low expansion efficiency.

[0099] The reference state dictionary stores historical high-frequency hidden states. When intermediate nodes are aggregated, there is no need to transmit the full intermediate state. Only the local dictionary and the current hidden state need to be combined for calculation, which greatly reduces the redundancy of intermediate state transmission. At the same time, the prefix tree caching mechanism reuses the subsequent sequence of high-frequency prefixes to avoid repeated calculations and further optimize resource consumption.

[0100] From the perspective of scenario adaptation and scalability improvement, the entire technical process is decoupled from the tree-like segmentation structure of the pre-trained language model. When the number of model parameters expands or hardware resources change, only the granularity of the computational subtree segmentation, topology depth, or number of node layers needs to be adjusted, without reconstructing the overall architecture. The reference state dictionary can store domain-related high-frequency hidden states, and the dynamic update mechanism ensures that it always contains representative semantic features, enhancing domain adaptability. The dual-path design of cached matching and probability generation improves the efficiency of high-frequency scenario generation while retaining the flexibility of probability generation, adapting to various scenarios such as dialogue generation, reasoning question answering, and creative writing. In addition, the quantitative calculation logic of each technical link (such as topology depth calculation, layer allocation formula, weight generation algorithm, etc.) abandons subjective experience judgment, reduces human error, improves the reusability and operability of the solution, and ensures that the technical method has significant application value in the efficient and stable processing requirements of large language models with hundreds of billions of nodes.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0102] like Figure 3 As shown, embodiments of this application also provide a text reasoning apparatus, the apparatus comprising: The acquisition module 301 is used to acquire the original text and determine the target computation subtree that semantically matches the original text; The determination module 302 is used to calculate the tree topology depth of the subtree based on the target and determine the number of layers of the task execution unit of each node; The inference module 303 is used to input the original text into the target computation subtree and obtain the core feature vector output by the target computation subtree. The target computation subtree is used to determine the hidden state of the original text by the task execution unit of the leaf node based on the number of layers of the task execution unit of each node. The task execution unit of the intermediate node performs semantic aggregation on the hidden state and the reference state to obtain the aggregated state. The task execution unit of the root node performs feature transformation on the aggregated state to obtain the core feature vector. The reference state is the hidden state that appears frequently in history. The conversion module 304 is used to convert the core feature vector into the natural language text corresponding to the original text.

[0103] As an optional implementation provided in this application, the acquisition module 301 is used to: perform word segmentation on the original text to generate a word sequence; perform embedding encoding on the word sequence to obtain an embedding matrix; map the embedding matrix into a semantic feature vector; calculate the cosine similarity between the semantic feature vector and the semantic center of each computational subtree; wherein each computational subtree is a model substructure after tree segmentation of the pre-trained language model; and determine the target computational subtree corresponding to the original text input based on the cosine similarity.

[0104] As an optional implementation provided in this application, the acquisition module 301, in the process of mapping the embedding matrix to the semantic feature vector, specifically calculates the weighted sum of the dimensional features and the corresponding hash matrix for each element in the embedding matrix, and superimposes the bias term to obtain the intermediate calculation result corresponding to the dimensional features; each element in the hash matrix is ​​a pre-trained feature extraction parameter, and each hash dimension is configured with a corresponding bias term; the intermediate calculation result is input into the sign function to obtain the semantic feature vector.

[0105] As an optional implementation provided in this application, the determining module 302 is used to: obtain the number of available computing nodes and the number of basic configuration nodes of the pre-trained language model; calculate the tree topology depth based on the number of available computing nodes and the number of basic configuration nodes; and calculate the first initial layer number of the task execution unit of the leaf node, the second initial layer number of the task execution unit of the intermediate node, and the third initial layer number of the task execution unit of the root node based on the tree topology depth.

[0106] As an optional implementation provided in this application, the determining module 302, after calculating the first initial layer number of the task execution unit of the leaf node, the second initial layer number of the task execution unit of the intermediate node, and the third initial layer number of the root node according to the depth of the tree topology, is further configured to: obtain the state space of each node when executing a task, the state space including the hardware utilization, memory pressure, cross-node latency, and computational complexity of the current task of each node; determine the target adjustment action for the layer number of the task execution unit based on the state space and the near-end policy optimization engine; and reorganize the layer number of the task execution unit of each node according to the target adjustment action.

[0107] As an optional implementation provided in this application, the determining module 302, based on the state space and the near-end policy optimization engine, determines the target adjustment action for the number of task execution unit layers. This is used to: input the state space into the near-end policy optimization engine to obtain the predicted adjustment action, which includes the adjustment amount of the number of task execution unit layers and the adjustment amount of parallel segments; calculate the reward function result corresponding to the predicted adjustment action, where the reward function result reflects the throughput, latency, and energy consumption corresponding to the predicted adjustment action; and, if the reward function result meets preset adjustment conditions, use the predicted adjustment action as the target adjustment action.

[0108] As an optional implementation provided in this application, the determining module 302, after reorganizing the number of layers of each node task execution unit according to the target adjustment action, is further used to: redivide the tasks of each node task execution unit according to the adjustment amount of the number of task execution units and the adjustment amount of parallel segments included in the target adjustment action, so as to balance the load of each node.

[0109] As an optional implementation provided in this application, the task execution unit of the intermediate node is used to: query the reference state from the reference state dictionary; calculate the cosine similarity between the hidden state and the reference state; determine the weight coefficient of the reference state based on the cosine similarity; and concatenate the hidden state, the reference state and their weight coefficients to obtain the aggregate state.

[0110] As an optional implementation provided in this application, the task execution unit of the intermediate node, in the process of determining the weight coefficient of the reference state based on the cosine similarity, is specifically used to: divide the cosine similarity by the smoothing coefficient and then input it into the natural exponent calculation to obtain the exponent mapping result corresponding to the reference state; sum the exponent mapping results corresponding to the reference state to obtain the global exponent sum; and perform a division operation with the exponent mapping result of a single reference state as the numerator and the global exponent sum as the denominator to obtain the weight coefficient of the reference state.

[0111] As an optional implementation provided in this application, the task execution unit of the intermediate node, in the process of concatenating the hidden state and the reference state and their weight coefficients to obtain the aggregated state, is specifically used for: performing a vector concatenation operation on the hidden state and the reference state to obtain combined features; obtaining feature fusion results based on the combined features and the multilayer perceptron fusion network; performing element-wise multiplication operations on the feature fusion results and the weight coefficients of the reference state to obtain fused features; and summing the fused features to obtain the aggregated state.

[0112] As an optional implementation provided in this application, the conversion module 304 is used to: perform a linear transformation on the core feature vector to obtain a log-odds vector; the log-odds vector includes the original prediction scores of each candidate word in the vocabulary; calculate the generation probability of each candidate word through temperature sampling; and determine the natural language text corresponding to the original text based on the generation probability.

[0113] As an optional implementation provided in this application, the conversion module 304 performs a linear transformation on the core feature vector to obtain a log-odds vector, and then further uses it to: determine whether the text prefix of the candidate word matches the cached result in the preset prefix tree; if the text prefix matches the target cached result, obtain the natural language text corresponding to the original text based on the subsequent sequence and text prefix corresponding to the target cached result.

[0114] As an optional implementation provided in this application, the determining module 302 calculates the tree topology depth based on the number of available computing nodes and the number of basic configuration nodes, and is used to: calculate the ratio of the number of available computing nodes to the number of basic configuration nodes; perform a logarithmic operation on the ratio to obtain the logarithmic result; and perform a floor operation on the logarithmic result to obtain the tree topology depth.

[0115] As an optional implementation provided in this application, the leaf node task execution unit is used to: perform a linear transformation on the embedding matrix through the leaf node task execution unit, and then fine-tune it by superimposing a bias term to obtain the transformation result; process the transformation result according to the layer normalization function to obtain the hidden state of the original text.

[0116] For a description of the features in the embodiment corresponding to the text reasoning device, please refer to the relevant description of the embodiment corresponding to the text reasoning method, which will not be repeated here.

[0117] like Figure 4 As shown, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described text reasoning method embodiments.

[0118] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described text reasoning method embodiments at runtime.

[0119] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0120] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described text reasoning method embodiments.

[0121] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described text reasoning method embodiments.

[0122] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-chip (SoC), a complex programmable logic device (CPLD), a microcontroller unit (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0123] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0124] The foregoing has provided a detailed description of a text reasoning method, apparatus, electronic device, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A text-based reasoning method, characterized in that, include: Obtain the original text and determine the target computation subtree that semantically matches the original text; Calculate the tree topology depth of the subtree based on the target, and determine the number of layers of the task execution unit of each node; The step of calculating the tree topology depth of the subtree based on the target and determining the number of layers of task execution units at each node includes: obtaining the number of available computing nodes and the number of basic configuration nodes of the pre-trained language model; calculating the tree topology depth based on the number of available computing nodes and the number of basic configuration nodes; and calculating the first initial layer of task execution units at leaf nodes, the second initial layer of task execution units at intermediate nodes, and the third initial layer of task execution units at the root node based on the tree topology depth. The state space of each node during task execution is obtained, including hardware utilization, memory pressure, cross-node latency, and computational complexity of the current task. This state space is then input into a near-end policy optimization engine to obtain predicted adjustment actions, including adjustments to the number of layers in task execution units and the number of parallel segments. The reward function result corresponding to the predicted adjustment action is calculated, reflecting the throughput, latency, and energy consumption. If the reward function result meets preset adjustment conditions, the predicted adjustment action is used as the target adjustment action. The number of layers in the task execution units of each node is then reorganized based on the target adjustment action. Based on the target adjustment actions, including the adjustment amount of the number of layers of task execution units and the adjustment amount of parallel segments, the tasks of each node's task execution units are re-divided to achieve load balancing of each node. The original text is input into the target computation subtree to obtain the core feature vector output by the target computation subtree. The target computation subtree is used to determine the hidden state of the original text through the task execution units of the leaf nodes based on the number of layers of the task execution units of each node. The task execution units of the intermediate nodes perform semantic aggregation on the hidden state and the reference state to obtain the aggregated state. The task execution unit of the root node performs feature transformation on the aggregated state to obtain the core feature vector. The reference state is the historically frequently occurring hidden state. The core feature vector is converted into the natural language text corresponding to the original text.

2. The method according to claim 1, characterized in that, The step of obtaining the original text and determining the target computation subtree that semantically matches the original text includes: The original text is segmented to generate a word sequence; The word sequence is embedded and encoded to obtain an embedding matrix; Map the embedding matrix to a semantic feature vector; Calculate the cosine similarity between the semantic feature vector and the semantic center of each computational subtree; wherein, each computational subtree is a substructure of the model after tree segmentation of the pre-trained language model; Based on the cosine similarity, the target computation subtree corresponding to the original text is determined.

3. The method according to claim 2, characterized in that, The step of mapping the embedding matrix to a semantic feature vector includes: For each element in the embedding matrix, the intermediate calculation result corresponding to the dimensional feature is obtained by calculating the weighted sum of the dimensional feature and the corresponding hash matrix, and then adding the bias term; each element in the hash matrix is ​​a pre-trained feature extraction parameter, and each hash dimension is configured with a corresponding bias term; The intermediate calculation results are input into the symbol function to obtain the semantic feature vector.

4. The method according to claim 1, characterized in that, The task execution unit of the intermediate node is used for: Retrieve the reference status from the reference status dictionary; Calculate the cosine similarity between the hidden state and the reference state; The weight coefficients of the reference state are determined based on the cosine similarity. The aggregated state is obtained by concatenating the hidden state, the reference state, and their weight coefficients.

5. The method according to claim 4, characterized in that, The step of determining the weight coefficients of the reference state based on the cosine similarity includes: Divide the cosine similarity by the smoothing coefficient and then input it into the natural exponent calculation to obtain the exponential mapping result corresponding to the reference state; The total exponent sum is obtained by summing the exponent mapping results corresponding to the reference state. Using the index mapping result of a single reference state as the numerator and the sum of the global indices as the denominator, a division operation is performed to obtain the weight coefficient of the reference state.

6. The method according to claim 4, characterized in that, The process of concatenating the hidden state and the reference state with their weight coefficients to obtain the aggregated state includes: Perform a vector concatenation operation on the hidden state and the reference state to obtain combined features; Based on the combined features and the multilayer perceptron fusion network, the feature fusion result is obtained; The fused feature is obtained by performing an element-wise multiplication operation between the feature fusion result and the weight coefficients of the reference state; The aggregation state is obtained by summing the fusion features.

7. The method according to claim 1, characterized in that, The process of converting the core feature vector into natural language text corresponding to the original text includes: A linear transformation is performed on the core feature vector to obtain a log-odds vector; the log-odds vector includes the original prediction scores of each candidate word in the vocabulary. The generation probability of each candidate word is calculated by temperature sampling; The natural language text corresponding to the original text is determined based on the generation probability.

8. The method according to claim 7, characterized in that, After performing a linear transformation on the core feature vector to obtain the log-odds vector, the method further includes: Determine whether the text prefix of the candidate word matches the cached result in the preset prefix tree; If the text prefix matches the target cache result, the natural language text corresponding to the original text is obtained based on the subsequent sequence corresponding to the target cache result and the text prefix.

9. The method according to claim 1, characterized in that, The step of calculating the tree topology depth based on the number of available computing nodes and the number of basic configuration nodes includes: Calculate the ratio of the number of available computing nodes to the number of basic configuration nodes; Perform a logarithmic operation on the ratio to obtain the logarithmic result; The logarithmic result is truncated by performing a floor operation to obtain the tree topology depth.

10. The method according to claim 2, characterized in that, The hidden state of the original text input is determined by the execution unit of the leaf node task, including: The task execution unit of the leaf node performs a linear transformation on the embedding matrix, and then fine-tunes it by adding a bias term to obtain the transformation result. The transformation result is processed according to the layer normalization function to obtain the hidden state of the original text.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the text reasoning method as described in any one of claims 1 to 10 when executing the computer program.

Images