Model inference method and electronic device

Abstract

The application provides a model inference method and an electronic device, including: determining an activity result of each neuron in each layer of a target model based on a structure analysis result of the target model; performing neuron-level physical reorganization on model weights of each layer in the target model based on the activity result to obtain a physical reorganization result; obtaining a local calculation result of a current layer in a process of inference calculation on the current layer of the target model based on the physical reorganization result; predicting a neuron activation state of a next layer and generating a neuron activation mask of the next layer based on the local calculation result of the current layer; determining target weight data of the next layer based on the neuron activation mask and the physical reorganization result of the next layer, and performing gated reconstruction on an initial calculation array based on the neuron activation mask of the next layer to obtain a target calculation array; and performing inference calculation on the next layer of the target model based on the target weight data and the target calculation array of the next layer, thereby improving the execution efficiency of model inference.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model reasoning method and an electronic device. Background Technology

[0002] In recent years, with the development of artificial intelligence technology, edge computing technology and heterogeneous hardware, large models have been widely used in computing boxes, mobile robots, intelligent exoskeletons and other electronic devices because they can support functions such as environmental perception, semantic understanding, real-time decision-making and motion control.

[0003] However, taking large-model inference in electronic devices as an example, the model weights are typically much larger than the processor's on-chip cache capacity. Existing inference methods struggle to finely organize the weight data by combining model structural features and neuron activity levels, leading to frequent loading of large amounts of weight data during inference, which can easily cause excessive bus bandwidth usage and increased inference latency. Simultaneously, traditional computing arrays struggle to predetermine the target weight data and computational regions to be used based on the actual activation state of the next layer of neurons, resulting in numerous invalid weight reads and redundant calculations. Summary of the Invention

[0004] The model inference method and electronic device provided in this application are intended to solve the defects of increased inference latency and excessive redundant calculations in existing model inference methods.

[0005] In a first aspect, embodiments of this application provide a model inference method. This model inference method includes: determining the activity results of neurons in each layer of the target model based on structural analysis results; performing neuron-level physical reorganization on the model weights of each layer of the target model based on the activity results to obtain physical reorganization results; acquiring local computation results of the current layer during inference computation of the target model based on the physical reorganization results; predicting the neuron activation state of the next layer based on the local computation results of the current layer and generating a neuron activation mask for the next layer; determining target weight data for the next layer based on the neuron activation mask and the physical reorganization results, and performing gated reconstruction on an initial computation array based on the neuron activation mask of the next layer to obtain a target computation array; and performing inference computation on the next layer of the target model based on the target weight data and the target computation array.

[0006] In some embodiments, determining the activity results of neurons in each layer of the target model based on the structural analysis results of the target model includes: performing multiple rounds of forward inference sampling on the target model on the training dataset and the running task set based on the structural analysis results of the target model to obtain multiple rounds of forward inference sampling results; recording the activation count, activation duration, and joint activation probability of all neurons in the target model under different input modes layer by layer based on the multi-round forward inference sampling results; generating a neuron activity distribution matrix based on the activation count, activation duration, and joint activation probability; and determining the activity results of neurons in each layer of the target model based on the neuron activity distribution matrix.

[0007] In some embodiments, the step of performing neuron-level physical reorganization on the model weights of each layer in the target model based on the activity results to obtain physical reorganization results includes: determining the activation count, activation duration, and joint activation probability of each neuron in the target model based on the activity results; designating the weight blocks corresponding to neurons whose activation count is greater than or equal to a first preset count and whose activation duration fluctuation value is less than or equal to a preset fluctuation value as first neuron weight blocks; designating the weight blocks corresponding to neurons that are triggered in the target scene and whose activation count is less than or equal to a second preset count as second neuron weight blocks; forming a first weight set based on the first neuron weight blocks and forming a second weight set based on the second neuron weight blocks; mapping the weight blocks corresponding to adjacent neurons whose joint activation probability is greater than or equal to a preset threshold to adjacent physical storage addresses; and performing hierarchical storage configuration on the model weights of each layer in the target model based on the first weight set, the second weight set, and the adjacent physical storage addresses, and establishing a correspondence between neuron identifiers and weight storage locations to obtain the physical reorganization results.

[0008] In some embodiments, the step of performing hierarchical storage configuration on the model weights of each layer in the target model based on the first weight set, the second weight set, and the adjacent physical storage addresses, and establishing a correspondence between neuron identifiers and weight storage locations to obtain the physical reorganization result includes: deploying the first weight set to a first cache area located adjacent to the computing array, and recording the first weight storage location of each weight block in the first weight set in the first cache area; splitting the second weight set into multiple neuron micro-block units according to the neuron correspondence, and mapping the multiple neuron micro-block units to off-chip main memory according to the adjacent physical storage addresses, and recording the second weight storage location of each neuron micro-block unit in the off-chip main memory; configuring a neuron identifier for each neuron micro-block unit; and establishing a correspondence between the neuron identifier and weight storage location based on the first weight storage location, the second weight storage location, and the neuron identifier to obtain the physical reorganization result.

[0009] In some embodiments, obtaining the local computation result of the current layer during the inference calculation of the current layer of the target model based on the physical recombination result includes: performing matrix multiplication and addition calculations on the current layer through a computation array during the inference calculation of the current layer of the target model based on the physical recombination result; obtaining the local accumulation result output by the computation array if the matrix multiplication and addition calculation of the current layer has not ended; and performing truncation processing on the local accumulation result according to a preset sampling window to obtain the local computation result of the current layer.

[0010] In some embodiments, predicting the neuron activation state of the next layer based on the local computation results of the current layer and generating the neuron activation mask of the next layer includes: performing linear projection processing on the local computation results of the current layer to obtain the activation prediction value of each neuron in the next layer; generating a first state bit for the corresponding neuron when the activation prediction value is greater than or equal to a preset activation threshold; generating a second state bit for the corresponding neuron when the activation prediction value is less than the preset activation threshold; and performing bitmap encoding on the activation prediction state of each neuron in the next layer based on the first state bit and the second state bit to generate the neuron activation mask of the next layer; wherein each bit in the neuron activation mask corresponds to a neuron in the next layer, the first state bit is used to indicate that the corresponding neuron participates in the inference computation of the next layer, and the second state bit is used to indicate that the corresponding neuron is skipped in the inference computation of the next layer.

[0011] In some embodiments, determining the target weight data of the next layer based on the neuron activation mask of the next layer and the physical recombination result includes: determining the target neurons participating in inference computation in the next layer based on the neuron activation mask of the next layer; determining the prefetching method of the target neurons as block prefetching when the target neurons are continuously distributed in the neuron activation mask; determining the prefetching method of the target neurons as skip prefetching when the target neurons are not continuously distributed in the neuron activation mask; determining the weight storage location of the target neurons based on the physical recombination result; reading the weight data of the target neurons based on the prefetching method and the weight storage location; and rearranging the read weight data based on the inference computation order of the next layer to obtain the target weight data of the next layer.

[0012] In some embodiments, the gating reconstruction of the initial computing array based on the neuron activation mask of the next layer to obtain the target computing array includes: determining the hit neurons participating in inference computation and the missed neurons skipping inference computation in the next layer based on the level state of each mask bit in the neuron activation mask of the next layer; determining the hit operation clusters with the hit neurons and the missed operation clusters with the missed neurons in the initial computing array; generating a first gating parameter based on the hit operation clusters and generating a second gating parameter based on the missed operation clusters; and performing gating reconstruction on the initial computing array based on the first gating parameter and the second gating parameter to obtain the target computing array; wherein the first gating parameter is used to control the hit operation clusters to open local power supply paths, restore local clock signals, and conduct input / output switching paths, and the second gating parameter is used to control the missed operation clusters to cut off local power supply paths, maintain the current state of registers, and close data switching paths.

[0013] In some embodiments, the step of performing inference computation on the next layer of the target model based on the target weight data of the next layer and the target computing array includes: controlling the target computing array to perform inference computation on the next layer of the target model based on the target weight data of the next layer, and obtaining the inference result of the next layer; if the next layer is the output layer of the target model, determining the semantic feature encoding of the target model based on the inference result of the next layer; performing state recognition on the semantic feature encoding to obtain the current environment state label; determining the corresponding control template from a preset state mapping table based on the current environment state label; adjusting the control parameters in the control template based on real-time sensor feedback to generate a control parameter vector; controlling the actuator to adjust the execution state based on the control parameter vector, and triggering the target model to enter the next round of inference based on the execution feedback of the actuator.

[0014] Secondly, embodiments of this application provide an electronic device. The electronic device includes: a memory, a computing array, a controller, and a processor. The memory stores computer program instructions, which, when invoked by the processor, cause the processor to cooperate with the controller and the computing array to execute the model inference method described above.

[0015] The beneficial effects of the model inference method provided in this application are as follows: On the one hand, by combining the structural analysis of the target model and the activity of neurons, the model weights are physically reorganized at the neuron level, which makes the storage layout of the model weights more adaptable to the actual rules of neurons participating in inference calculations, providing a foundation for the rapid location and retrieval of subsequent weight data; on the other hand, by obtaining local calculation results during the current layer inference calculation process and predicting the activation state of neurons in the next layer in advance, the target weight data to be called can be determined before the start of the next layer inference calculation, and the calculation array can be gated and reconstructed, so that the weight preparation process and the calculation resource configuration process match the actual inference needs of the next layer, thereby reducing invalid weight reading and redundant calculations and improving the execution efficiency of model inference. Attached Figure Description

[0016] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0017] Figure 1 A schematic diagram of the model reasoning method provided in the embodiments of this application; Figure 2 A schematic diagram of a sub-process of step S110 of the model reasoning method provided in the embodiments of this application; Figure 3 A schematic diagram of a sub-process of step S120 of the model reasoning method provided in the embodiments of this application; Figure 4 A schematic diagram of a sub-process of step S126 of the model inference method provided in the embodiments of this application; Figure 5 A schematic diagram of a sub-process of step S130 of the model reasoning method provided in the embodiments of this application; Figure 6 A schematic diagram of a sub-process of step S140 of the model inference method provided in the embodiments of this application; Figure 7 A schematic diagram of the sub-process of step S150 of the model inference method provided in the embodiments of this application. Figure 1 ; Figure 8A schematic diagram of the sub-process of step S150 of the model inference method provided in the embodiments of this application. Figure 2 ; Figure 9 A schematic diagram of a sub-process of step S160 of the model reasoning method provided in the embodiments of this application; Figure 10 A schematic diagram of the architecture of the model inference device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0020] The technical solution of this application is described in detail below. The technical solution of this application can be applied to computing boxes, mobile robots, intelligent exoskeletons or other electronic devices to achieve functions such as large model inference acceleration and accurate weight data retrieval.

[0021] Figure 1 This is a schematic diagram illustrating the model reasoning method provided in an embodiment of this application. For example... Figure 1 As shown, the inference method of this model includes: S110. Based on the structural analysis results of the target model, determine the activity level of neurons in each layer of the target model; The "target model" refers to the neural network model that needs to be deployed on a hardware platform to perform inference computation. The target model can be a large language model, a visual language model, a convolutional neural network model, or other artificial intelligence models containing multiple layers of neurons.

[0022] "Structural analysis results" refer to the results obtained after analyzing the network hierarchical structure, neuron connectivity, weight distribution, input-output dependencies, and historical inference activation of the target model.

[0023] "Activity results" refer to statistical results used to characterize the frequency of computation and activation stability of neurons in each layer of the target model during the inference process.

[0024] In some embodiments, the network hierarchy, number of neurons in each layer, and connection relationships between neurons in the target model can be determined first based on the structural analysis results of the target model. Then, the activation status of neurons in each layer is statistically analyzed in conjunction with the inference performance of the target model under different input modes, and the activity result of neurons in each layer in the target model is determined based on the statistical results.

[0025] S120. Based on the activity results, perform neuron-level physical reorganization on the model weights of each layer in the target model to obtain the physical reorganization results; Here, "model weights" refers to the parameter data in the target model used for inference computation. Model weights can include weights of fully connected layers, attention layers, feedforward network layers, convolutional layers, etc.

[0026] "Physical Reorganization Result" refers to the storage layout result obtained after performing neuron-level physical reorganization on the model weights.

[0027] In some embodiments, the model weights of each layer in the target model can be re-partitioned and remapped based on the activity results of neurons in each layer. For example, the model weights corresponding to neurons with high activity can be centrally configured in a storage location that is easy to access quickly, while the model weights corresponding to neurons with low activity can be configured independently according to the neuron granularity, and a correspondence between each neuron and the corresponding model weight storage location can be established to obtain the physical recombination result.

[0028] S130. During the inference calculation of the current layer of the target model based on the physical recombination result, the local calculation result of the current layer is obtained; Here, "current layer" refers to the network layer that the target model is currently performing computations in the current inference cycle.

[0029] "Local computation results" refers to the partial intermediate results generated by the computation array before the current layer inference computation has been fully completed.

[0030] In some embodiments, when performing inference computation on the current layer of the target model based on the physical recombination result, the model weights corresponding to the current layer can be invoked, and the computation process of the current layer can be executed through a computation array. If the inference computation of the current layer is not yet complete, some intermediate results generated during the computation process can be obtained and used as local computation results of the current layer.

[0031] S140. Based on the local calculation results of the current layer, predict the neuron activation state of the next layer and generate the neuron activation mask of the next layer; Here, "next layer" refers to the network layer that follows the current layer in the inference sequence of the target model and is about to perform inference computation.

[0032] "Neuron activation state" refers to the state of each neuron in the next layer, indicating whether it will participate in effective computation during the upcoming inference process. Neuron activation state can include active and inactive states.

[0033] A "neuron activation mask" is a bitmap information used to represent the activation state of neurons in the next layer. Each bit in the neuron activation mask corresponds to a neuron in the next layer.

[0034] In some embodiments, the local computation results of the current layer can be input into the activation prediction module to predict the activation trend of each neuron in the next layer. When the prediction result indicates that a neuron may participate in the inference computation of the next layer, the neuron can be marked as active; when the prediction result indicates that a neuron does not participate in the inference computation of the next layer, the neuron can be marked as inactive. Subsequently, according to the arrangement order of the neurons in the next layer, the activation state of each neuron is encoded to generate the neuron activation mask for the next layer.

[0035] S150. Based on the neuron activation mask of the next layer and the physical recombination result, determine the target weight data of the next layer, and perform gated reconstruction on the initial computing array based on the neuron activation mask of the next layer to obtain the target computing array. Among them, "target weight data" refers to the weight data that needs to be actually read and participate in the calculation in the next layer of inference computation, which is determined based on the activation mask of the next layer of neurons.

[0036] The “initial computation array” refers to the computation array prior to the gating reconstruction.

[0037] "Target computation array" refers to a computation array that, after gating reconstruction, matches the activation state of neurons in the next layer.

[0038] In some embodiments, the neurons that need to participate in inference computation in the next layer can be determined based on the neuron activation mask of the next layer, and the corresponding weight storage location can be determined in combination with the physical recombination result, thereby reading and organizing the target weight data of the next layer. At the same time, the computation regions that need to be turned on and those that need to be turned off or frozen in the initial computation array can be determined based on the neuron activation mask of the next layer, and gating reconstruction can be performed on the initial computation array to obtain a target computation array that matches the activation state of the neurons in the next layer.

[0039] S160. Based on the target weight data of the next layer and the target computing array, perform inference calculation on the next layer of the target model.

[0040] In some embodiments, at the start of the next layer inference computation, the already determined target weight data for the next layer can be invoked, and the computational regions in the target computation array that are in an active state can be controlled to perform the corresponding inference computation for the next layer. Since both the target weight data and the target computation array are determined based on the activation state of the neurons in the next layer, the next layer inference computation can be performed only on the neurons that need to participate in the computation, thereby reducing invalid weight invocations and redundant computations.

[0041] In this embodiment, on the one hand, by combining the structural analysis of the target model and the activity of neurons, the model weights are physically reorganized at the neuron level. This makes the storage layout of the model weights more adaptable to the actual rules of neurons participating in inference calculations, providing a foundation for the rapid location and retrieval of subsequent weight data. On the other hand, by obtaining local calculation results during the current layer's inference calculation process and predicting the activation state of neurons in the next layer in advance, the target weight data to be called can be determined before the next layer's inference calculation begins. The calculation array is then gated and reconstructed, so that the weight preparation process and the calculation resource configuration process match the actual inference needs of the next layer, thereby reducing invalid weight readings and redundant calculations and improving the execution efficiency of model inference.

[0042] In some embodiments, such as Figure 2 As shown, S110 specifically includes: S111. Based on the structural analysis results of the target model, perform multiple rounds of forward inference sampling on the target model on the training dataset and the running task set to obtain the results of multiple rounds of forward inference sampling. The "training dataset" refers to the collection of data used to train, validate, or evaluate the performance of the target model. Training datasets can include text data, image data, speech data, multimodal data, etc.

[0043] "Running task set" refers to a collection of typical task samples that the target model may handle in a real-world deployment scenario. The running task set may include mobile robot environmental perception tasks, intelligent exoskeleton motion control tasks, edge semantic recognition tasks, etc.

[0044] "Multi-round forward inference sampling results" refers to the statistical results generated by the target model during multiple forward inference processes.

[0045] In some embodiments, the network layers, number of neurons in each layer, and inter-layer connections of the target model can be determined based on the structural analysis results of the target model. Then, multiple samples from the training dataset and the running task set are sequentially input into the target model to enable the target model to perform multiple rounds of forward inference and record the activation status of neurons in each layer during each round of inference, thereby obtaining the sampling results of multiple rounds of forward inference.

[0046] S112. Based on the multi-round forward inference sampling results, record the activation count, activation duration, and joint activation probability of all neurons in the target model under different input modes layer by layer. "Input mode" refers to the scene type, task type, or data feature type corresponding to the data input to the target model. For example, input mode can include scene modes such as flat ground, slippery ground, obstacles, steps, and slopes, as well as task modes such as text question answering, image recognition, and semantic understanding.

[0047] "Activation count" refers to the number of times a neuron meets the preset activation conditions during multiple rounds of forward inference sampling.

[0048] "Activation duration" refers to the length of time a neuron remains in an activated state during a continuous inference cycle.

[0049] "Joint activation probability" refers to the probability that two or more neurons are simultaneously activated in the same input pattern.

[0050] In some embodiments, the output state of each neuron in the target model can be detected layer by layer based on the sampling results of multiple rounds of forward inference, and the activation count and activation duration of each neuron under different input modes can be counted. Simultaneously, the number of times adjacent neurons are simultaneously activated in the same input sample can be counted, and the joint activation probability between adjacent neurons can be determined based on the ratio of this simultaneous activation count to the total number of activations.

[0051] S113. Generate a neuron activity distribution matrix based on the number of activations, the activation duration, and the joint activation probability; The “neuron activity distribution matrix” refers to a matrix data structure used to characterize the activity level of neurons in each layer of the target model and their related activation relationships.

[0052] In some embodiments, each neuron in the target model can be used as a statistical object, and the activation count, activation duration, and joint activation probability between adjacent neurons can be written into the matrix elements to generate a neuron activity distribution matrix corresponding to each layer of the target model.

[0053] S114. Based on the neuron activity distribution matrix, determine the activity results of each layer of neurons in the target model.

[0054] In some embodiments, neurons that frequently participate in inference calculations over a long period and have a stable activation state can be identified as high-activity neurons based on the number of activations and activation duration in the neuron activity distribution matrix; neurons that are triggered only under specific input patterns can be identified as conditionally active neurons; and neurons that have fewer activations and shorter activation durations can be identified as low-activity neurons. Furthermore, combinations of neurons with strong activation relationships can be determined based on joint activation probabilities.

[0055] In this embodiment, by performing multiple rounds of forward inference sampling on the target model on the training dataset and the running task set, the neuron activation status of the target model under different input modes can be obtained, thereby providing a data foundation for subsequent neuron activity statistics.

[0056] In some embodiments, such as Figure 3 As shown, S120 specifically includes: S121. Based on the activity results, determine the number of activations, activation duration, and joint activation probability of each neuron in the target model; In some embodiments, the activation count and activation duration of each neuron can be extracted based on the activity results of each layer of neurons, and the joint activation probability between adjacent neurons can be calculated, thereby providing a basis for subsequent division of different types of neuron weight blocks.

[0057] S122. The weight block corresponding to the neuron whose activation count is greater than or equal to the first preset count and whose fluctuation value of the activation duration period is less than or equal to the preset fluctuation value is taken as the first neuron weight block. The "first preset number of times" refers to the threshold number of times used to determine whether a neuron belongs to a high-frequency activated neuron.

[0058] "Activation duration fluctuation value" refers to the degree of change in the activation duration of a neuron during different inference sampling processes.

[0059] The “first neuron weight block” refers to the weight data block corresponding to a neuron that is frequently activated and has a stable activation pattern.

[0060] In some embodiments, when the number of activations of a neuron is greater than or equal to a first preset number, and the fluctuation value of its activation duration is less than or equal to a preset fluctuation value, the neuron can be determined to be a high-frequency stable activation neuron, and the weight block corresponding to the neuron is taken as the first neuron weight block.

[0061] S123. The weight block corresponding to the neuron that is triggered in the target scene and whose activation count is less than or equal to the second preset count is used as the second neuron weight block. The "target scenario" refers to a specific input scenario or task scenario that the target model may encounter during actual operation.

[0062] "Second preset number of times" refers to the threshold number of times used to determine whether a neuron belongs to a low-frequency triggering neuron.

[0063] "Second neuron weight block" refers to the weight data block corresponding to neurons that are activated at low frequency or only triggered under specific input scenarios.

[0064] In some embodiments, when a neuron is triggered only in the target scene and its activation count is less than or equal to a second preset count, it can be determined that the neuron belongs to a low-frequency scene triggering neuron, and the weight block corresponding to the neuron is used as the weight block of the second neuron.

[0065] S124. A first weight set is formed based on the first neuron weight block, and a second weight set is formed based on the second neuron weight block; Here, "first weight set" refers to the weight set composed of multiple first neuron weight blocks.

[0066] The “second weight set” refers to a weight set composed of multiple weight blocks of second neurons.

[0067] In some embodiments, multiple weight blocks identified as first neuron weight blocks in the same layer can be aggregated to form a first weight set, and multiple weight blocks identified as second neuron weight blocks can be aggregated to form a second weight set.

[0068] S125. Map the weight blocks corresponding to adjacent neurons whose joint activation probability is greater than or equal to a preset threshold to adjacent physical storage addresses. The "preset threshold" refers to the probability threshold used to determine whether adjacent neurons have a strong correlation activation relationship.

[0069] "Adjacent physical storage addresses" refer to storage locations that are physically adjacent or have consecutive addresses in the storage space.

[0070] In some embodiments, when the joint activation probability of adjacent neurons is greater than or equal to a preset threshold, a strong correlation activation relationship can be determined between the adjacent neurons, and their corresponding weight blocks can be mapped to adjacent physical storage addresses. Thus, during subsequent inference, when the weight block corresponding to one neuron is read, the weight blocks of neurons strongly correlated with it can also be quickly located and read.

[0071] S126. Based on the first weight set, the second weight set, and the adjacent physical storage addresses, perform hierarchical storage configuration on the model weights of each layer in the target model, and establish the correspondence between neuron identifiers and weight storage locations to obtain the physical recombination result.

[0072] Among them, "neuron identifier" refers to the information used to uniquely identify neurons in the target model.

[0073] "Weight storage location" refers to the location of the weight block corresponding to the neuron in the physical storage space.

[0074] In some embodiments, the model weights of each layer in the target model can be configured for hierarchical storage based on a first weight set, a second weight set, and adjacent physical storage addresses. For example, the first weight set can be configured in a storage area that is easy to access quickly, the second weight set can be configured in a storage area that can be accessed independently at the neuron level, and weight blocks with adjacent physical storage address relationships can be configured according to their corresponding addresses. Subsequently, a correspondence between neuron identifiers and weight storage locations is established, enabling each neuron to locate the corresponding storage location of its model weights, thereby obtaining the physical reconstruction result.

[0075] In this embodiment, by determining the activation count, activation duration, and joint activation probability between adjacent neurons based on the activity results, neurons in the target model can be classified and identified, thereby distinguishing high-frequency stable activation neurons, low-frequency scene-triggered neurons, and neurons with strong correlation activation relationships.

[0076] In some embodiments, such as Figure 4 As shown, S126 specifically includes: S1261. Deploy the first weight set to a first cache area located adjacent to the computing array, and record the first weight storage location of each weight block in the first weight set in the first cache area; The "first cache area" refers to the on-chip cache area located near the computing array and used to store frequently called weight data.

[0077] "First weight storage location" refers to the physical storage address of the weight block in the first weight set in the first cache area.

[0078] In some embodiments, neuron weight blocks with high-frequency activation and stable activation patterns in the first weight set can be deployed to the first cache, and the physical storage address of each weight block in the first cache can be recorded so that the corresponding weight data can be directly called from the first cache during subsequent inference.

[0079] S1262. The second weight set is divided into multiple neuron micro-block units according to the neuron correspondence, and the multiple neuron micro-block units are mapped to off-chip main memory according to the adjacent physical storage addresses. The storage location of the second weight of each neuron micro-block unit in the off-chip main memory is recorded. Among them, "neuron correspondence" refers to the correspondence between the weight data in the second weight set and each neuron in the target model.

[0080] "Neuron micro-block unit" refers to a weight data unit that can be independently addressed and read, obtained by splitting it from the second weight set according to the neuron correspondence.

[0081] "Off-chip main memory" refers to memory located outside the computing chip or computing array, used to store low-frequency weight data or large-capacity weight data.

[0082] "Second weight storage location" refers to the physical storage address of the neuronal micro-block unit in off-chip main memory.

[0083] In some embodiments, the second weight set can be divided into multiple neuronal micro-block units based on the correspondence between each weight block in the second weight set and neurons. Subsequently, based on the joint activation relationship between adjacent neurons, neuronal micro-block units with strong correlation activation relationships can be mapped to adjacent physical storage addresses, and the physical storage address of each neuronal micro-block unit in off-chip main memory can be recorded.

[0084] S1263. Configure a neuron identifier for each neuron micro-block unit; In some embodiments, each neuron microblock unit can be configured with a corresponding neuron identifier, so that each neuron microblock unit can establish a correspondence with a specific neuron or neuron group in the target model.

[0085] S1264. Based on the first weight storage location, the second weight storage location, and the neuron identifier, establish a correspondence between the neuron identifier and the weight storage location to obtain the physical recombination result.

[0086] In some embodiments, a correspondence between neuron identifiers and weight storage locations can be established based on the first weight storage location of each weight block in the first weight set, the second weight storage location of each neuron microblock unit in the second weight set, and the neuron identifier corresponding to each neuron microblock unit, to obtain the physical recombination result.

[0087] In this embodiment, on the one hand, by deploying the first weight set to the first cache area close to the computing array, the calling speed of high-frequency neuron weights can be improved; by splitting the second weight set into multiple neuron micro-block units and mapping them to off-chip main memory, independent addressing access of low-frequency or scene-triggered neuron weights can be achieved; on the other hand, by configuring neuron identifiers for neuron micro-block units and establishing the correspondence between neuron identifiers and weight storage locations, physical recombination results that support precise neuron-level positioning and non-continuous reading can be obtained.

[0088] In some embodiments, such as Figure 5 As shown, S130 specifically includes: S131. During the current layer inference calculation of the target model based on the physical recombination result, matrix multiplication and addition calculations are performed on the current layer through a computing array; In some embodiments, the model weights required for the current layer can be called according to the physical recombination result, and matrix multiplication and addition calculations can be performed on the input data of the current layer and the corresponding model weights through a computing array to generate the intermediate accumulation result of the current layer.

[0089] S132. If the matrix multiplication and addition calculation in the current layer has not ended, obtain the local accumulation result output by the calculation array; Among them, "local accumulation result" refers to the accumulation result that the calculation array has generated in part of the calculation cycle during the matrix multiplication and addition calculation of the current layer.

[0090] In some embodiments, before the matrix multiplication and addition calculations of the current layer are completed, the local accumulation result can be obtained from the local registers of the computation array, the accumulated output terminal, or the intermediate result buffer. Since the local accumulation result is obtained before the complete output result of the current layer is generated, it can be used to predict the activation state of the neurons in the next layer in advance.

[0091] S133. Perform truncation processing on the local accumulation result according to the preset sampling window to obtain the local calculation result of the current layer.

[0092] Among them, "preset sampling window" refers to the sampling rules or sampling range used to select a portion of data from the local accumulation results.

[0093] "Local calculation results" refers to intermediate calculation data obtained after performing truncation processing on local cumulative results.

[0094] In some embodiments, a local accumulation result can be extracted once every preset number of calculation cycles, or the accumulation result corresponding to a specified neuron number range or a specified matrix block position in the current layer can be extracted to obtain the local calculation result of the current layer.

[0095] In this embodiment, by performing matrix multiplication and addition during the current layer inference calculation and obtaining the local accumulation result of the calculation array output before the current layer matrix multiplication and addition calculation is completed, the intermediate calculation state of the current layer can be obtained in advance. Subsequently, by truncating the local accumulation result according to the preset sampling window, the local calculation result used to predict the activation state of the next layer of neurons can be obtained, thereby providing a data basis for the subsequent generation of the activation mask of the next layer of neurons.

[0096] In some embodiments, such as Figure 6 As shown, S140 specifically includes: S141. Perform linear projection processing on the local calculation results of the current layer to obtain the activation prediction value of each neuron in the next layer; Among them, "activation prediction value" refers to the predicted value used to characterize the probability that each neuron in the next layer will be activated in subsequent inference calculations.

[0097] In some embodiments, the local computation results of the current layer can be input into the feedforward bypass prediction channel, and linear projection processing can be performed on the local computation results using a pre-configured low-precision prediction coefficient matrix to obtain the activation prediction values ​​corresponding to each neuron in the next layer. Since this process is executed in parallel with the main inference computation of the current layer, the activation prediction results can be obtained in advance before the formal computation of the next layer.

[0098] S142. If the activation prediction value is greater than or equal to the preset activation threshold, generate the first state bit of the corresponding neuron. Among them, the "preset activation threshold" refers to the comparison threshold used to determine whether a neuron may participate in the next layer of inference computation.

[0099] The "first state bit" refers to the bit value used to indicate that the corresponding neuron is in a predicted activation state. The first state bit can indicate that the neuron will participate in effective computation in the next layer of inference computation.

[0100] In some embodiments, the activation prediction value corresponding to each neuron can be compared with a preset activation threshold. When the activation prediction value of a neuron is greater than or equal to the preset activation threshold, it indicates that the neuron has a high probability of being activated in the next layer of inference computation, and at this time, the first state bit corresponding to the neuron is generated.

[0101] S143. If the activation prediction value is less than the preset activation threshold, generate the second state position of the corresponding neuron. The "second state bit" refers to the bit value used to indicate that the corresponding neuron is in a predicted inactive state. The second state bit can indicate that the neuron is skipped or temporarily not involved in the effective calculation in the next layer of inference computation.

[0102] In some embodiments, when the activation prediction value corresponding to a neuron is less than a preset activation threshold, it indicates that the probability of the neuron being activated in the next layer of inference computation is low, and at this time, the second state bit corresponding to the neuron is generated.

[0103] S144. Based on the first state bit and the second state bit, perform bitmap encoding on the activation prediction state of each neuron in the next layer to generate the neuron activation mask of the next layer.

[0104] In some embodiments, the first and second state bits corresponding to each neuron can be combined to form a neuron activation bitmap according to the numbering order of the neurons in the next layer, and this neuron activation bitmap can be used as the neuron activation mask for the next layer. For example, when the next layer includes multiple neurons, if the activation prediction value of some neurons is greater than or equal to a preset activation threshold, the corresponding position is written with the first state bit; if the activation prediction value of the remaining neurons is less than the preset activation threshold, the corresponding position is written with the second state bit.

[0105] In this embodiment, by performing linear projection processing on the local calculation results of the current layer, the activation prediction values ​​of each neuron in the next layer can be obtained in advance before the inference calculation of the current layer is finished. Subsequently, by comparing the activation prediction values ​​with a preset activation threshold and performing bitmap encoding on the comparison results, the neuron activation mask of the next layer can be generated, thereby determining the target neurons participating in the inference calculation in the next layer in advance.

[0106] In some embodiments, such as Figure 7 As shown, S150 specifically includes: S211. Based on the neuron activation mask of the next layer, determine the target neurons in the next layer that participate in inference calculation; Here, "target neuron" refers to the neuron that is determined based on the neuron activation mask of the next layer and needs to participate in effective computation in the inference computation of the next layer.

[0107] In some embodiments, the activation mask of the next layer of neurons can be parsed bit by bit, and the neurons corresponding to the mask bits that are in a valid state can be identified as target neurons, thereby filtering out the neurons in the next layer that actually need to participate in inference calculation.

[0108] S212. When the target neurons are continuously distributed in the neuron activation mask, the prefetching method of the target neurons is determined to be block prefetching; In some embodiments, when multiple valid bits in the neuron activation mask are identified to be continuously distributed, it can be determined that the target neurons corresponding to these valid bits have continuous access requirements in the next layer of inference computation, and the prefetching method of the target neurons is determined to be block prefetching, which can improve the reading efficiency of continuous weight data.

[0109] S213. When the target neuron is not distributed discontinuously in the neuron activation mask, determine that the prefetching method of the target neuron is skip prefetching; In some embodiments, when the effective bits in the neuron activation mask are found to be discretely distributed, it can be determined that the weight data corresponding to the target neuron is not suitable for reading as a whole in a continuous block, and the prefetching method of the target neuron is determined to be a skip prefetching, which can avoid reading the weight data corresponding to invalid neurons, reduce off-chip memory access overhead and bus bandwidth occupation.

[0110] S214. Based on the physical recombination results, determine the weight storage location of the target neuron; In some embodiments, the weight storage location corresponding to the neuron identifier of the target neuron can be found in the physical reconstruction result based on the neuron identifier of the target neuron. If the target neuron is a high-frequency activated neuron, its weight data can be determined to be located in the first cache area; if the target neuron is a low-frequency or scene-triggered neuron, its weight data can be determined to be located in the neuron micro-block unit in the off-chip main memory.

[0111] S215. Based on the prefetching method and the weight storage location, read the weight data of the target neuron; Here, "weight data" refers to the model parameter data that the target neuron needs to participate in matrix multiplication and addition calculations in the next layer of inference computation.

[0112] In some embodiments, when the prefetching method of the target neuron is block prefetching, weight data reading within a continuous address range can be performed according to the continuous weight storage location corresponding to the target neuron; when the prefetching method of the target neuron is skip prefetching, non-continuous skip weight data reading can be performed according to the discrete weight storage location corresponding to the target neuron.

[0113] S216. Based on the reasoning and calculation order of the next layer, the read weight data is rearranged to obtain the target weight data of the next layer.

[0114] In some embodiments, the read target neuron weight data can be written into an on-chip prefetch buffer, and the weight data can be rearranged according to the next layer inference computation order so that the rearranged weight data corresponds to the computation area in the target computation array.

[0115] In this embodiment, by selecting block prefetching or skip prefetching based on the distribution of the target neurons in the neuron activation mask, the weight reading method can be matched with the neuron activation distribution. Subsequently, by combining the physical recombination results to determine the weight storage location of the target neurons, and rearranging the read weight data according to the next layer inference calculation order, target weight data suitable for direct call in the next layer inference calculation can be obtained, thereby reducing invalid weight reading and improving inference calculation efficiency.

[0116] In some embodiments, such as Figure 8 As shown, S150 specifically includes: S221. Based on the level state of each mask bit in the neuron activation mask of the next layer, determine the hit neurons that participate in the inference calculation and the miss neurons that skip the inference calculation in the next layer. Here, "mask bits" refers to the bit information in the neuron activation mask used to represent the predicted activation state of the corresponding neuron.

[0117] "Level state" refers to the logic state corresponding to the mask bit. Level states can include high-level states and low-level states. A high-level state can be used to indicate that the corresponding neuron participates in the next layer of inference computation, while a low-level state can be used to indicate that the corresponding neuron is skipped in the next layer of inference computation.

[0118] "Hit neuron" refers to the neuron that, based on the neuron activation mask, needs to participate in effective computation in the next layer of inference computation.

[0119] "Missed neurons" refer to neurons that, based on their activation masks, do not participate in effective computation in the next layer of inference or can be temporarily skipped.

[0120] In some embodiments, when a mask bit is high, the neuron corresponding to that mask bit is identified as a hit neuron; when a mask bit is low, the neuron corresponding to that mask bit is identified as a miss neuron. This allows for the differentiation between neurons in the next layer that require computing resources and those that can have their computing resources disabled or frozen.

[0121] S222. Determine the hit operation clusters with the hit neurons and the miss operation clusters with the miss neurons in the initial computing array; Here, "computation cluster" refers to the local hardware region in the computation array used to perform neuron-related computations.

[0122] "Hit cluster" refers to the cluster of operations in the initial computation array that corresponds to the hit neuron.

[0123] "Missed operation clusters" refer to the operation clusters in the initial computation array that correspond to the missed neurons.

[0124] In some embodiments, the hit operation cluster corresponding to a hit neuron can be determined based on the mapping relationship between neurons and operation clusters, and the miss operation cluster corresponding to a miss neuron can be determined. For example, when neurons numbered N1 to N10 in the next layer are determined to be hit neurons, the operation clusters corresponding to N1 to N10 can be determined as hit operation clusters; when neurons numbered N11 to N20 are determined to be miss neurons, the operation clusters corresponding to N11 to N20 can be determined as miss operation clusters.

[0125] S223. Generate a first gating parameter based on the hit operation cluster, and generate a second gating parameter based on the missed operation cluster; The "first gating parameter" refers to the control parameter used to control the hit arithmetic cluster to enter the working state. The first gating parameter is used to control the hit arithmetic cluster to open the local power supply path, restore the local clock signal, and conduct the input / output switching path.

[0126] The "second gating parameter" refers to the control parameter used to control the missed arithmetic clusters to enter a low-power state or a frozen state. The second gating parameter is used to control the missed arithmetic clusters to cut off the local power supply path, maintain the current state of the registers, and close the data exchange path.

[0127] In some embodiments, a first gating control parameter can be generated based on the location and number of hit arithmetic clusters and the hardware resource mapping relationship to control the hit arithmetic clusters to complete the power supply, clock, and data path configuration before the next layer of inference computation begins. Simultaneously, a second gating control parameter can be generated based on the location and number of miss arithmetic clusters to control the miss arithmetic clusters to shut down invalid hardware activities and reduce unnecessary power consumption.

[0128] S224. Based on the first gating parameter and the second gating parameter, perform gating reconstruction on the initial computing array to obtain the target computing array.

[0129] In some embodiments, the initial computing array can be first reconfigured by gating based on the positions of the hit and miss clusters within the initial computing array to obtain the target computing array. The gating reconfiguration includes: configuring the local power supply path, local clock signal, and input / output exchange path corresponding to the hit clusters in the target computing array to be enabled based on a first gating parameter, thus enabling them to perform the next layer of inference computation; and configuring the local power supply path, register state, and data exchange path corresponding to the miss clusters in the target computing array to be disabled or kept in a closed state based on a second gating parameter, to reduce invalid computation and redundant power consumption.

[0130] In this embodiment, by performing gating reconstruction on the initial computing array based on the first gating parameter and the second gating parameter, a target computing array that matches the activation state of the next layer of neurons can be obtained. The target computing clusters are put into working state, while the untargeted computing clusters are put into low-power state or frozen state, thereby reducing redundant power consumption.

[0131] In some embodiments, such as Figure 9 As shown, S160 specifically includes: S161. Based on the target weight data of the next layer, control the target computing array to perform inference calculations on the next layer of the target model to obtain the inference result of the next layer; Among them, "the next layer inference result" refers to the calculation result obtained by the target computing array after performing inference calculation on the next layer of the target model based on the target weight data.

[0132] In some embodiments, when the target model enters the next layer of inference computation, the target weight data has been prefetched and organized into a data format suitable for use by the target computation array. At this time, the operational clusters in the target computation array can be controlled to call the target weight data and perform inference computation on the input data of the next layer, thereby obtaining the inference result of the next layer.

[0133] S162. If the next layer is the output layer of the target model, determine the semantic feature encoding of the target model based on the inference result of the next layer; The "output layer" refers to the network layer in the target model that is used to output the final inference result.

[0134] "Semantic feature encoding" refers to the feature encoding generated based on the inference results of the target model's output layer, used to represent the semantic meaning of the current input data.

[0135] In some embodiments, when the next layer is the output layer of the target model, feature parsing or encoding conversion can be performed on the inference results of the next layer to obtain the semantic feature encoding corresponding to the target model. For example, in mobile robot or intelligent exoskeleton scenarios, semantic feature encoding can be used to represent environmental semantic states such as flat ground, slippery ground, obstacles, steps, and slopes.

[0136] S163. Perform state recognition on the semantic feature encoding to obtain the current environment state label; Among them, "current environment status label" refers to the environment status identifier identified based on semantic feature encoding.

[0137] In some embodiments, semantic feature encoding can be input into a hardware semantic mapping control module, which then performs state recognition on the semantic feature encoding. When the semantic feature encoding matches a preset environmental state feature, a corresponding current environmental state label can be generated. For example, when the semantic feature encoding indicates a decrease in ground friction or the presence of slippery reflective features in the visual area, a slippery ground label can be identified.

[0138] S164. Based on the current environment status label, determine the corresponding control template from the preset status mapping table; The "preset status mapping table" refers to a pre-configured data table used to represent the correspondence between environmental status labels and control templates.

[0139] "Control template" refers to the set of initial control parameters corresponding to the current environment status label.

[0140] In some embodiments, the hardware semantic mapping control module can query a preset state mapping table based on the current environmental state label to determine a control template that matches the current environmental state label. For example, when the current environmental state label is a slippery ground label, the corresponding slippery ground control template can be determined from the preset state mapping table. This control template may include joint damping enhancement parameters, driving torque limiting parameters, and gait stability gain parameters.

[0141] S165. Based on real-time sensor feedback, adjust the control parameters in the control template to generate a control parameter vector; "Real-time sensing feedback" refers to the state data collected and fed back by sensors in real time. Real-time sensing feedback can include joint angles, joint velocities, plantar pressure, posture angles, acceleration, etc.

[0142] "Control parameters" refer to the parameters used in the control template to control the operating state of the actuator. Control parameters may include damping parameters, torque limiting parameters, speed limiting parameters, attitude compensation parameters, etc.

[0143] "Control parameter vector" refers to the set of parameters generated after adjusting the control parameters in the control template.

[0144] In some embodiments, when the current environment status label is a slippery ground label, if the real-time sensing feedback shows that the foot pressure fluctuation is large or the posture deviation is increased, the gait stability gain can be further increased and the upper limit of the driving torque can be reduced, thereby generating a control parameter vector that adapts to the current execution state.

[0145] S166. Based on the control parameter vector, control the actuator to adjust the execution state, and based on the execution feedback of the actuator, trigger the target model to enter the next round of inference.

[0146] An "actuator" refers to a hardware component used to perform actions or adjust physical states according to control parameter vectors. Actuators can include motors, joint actuators, robotic arm actuators, mobile robot chassis actuators, intelligent exoskeleton joint actuators, etc.

[0147] "Execution state" refers to the working state of an actuator under the action of control parameters. Execution state can include motor torque state, joint damping state, attitude adjustment state, etc.

[0148] "Execution feedback" refers to the feedback information generated and returned by the actuator after performing a control action. Execution feedback can include joint motion results, posture change results, torque output results, etc.

[0149] In some embodiments, in intelligent exoskeleton applications, the working state of the actuator can be adjusted based on control parameter vectors, such as increasing joint damping, limiting driving torque, and increasing gait stability gain. After the actuator completes the state adjustment, the joint motion results, torque output results, or posture change results can be used as execution feedback and re-inputted into the front-end perception module to trigger the target model to enter the next round of inference, thereby forming a closed-loop control between perception, inference, and execution.

[0150] In this embodiment, by determining the corresponding control template from the preset state mapping table based on the current environment state label and generating a control parameter vector in combination with real-time sensor feedback, the model inference results can be directly converted into control parameters that can be called by the actuator, thereby reducing the latency introduced by general operating system scheduling and software protocol stack, and improving the response speed and control stability of electronic devices in real-time control scenarios.

[0151] Figure 10 This is a schematic diagram of the architecture of the model inference device provided in an embodiment of this application. Figure 10As shown, the model inference device 200 includes: a first determining module 210, a recombination module 220, an acquisition module 230, a prediction module 240, a second determining module 250, and an inference module 260, wherein: The first determining module 210 is used to determine the activity level of each layer of neurons in the target model based on the structural analysis results of the target model. Reorganization module 220 is used to perform neuron-level physical reorganization of the model weights of each layer in the target model based on the activity result, and obtain physical reorganization result; The acquisition module 230 is used to acquire the local calculation results of the current layer during the inference calculation of the current layer of the target model based on the physical recombination results; The prediction module 240 is used to predict the neuron activation state of the next layer based on the local calculation results of the current layer, and generate the neuron activation mask of the next layer. The second determining module 250 is used to determine the target weight data of the next layer based on the neuron activation mask of the next layer and the physical recombination result, and to perform gated reconstruction on the initial computing array based on the neuron activation mask of the next layer to obtain the target computing array. The inference module 260 is used to perform inference calculations on the next layer of the target model based on the target weight data of the next layer and the target calculation array.

[0152] In some embodiments, the first determining module 210 is specifically configured to: perform multiple rounds of forward inference sampling on the target model on the training dataset and the running task set based on the structural analysis results of the target model, to obtain multiple rounds of forward inference sampling results; based on the multiple rounds of forward inference sampling results, record the activation count, activation duration, and joint activation probability of all neurons in the target model under different input modes layer by layer; generate a neuron activity distribution matrix based on the activation count, activation duration, and joint activation probability; and determine the activity result of each layer of neurons in the target model based on the neuron activity distribution matrix.

[0153] In some embodiments, the recombination module 220 includes: a first determining unit, a first comparing unit, a second comparing unit, a generating unit, a mapping unit, and a configuring unit, wherein: the first determining unit is used to determine the activation count, activation duration, and joint activation probability of each neuron in the target model based on the activity result; the first comparing unit is used to take the weight blocks corresponding to neurons whose activation count is greater than or equal to a first preset count and whose fluctuation value of the activation duration is less than or equal to a preset fluctuation value as first neuron weight blocks; the second comparing unit is used to take the weight blocks corresponding to neurons that are triggered in the target scene and whose activation count is less than or equal to a second preset count as second neuron weight blocks; the generating unit is used to form a first weight set based on the first neuron weight blocks and a second weight set based on the second neuron weight blocks; the mapping unit is used to map the weight blocks corresponding to adjacent neurons whose joint activation probability is greater than or equal to a preset threshold to adjacent physical storage addresses; the configuring unit is used to perform hierarchical storage configuration on the model weights of each layer in the target model based on the first weight set, the second weight set, and the adjacent physical storage addresses, and establish a correspondence between neuron identifiers and weight storage locations to obtain the physical recombination result.

[0154] In some embodiments, the configuration unit is specifically configured to: deploy the first weight set to a first cache area located adjacent to the computing array, and record the first weight storage location of each weight block in the first weight set in the first cache area; split the second weight set into multiple neuron micro-block units according to the neuron correspondence, and map the multiple neuron micro-block units to off-chip main memory according to the adjacent physical storage addresses, and record the second weight storage location of each neuron micro-block unit in the off-chip main memory; configure a neuron identifier for each neuron micro-block unit; and establish a correspondence between the neuron identifier and the weight storage location based on the first weight storage location, the second weight storage location, and the neuron identifier to obtain the physical recombination result.

[0155] In some embodiments, the acquisition module 230 is specifically used to: perform matrix multiplication and addition calculations on the current layer of the target model through a computing array during the inference calculation of the current layer based on the physical recombination result; if the matrix multiplication and addition calculation of the current layer has not ended, acquire the local accumulation result output by the computing array; and perform truncation processing on the local accumulation result according to a preset sampling window to obtain the local calculation result of the current layer.

[0156] In some embodiments, the prediction module 240 is specifically configured to: perform linear projection processing on the local calculation results of the current layer to obtain the activation prediction value of each neuron in the next layer; generate a first state bit for the corresponding neuron when the activation prediction value is greater than or equal to a preset activation threshold; generate a second state bit for the corresponding neuron when the activation prediction value is less than the preset activation threshold; and perform bitmap encoding on the activation prediction state of each neuron in the next layer based on the first state bit and the second state bit to generate a neuron activation mask for the next layer; wherein each bit in the neuron activation mask corresponds to a neuron in the next layer, the first state bit is used to indicate that the corresponding neuron participates in the inference calculation of the next layer, and the second state bit is used to indicate that the corresponding neuron is skipped in the inference calculation of the next layer.

[0157] In some embodiments, the second determining module 250 is specifically configured to: determine the target neurons participating in inference computation in the next layer based on the neuron activation mask of the next layer; determine the prefetching method of the target neurons as block prefetching when the target neurons are continuously distributed in the neuron activation mask; determine the prefetching method of the target neurons as skip prefetching when the target neurons are not continuously distributed in the neuron activation mask; determine the weight storage location of the target neurons based on the physical recombination result; read the weight data of the target neurons based on the prefetching method and the weight storage location; and rearrange the read weight data based on the inference computation order of the next layer to obtain the target weight data of the next layer.

[0158] In some embodiments, the second determining module 250 is specifically configured to: determine the hit neurons participating in inference computation and the miss neurons skipping inference computation in the next layer based on the level state of each mask bit in the neuron activation mask of the next layer; determine the hit operation clusters with the hit neurons and the miss operation clusters with the miss neurons in the initial computing array; generate a first gating parameter based on the hit operation clusters and generate a second gating parameter based on the miss operation clusters; and perform gating reconstruction on the initial computing array based on the first gating parameter and the second gating parameter to obtain the target computing array; wherein, the first gating parameter is used to control the hit operation clusters to open the local power supply path, restore the local clock signal and conduct the input / output switching path, and the second gating parameter is used to control the miss operation clusters to cut off the local power supply path, maintain the current state of the registers and close the data switching path.

[0159] In some embodiments, the inference module 260 is specifically configured to: control the target computing array to perform inference calculations on the next layer of the target model based on the target weight data of the next layer, and obtain the inference result of the next layer; if the next layer is the output layer of the target model, determine the semantic feature encoding of the target model based on the inference result of the next layer; perform state recognition on the semantic feature encoding to obtain the current environment state label; determine the corresponding control template from a preset state mapping table based on the current environment state label; adjust the control parameters in the control template based on real-time sensor feedback to generate a control parameter vector; control the actuator to adjust the execution state based on the control parameter vector, and trigger the target model to enter the next round of inference based on the execution feedback of the actuator.

[0160] Each functional module in the model inference device described in one or more of the foregoing embodiments can be logically regarded as a virtual functional unit undertaking a specific processing task. Its function can be implemented by software program instructions, processor execution logic or dedicated hardware circuit. The specific implementation method can be flexibly selected according to the actual deployment requirements, and is not specifically limited here.

[0161] Figure 11 This is a schematic diagram of an electronic device provided in an embodiment of this application. This embodiment of the application does not limit the specific implementation of the electronic device.

[0162] like Figure 11 As shown, the electronic device 400 may include: a processor 410, a communication interface 420, a memory 430, a computing array 440, a controller 450, and a communication bus 460.

[0163] The processor 410, communication interface 420, memory 430, computing array 440, and controller 450 communicate with each other via communication bus 460. Communication interface 420 is used for communication connections with other external devices. The processor 410 executes program 470 to implement the methods in one or more of the above embodiments.

[0164] Specifically, program 470 may include program code that includes computer operation instructions. When program 470 is invoked, processor 410 executes the computer operation instructions to implement the steps in one or more embodiments of the method.

[0165] Depending on the actual application scenario, the processor 410 can be of the appropriate type, including but not limited to mainstream embedded processors such as microcontrollers (MCUs) and ARM architecture processors, as well as other types of processors such as digital signal processors (DSPs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs), as long as they can provide the computing and control capabilities required by the actual application scenario.

[0166] The memory 430 is used to store the program 470. It includes a program storage area and a data storage area. The program storage area is used to store firmware programs, embedded applications, and various functional modules; the data storage area is used to store data and calculation results during program execution. For example, the memory 430 may include: on-chip RAM (for temporary data storage during program execution); on-chip Flash memory (for storing program code and configuration data); and EEPROM or other types of non-volatile memory (for storing parameters that need to be retained when power is off).

[0167] This application also provides a computer-readable storage medium. This computer-readable storage medium can be a non-volatile computer-readable storage medium. This computer-readable storage medium stores a computer program.

[0168] When executed by a processor, the computer program implements one or more steps of the method disclosed in the embodiments of this application. A complete computer program product is embodied on one or more computer-readable storage media containing the computer program disclosed in the embodiments of this application.

[0169] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above. For the sake of brevity, they are not provided in detail. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A model reasoning method, characterized in that, The method includes: Based on the structural analysis results of the target model, the activity level of neurons in each layer of the target model is determined. Based on the activity results, neuron-level physical reorganization is performed on the model weights of each layer in the target model to obtain the physical reorganization results; During the inference calculation of the current layer of the target model based on the physical recombination results, the local calculation results of the current layer are obtained; Based on the local calculation results of the current layer, predict the neuron activation state of the next layer and generate the neuron activation mask of the next layer; Based on the neuron activation mask of the next layer and the physical recombination result, the target weight data of the next layer is determined, and the initial computing array is reconstructed by gating based on the neuron activation mask of the next layer to obtain the target computing array. Based on the target weight data of the next layer and the target computation array, inference computation is performed on the next layer of the target model.

2. The model reasoning method according to claim 1, characterized in that, The structural analysis results based on the target model determine the activity level of neurons in each layer of the target model, including: Based on the structural analysis results of the target model, multiple rounds of forward inference sampling are performed on the target model on the training dataset and the running task set to obtain the results of multiple rounds of forward inference sampling. Based on the multi-round forward inference sampling results, the activation count, activation duration, and joint activation probability of adjacent neurons of all neurons in the target model under different input modes are recorded layer by layer. Based on the number of activations, the duration of activation, and the joint activation probability, a neuron activity distribution matrix is ​​generated. Based on the neuron activity distribution matrix, the activity results of neurons in each layer of the target model are determined.

3. The model reasoning method according to claim 1, characterized in that, Based on the activity results, neuron-level physical reorganization is performed on the model weights of each layer in the target model to obtain physical reorganization results, including: Based on the activity results, the activation count, activation duration, and joint activation probability of each neuron in the target model are determined. The weight block corresponding to the neuron whose activation count is greater than or equal to the first preset count and whose fluctuation value of the activation duration is less than or equal to the preset fluctuation value is used as the first neuron weight block; The weight block corresponding to the neuron that is triggered in the target scene and whose activation count is less than or equal to the second preset count is used as the second neuron weight block; A first weight set is formed based on the weight block of the first neuron, and a second weight set is formed based on the weight block of the second neuron. Map the weight blocks corresponding to adjacent neurons whose joint activation probability is greater than or equal to a preset threshold to adjacent physical storage addresses; Based on the first weight set, the second weight set, and the adjacent physical storage addresses, a hierarchical storage configuration is performed on the model weights of each layer in the target model, and a correspondence is established between neuron identifiers and weight storage locations to obtain the physical recombination result.

4. The model reasoning method according to claim 3, characterized in that, The step involves configuring hierarchical storage for the model weights of each layer in the target model based on the first weight set, the second weight set, and the adjacent physical storage addresses, and establishing a correspondence between neuron identifiers and weight storage locations to obtain the physical reorganization result, including: The first weight set is deployed to a first cache area located adjacent to the computing array, and the first weight storage location of each weight block in the first weight set in the first cache area is recorded; The second weight set is divided into multiple neuron micro-block units according to the neuron correspondence, and the multiple neuron micro-block units are mapped to off-chip main memory according to the adjacent physical storage addresses. The storage location of the second weight of each neuron micro-block unit in the off-chip main memory is recorded. Configure a neuron identifier for each neuronal microblock unit; Based on the first weight storage location, the second weight storage location, and the neuron identifier, a correspondence between the neuron identifier and the weight storage location is established to obtain the physical recombination result.

5. The model reasoning method according to claim 1, characterized in that, The process of obtaining the local calculation results of the current layer in the inference calculation of the target model based on the physical recombination results includes: During the current layer inference calculation of the target model based on the physical recombination results, matrix multiplication and addition calculations are performed on the current layer through a computing array; If the matrix multiplication and addition calculation in the current layer has not been completed, obtain the local accumulation result of the calculation array output; The local accumulation result is truncated according to the preset sampling window to obtain the local calculation result of the current layer.

6. The model reasoning method according to claim 1, characterized in that, The step of predicting the neuron activation state of the next layer based on the local calculation results of the current layer and generating the neuron activation mask of the next layer includes: The local calculation results of the current layer are linearly projected to obtain the activation prediction value of each neuron in the next layer. If the activation prediction value is greater than or equal to the preset activation threshold, the first state bit of the corresponding neuron is generated; If the activation prediction value is less than the preset activation threshold, a second state bit of the corresponding neuron is generated; Based on the first state bit and the second state bit, bitmap encoding is performed on the activation prediction state of each neuron in the next layer to generate the neuron activation mask of the next layer. In this context, each bit in the neuron activation mask corresponds to a neuron in the next layer. The first state bit is used to indicate that the corresponding neuron participates in the inference calculation of the next layer, and the second state bit is used to indicate that the corresponding neuron is skipped in the inference calculation of the next layer.

7. The model reasoning method according to claim 1, characterized in that, The determination of the target weight data for the next layer based on the neuron activation mask of the next layer and the physical recombination result includes: Based on the neuron activation mask of the next layer, the target neurons participating in inference computation in the next layer are determined. When the target neurons are continuously distributed in the neuron activation mask, the prefetching method of the target neurons is determined to be block prefetching; When the target neurons are not distributed discontinuously in the neuron activation mask, the prefetching method of the target neurons is determined to be skip prefetching; Based on the physical recombination results, the weight storage location of the target neuron is determined; Based on the prefetching method and the weight storage location, the weight data of the target neuron is read; Based on the reasoning and calculation order of the next layer, the read weight data is rearranged to obtain the target weight data of the next layer.

8. The model reasoning method according to claim 1, characterized in that, The gating reconstruction of the initial computation array based on the neuron activation mask of the next layer to obtain the target computation array includes: Based on the level state of each mask bit in the neuron activation mask of the next layer, determine the hit neurons that participate in the inference calculation and the miss neurons that skip the inference calculation in the next layer; Determine the hit operation clusters with the hit neurons and the miss operation clusters with the miss neurons in the initial computation array; A first gating parameter is generated based on the hit operation cluster, and a second gating parameter is generated based on the miss operation cluster; Based on the first gating parameter and the second gating parameter, the initial computing array is reconstructed by gating to obtain the target computing array; The first gating parameter is used to control the hit arithmetic cluster to open the local power supply path, restore the local clock signal and conduct the input / output exchange path, while the second gating parameter is used to control the miss arithmetic cluster to cut off the local power supply path, maintain the current state of the register and close the data exchange path.

9. The model reasoning method according to claim 1, characterized in that, The inference calculation for the next layer of the target model based on the target weight data of the next layer and the target computation array includes: Based on the target weight data of the next layer, the target computing array is controlled to perform inference calculations on the next layer of the target model to obtain the inference result of the next layer; When the next layer is the output layer of the target model, the semantic feature encoding of the target model is determined based on the inference results of the next layer; The semantic feature encoding is used for state recognition to obtain the current environment state label; Based on the current environment status label, determine the corresponding control template from the preset status mapping table; Based on real-time sensor feedback, the control parameters in the control template are adjusted to generate a control parameter vector; The actuator is controlled based on the control parameter vector to adjust its execution state, and the target model is triggered to enter the next round of inference based on the execution feedback of the actuator.

10. An electronic device, characterized in that, include: The system includes a memory, a computing array, a controller, and a processor, wherein the memory stores computer program instructions that, when invoked by the processor, cause the processor to cooperate with the controller and the computing array to execute the model inference method as described in any one of claims 1 to 9.

Images