An edge node-oriented lightweight AI model inference deployment system

Abstract

The application discloses an edge node-oriented lightweight AI model inference deployment system, which comprises a model analysis and layer block identification module, a performance portrait generation module, a memory quota joint optimization module, a resident layer block optimization filling module, a working area slice merging and adapting module and a comprehensive performance evaluation and scheme selection module.

Description

Technical Field

[0001] This invention relates to the field of AI model inference deployment technology, specifically a lightweight AI model inference deployment system for edge nodes. Background Technology

[0002] With the rapid popularization of edge computing technology and lightweight deep learning models, lightweight AI models are being deployed on a large scale in scenarios such as intelligent sensing, edge monitoring and control, and terminal intelligent analysis. These models perform inference locally at edge nodes, effectively reducing cloud transmission pressure and improving business response speed. Currently, most edge AI deployment models adopt a single-model independent deployment and static memory allocation approach, completing network loading, parameter initialization, and online inference for a single lightweight model, which can meet the basic intelligent inference needs of simple scenarios.

[0003] However, in complex edge scenarios with concurrent deployment of multiple models, existing deployment solutions have several technical shortcomings. First, existing technologies fail to exploit the common characteristics of network layers among multiple lightweight models. Each model's network layer is loaded and stored independently, resulting in a large number of network layers with similar structures and weights repeatedly occupying edge storage and memory resources. This leads to extremely low resource utilization at edge nodes and severe resource redundancy issues when deploying multiple models concurrently. Second, traditional edge inference deployment uses a fixed memory partitioning and static resource scheduling method, which cannot dynamically adjust memory quotas based on the multi-model layer reuse characteristics and hardware adaptability. This makes it difficult to adapt to the inference operation requirements of different model combinations, and it is prone to problems such as idle memory resources or memory overflow.

[0004] Meanwhile, existing edge inference solutions lack refined layer / block performance evaluation and memory sharding adaptation mechanisms. They cannot quantify the operator affinity, inference latency, and IO loading time of different network layers on edge hardware. In memory-constrained scenarios, they can only adapt to resource constraints by crudely reducing model parameters or compressing inference accuracy, significantly reducing model inference accuracy and stability. Furthermore, existing technologies have not established a multi-dimensional deployment scheme evaluation mechanism, failing to consider both IO loading overhead and inference parallelism loss. It is difficult to achieve an optimal balance between resource constraints and inference performance, resulting in low overall operating efficiency and poor dynamic adaptability of multi-model edge inference. This severely restricts the large-scale and efficient deployment and application of lightweight AI models on resource-constrained edge nodes.

[0005] In summary, existing edge AI model inference deployment technologies suffer from drawbacks such as the inability to reuse multiple model layers, rigid memory resource allocation, imprecise layer performance evaluation, and the inability to dynamically optimize deployment schemes. Therefore, there is an urgent need to propose a lightweight AI model inference deployment system for edge nodes.

[0006] It should be noted that the above content falls within the inventor's technical knowledge and does not necessarily constitute prior art. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a lightweight AI model inference deployment system for edge nodes, solving the current problems.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a lightweight AI model inference deployment system for edge nodes, comprising: The model parsing and layer identification module is used to import multiple lightweight deep learning models to be deployed in batches, and parse the network structure, operator type, weight parameters and inter-layer dependencies of each model layer by layer. Calculate hash fingerprints for the weights of each network layer, construct a cross-model inter-layer shared relationship graph by fingerprint matching, and divide the cross-model reused shared layer blocks and single-model unique private layer blocks; combine edge node hardware parameters to generate a unified layer block description table. The performance profile generation module is used to receive the unified layer block description table, conduct edge hardware benchmark tests on each layer block, obtain the layer block operator affinity score and single inference execution latency, calculate the external storage loading IO time of private layer blocks, and collect and generate layer block computing performance profiles. The memory quota joint optimization module is used to parameterize the shared memory into a residence area and a working area, with the total capacity of the shared memory of the edge nodes as a constraint. It generates multiple sets of candidate quota pairs that correspond one-to-one between the residence area and the working area through parameter traversal and iteration. The resident block selection and filling module is used to traverse each group of candidate quota pairs, combine the unified block description table and the block computing performance profile to calculate the reuse value score of each shared block, and fill the resident area according to the score to determine the optimal resident block set corresponding to each candidate quota. The work area sharding merging and adaptation module is used to filter out unprocessed layer blocks, complete the initial memory sharding based on operator affinity, iteratively merge the shards in combination with work area memory constraints, adapt the memory quota, and record the sharding parameters and model prediction inference latency. The comprehensive performance evaluation and scheme selection module is used to summarize the deployment parameters and inference latency corresponding to each candidate quota, calculate the comprehensive performance evaluation score of each candidate scheme, select the scheme with the highest score as the optimal memory partitioning and sharding scheduling scheme, and complete the edge model inference deployment configuration.

[0009] In some embodiments, the process by which the model parsing and layer identification module performs layer splitting, parameter acquisition, and generates a unified layer description table includes: Import multiple lightweight deep learning model files in batches, disassemble the network topology structure of each model and layer, identify the types of convolution, pooling, normalization and activation operators of each network layer, extract the weights and bias weights of each layer, and sort out the relationship between the pre-dependencies and post-calls between layers to complete the full model structure analysis. For each parsed layer weight and bias weight parameter, a fixed-length hash operation is performed to generate a unique hash fingerprint corresponding to each layer. The weight parameter vectors of different model network layers are dimensionally aligned, and the cosine similarity algorithm is used to calculate the similarity value of the weight vectors of two layers. Through a dual screening mechanism of global fingerprint traversal matching and similarity threshold, the weight similarity of different model network layers is compared, and a cross-model inter-layer shared relationship graph containing layer block reuse association, call frequency, and weight similarity value is constructed. The cosine similarity calculation method is as follows: the weights of the two network layers are mapped to high-dimensional vectors, and the cosine value of the angle between the vectors is used to quantify the similarity of the weight distribution. The closer the cosine similarity value is to 1, the higher the similarity of the weight distribution and feature extraction logic of the two network layers. When the similarity is greater than or equal to the preset similarity threshold, the two network layers are determined to be reusable. Based on the shared relationship graph, network layers that have successfully matched fingerprints among multiple models, have reached the cosine similarity threshold, and are reusable are aggregated into shared layer blocks, while network layers that are unique to a single model, have no matching reuse relationship, and have similarity below the threshold are divided into private layer blocks, thus completing the classification and splitting of all model layer blocks. Collect the total shared memory capacity, external storage read / write bandwidth, operator hardware adaptation parameters, and computing power frequency hardware parameters of edge nodes. Bind the hierarchical attributes, parameter sizes, dependencies, and reuse attributes of each shared and private layer block. Integrate all data to generate a standardized and unified layer block description table.

[0010] In some embodiments, the process by which the performance profile generation module performs benchmark testing, I / O time calculation, and constructs a layer-by-layer computing performance profile includes: Based on the native hardware environment of the edge node, an interference-free benchmark test environment is built. The hardware underlying interface is called to encapsulate and test all layers in the unified layer block description table one by one. According to the compatibility between each layer operator and the edge hardware core, the operator affinity score of 0-10 is quantified and generated. At the same time, the execution latency data of each layer block for a single complete inference is recorded. The method for setting up the interference-free benchmark testing environment is as follows: close the background processes of edge nodes, block the automatic scheduling and background update tasks of the system, lock the hardware computing power frequency and memory scheduling strategy, close peripheral read and write interference, and build an independent and isolated hardware test sandbox. For all private layers, extract the layer weight volume parameters from the unified layer description table, combine them with the transmission bandwidth and read / write latency parameters from the external storage to the shared memory preset by the edge nodes, and calculate the single load IO time of each private layer using the private layer IO time calculation formula. The formula for calculating the I / O time consumption of a single load of a private layer block is as follows: ; in, For the I / O time of a single load of a private layer block, For the private layer block weight data volume, Store transmission bandwidth for edge nodes. Store the base latency for read and write operations at edge nodes; Collect operator affinity scores, single inference execution latency, IO loading time, and layer block reuse frequency parameters for all shared and private layers, establish layer block performance index tags, and integrate all performance data to generate a structured layer block computing performance profile.

[0011] In some embodiments, the process by which the memory quota joint optimization module partitions memory and generates a set of candidate quota pairs includes: Read the total capacity parameter of the shared memory of the edge nodes and divide the shared memory into two major functional areas: a resident area for resident cache shared blocks and a working area for dynamically loading and running various blocks. The minimum step size for memory partitioning and the legal range of capacity for the resident area and the working area are preset to ensure that the sum of the quota for the resident area and the quota for the working area is equal to the total capacity of the shared memory, and that the quotas for both areas are greater than 0, thus satisfying the edge memory operation constraints. Using a preset segmentation step size as the iteration increment, the quota values ​​of the residence area are traversed and adjusted within the legal interval, and the corresponding quota values ​​of the work area are generated synchronously. All quota combinations that meet the constraints are collected one by one to generate a complete set of candidate quota pairs.

[0012] In some embodiments, the process by which the resident block preferential filling module calculates the shared block reuse value score and completes the block filling includes: Extract three core metrics from the layer block computing performance profile: cross-model reuse count of each shared layer block, single inference execution latency, and single load IO time. The reuse value scoring formula is constructed by pre-setting the reuse frequency weight coefficient, delay benefit weight coefficient, and IO saving weight coefficient, and using a weighted fusion algorithm to quantify the reuse value score of each shared layer block. The formula for calculating the reuse value score of shared layer blocks is as follows: ; in, For the value score of shared layer block reuse, The number of times a layer is reused across models. For the single inference execution delay of the layer, The IO time saved by loading a layer block per load. These are a preset reuse count weighting system, a latency benefit weighting coefficient, and an I / O saving weighting coefficient, respectively, and the weighting coefficients satisfy normalization constraints: ; Sort all shared blocks in descending order of reuse value score, and fill the blocks sequentially according to the current candidate resident area quota limit until the memory is full. Iterate through the filling of blocks for all candidate quota pairs to obtain the optimal resident block set corresponding to each quota group.

[0013] In some embodiments, the process of the work area fragmentation merging and adaptation module completing layer block fragmentation and iterative merging and adaptation includes: Filter out non-resident private blocks and low-reuse-value shared blocks under the current candidate quota and collect them into a set of blocks to be processed; retrieve the operator affinity score of each block to be processed, with a preset affinity score ≥7 as the high affinity threshold; at the same time, determine the operator association strength by the call dependency frequency of operators before and after the block and the data interaction dimension; the data interaction volume between the blocks before and after the block is greater than the preset interaction threshold and there is a strong temporal dependency relationship, and the blocks with high affinity and strong association are preferentially divided into the same initial memory shard, and the initial sharding of the blocks to be processed is completed; The total memory usage of all initial shards is calculated and compared with the current candidate workspace quota. If the total memory requirement is less than or equal to the workspace quota, the initial sharding result is directly retained; if the total memory requirement exceeds the workspace quota, the shard merging iteration mechanism is initiated. Calculate the cross-shard data transfer volume between each pair of adjacent shards, sort them according to the priority of the transfer volume from large to small, and prioritize merging adjacent shards with frequent data interaction to reduce cross-shard transmission overhead; After each shard merge is completed, the total memory requirement is recalculated. The merging operation is repeated until the total memory requirement meets the current working area memory constraint. Finally, the number, range, memory usage, transmission overhead parameters, and corresponding model prediction inference latency of all shards are recorded.

[0014] In some embodiments, the process by which the comprehensive performance evaluation and scheme selection module calculates the comprehensive score of the scheme and selects the optimal deployment scheme includes: Based on the set of resident blocks corresponding to each group of candidate quotas, the cumulative IO time saved by the resident blocks not needing to be reloaded is calculated as the IO saving benefit indicator of the solution. Based on the final sharding parameters corresponding to each group of candidate quotas, the cumulative inference latency increment caused by the reduction in operator parallelism after sharding merging is statistically analyzed as a performance loss indicator of the scheme. A comprehensive performance evaluation score calculation model for candidate solutions is constructed, and the calculation formula is as follows: ; in, The comprehensive performance evaluation score of the candidate scheme. The total IO savings resulting from a hit in the resident block. This refers to the latency increment caused by the total parallelism loss due to partitioning and merging. The comprehensive scores of all candidate schemes are sorted, and the scheme with the highest score and the best overall inference performance is selected. Its memory partitioning quota, resident layer block configuration, and sharding scheduling parameters are output as the optimal scheme.

[0015] In some embodiments, after the optimal memory partitioning and sharding scheduling scheme is output, the edge node inference deployment process includes: The memory sharding deployment of the layers to be processed is completed according to the optimal sharding parameters, and the hardware operator scheduling priority of each shard is bound. Based on the optimal solution's latency and IO performance parameters, an inference scheduling strategy is generated to drive lightweight AI models to complete efficient parallel inference deployment on edge nodes, achieving multi-model sharing and reuse, and optimal adaptation of memory resources.

[0016] Compared with existing technologies, this invention provides a lightweight AI model inference deployment system for edge nodes, which has the following advantages: 1. This lightweight AI model inference deployment system for edge nodes constructs a cross-model shared relationship graph by matching model layer block hash fingerprints, accurately divides shared layer blocks and private layer blocks, realizes cross-model reuse of multiple lightweight AI model network layers, completely solves the problem of repeated loading of multiple model layer blocks and resource redundancy waste in traditional edge deployment solutions, significantly improves the utilization rate of edge node storage and memory resources, and is suitable for multi-model concurrent deployment scenarios; 2. This lightweight AI model inference deployment system for edge nodes constructs a refined profile of layer-by-layer computing performance and combines edge hardware benchmark tests to quantify operator affinity, inference latency, and IO loading time. This enables accurate evaluation of the hardware adaptation performance of different layers, solving the shortcomings of traditional solutions that cannot quantify layer-by-layer performance and have blind resource scheduling. It provides accurate data support for subsequent memory optimization and sharding scheduling. 3. This lightweight AI model inference deployment system for edge nodes adopts a memory partition parameterized traversal optimization and reuse value-based filling mechanism to dynamically generate multiple sets of memory quota candidate schemes. Based on the high and low reuse value of the layer blocks, it prioritizes the residence of high-yield shared layer blocks, realizes the dynamic optimal division of the memory residence area and the working area, abandons the rigid mode of traditional fixed memory partitioning, and greatly improves the dynamic adaptation capability of edge memory resources. 4. This lightweight AI model inference deployment system for edge nodes is designed with an iterative merging and adaptation strategy based on cross-shard transmission volume. While meeting edge memory constraints, it maximizes operator parallelism, balancing memory resource constraints and inference parallel performance. This effectively solves the problems of decreased model inference accuracy and increased latency in memory-constrained scenarios, ensuring inference stability and efficiency. Simultaneously, a comprehensive performance evaluation system combining IO gains and parallel losses is constructed to quantitatively evaluate the comprehensive performance of each deployment scheme, automatically select the optimal memory partitioning and sharding scheduling scheme, and achieve multi-dimensional collaborative optimization of resource utilization, IO loading efficiency and inference latency for multi-model inference deployment on edge nodes, significantly improving the scalable deployment capability and inference performance of lightweight AI models in edge-constrained scenarios. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the module structure of the lightweight AI model inference deployment system of the present invention; Figure 2 This is a schematic diagram of the process for constructing the multi-model layer block splitting and sharing relationship of edge nodes proposed in this invention; Figure 3 This is a schematic diagram of the memory quota iterative optimization and fragment merging adaptation process proposed in this invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0020] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0022] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0023] Please see Figure 1-3 This implementation plan includes: a lightweight AI model inference deployment system for edge nodes, adapted to embedded edge gateways and computing-limited terminals for concurrent inference of multiple lightweight models, specifically including: a model parsing and layer block identification module, a performance profile generation module, a memory quota joint optimization module, a resident layer block selective filling module, a work area sharding and merging adaptation module, and a comprehensive performance evaluation and scheme selection module.

[0024] Step 1: Import, parse, split, and establish shared relationships for multiple models. Three lightweight deep learning models to be deployed are imported in batches: a lightweight object detection model, a lightweight classification model, and a lightweight semantic segmentation model. The network topology is disassembled model by model and layer by layer. Operator types such as convolution, pooling, BN normalization, and ReLU activation are identified. The weight parameters and bias parameters of each network layer are extracted. The inter-layer relationships of pre-dependencies and post-calls of each layer are sorted out, and the full model structure is analyzed. For each layer's weight parameters, a fixed-length SHA-256 hash operation is performed to generate a unique hash fingerprint corresponding to a single network layer. The weight parameters of different model network layers are dimensionally aligned and uniformly mapped to weight vectors of the same dimension. The cosine similarity algorithm is used to calculate the weight similarity across model network layers. The cosine similarity formula is as follows: ; in, For network layer weight vectors of different models, The value range is [0,1]. In this embodiment, the preset similarity threshold is 0.9. When the value is ≥0.9, it is determined that the feature extraction logic of the two-layer network is consistent and the weight distribution is highly similar, thus meeting the conditions for cross-model reuse. By combining the results of precise hash fingerprint matching and cosine similarity, the cross-model call frequency of each reusable layer is statistically analyzed, and a cross-model inter-layer sharing relationship graph containing reusability associations, call frequencies, and similarity values ​​is constructed. Network layers that meet fingerprint matching and similarity thresholds are aggregated into shared layer blocks, while network layers that are unique to a single model, have no reusability associations, and have similarity below a threshold are classified into private layer blocks, thus completing the classification and splitting of all model layer blocks. Collect hardware parameters such as the total shared memory capacity of edge nodes, external storage read / write bandwidth, operator hardware adaptation parameters, and CPU / GPU computing power frequency. Bind and archive the hardware parameters with the hierarchical attributes, parameter volume, dependencies, and reuse attributes of each shared and private layer block to generate a standardized unified layer block description table, providing standardized data support for subsequent performance testing and memory optimization.

[0025] Step 2: Interference-free benchmark testing and layer / block performance profile construction: To build an interference-free benchmark sandbox in the native Linux embedded environment of the edge node, the following steps were taken: disable interfering tasks such as automatic background upgrades, log printing, and background daemons; lock the hardware computing power frequency and memory scheduling priority; disable additional I / O overhead such as peripheral read / write and network transmission; and build a completely isolated hardware test environment to ensure that the test data is accurate and interference-free. The system calls the underlying driver interface of the edge hardware to encapsulate and test all shared and private layer blocks in the unified layer block description table. Based on the differences in native hardware support, compilation adaptation overhead, and scheduling latency, it quantitatively outputs an operator affinity score of 0-10: core operators with native hardware support and no adaptation overhead are rated 9-10 points, conventional compatible operators are rated 4-8 points, and niche operators that require software simulation adaptation and have higher overhead are rated 0-3 points. At the same time, the execution latency data of a single complete inference for each layer block is recorded. For all private layers, the layer weight data volume is extracted. Combined with the preset storage bandwidth and basic read / write latency of the edge nodes, the single-load IO time is calculated using the private layer IO time calculation formula: ; in, For the I / O time of a single load of a private layer block, For the private layer block weight data volume, Store transmission bandwidth for edge nodes. Store the base latency for read and write operations at edge nodes; Collect operator affinity, inference latency, IO loading time, and cross-model reuse frequency of all layers, establish layer performance index tags, and integrate to generate a structured and searchable layer computation performance profile; Step 3: Memory-parameterized segmentation and candidate quota pair generation: Read the total capacity of the shared memory of the edge node, divide the shared memory into the resident area of ​​the resident cache shared layer block, the working area of ​​the dynamically loaded private layer block and the low-value shared layer block, the preset minimum step size of memory partitioning is 64KB, and the quota of the resident area and the working area are both greater than 0 and the sum of the capacities of the two areas is equal to the total shared memory. Using 64KB as the iterative increment, the resident area quota is traversed and adjusted one by one within the legal capacity range, and the corresponding work area quota is generated synchronously. All quota combinations that meet the constraints are collected to generate a complete set of resident area-work area candidate quota pairs, covering all feasible solutions for memory partitioning in the entire range. Step 4: Shared layer block reuse value scoring and optimal filling of the dwell area: Based on the performance profile of each layer's computation, the number of times each shared layer is reused across models is extracted. Single inference delay Savings per I / O operation Set normalized weight coefficients ,satisfy The reuse value score is calculated using a weighted fusion formula: ; Sort all shared blocks in descending order of score from high to low. For each candidate quota pair, fill the resident memory in order of score priority until the resident memory is full. After traversing all candidate quota pairs, obtain the optimal resident block set corresponding to each quota, so as to realize that high-value reusable blocks are given priority to reside in memory and reduce the overhead of repeated IO loading. Step 5: Initial partitioning and iterative merging and adaptation of the working area: Private blocks and low-score shared blocks that are not resident under each quota group are filtered and collected into a set of blocks to be processed. Affinity score ≥7 is used as the threshold for high affinity. Strong correlation operators are determined by combining the amount of data interaction and temporal dependency between blocks. Blocks with high affinity, strong data interaction, and strong temporal dependency are grouped into the same initial memory shard to complete the initial shard splitting. The initial total memory usage of each shard is compared with the current working space quota. If the total memory requirement exceeds the working space capacity, an iterative merging mechanism is initiated: the total amount of tensor interactions in a single inference operation between any two adjacent shards is calculated, and the shards are sorted from largest to smallest cross-shard transfer volume. High-frequency interacting adjacent shards are merged first to minimize cross-shard data transfer overhead. Memory usage is recalculated after each merge, and this process is iterated until the working space memory constraint is met. Finally, the number of shards, memory usage, transfer overhead, and model prediction inference latency are recorded. Step Six: Comprehensive Evaluation of Multiple Options and Selection of Optimal Deployment Scheme For each set of candidate quotas and the corresponding deployment plan, calculate the total IO savings brought by the resident blocks in memory. Meanwhile, the total latency loss caused by the decrease in operator parallelism due to the statistical partitioning and merging is also calculated. The score for the proposed solution is calculated using a comprehensive scoring formula: ; in, The comprehensive performance evaluation score of the candidate scheme. The total IO savings resulting from a hit in the resident block. This refers to the latency increment caused by the total parallelism loss due to partitioning and merging. Sort all candidate solutions in descending order of their overall scores, select the deployment solution with the highest score, and extract the memory partition quota, resident layer block configuration, and work area sharding parameters corresponding to that solution as the globally optimal deployment solution. Step 7: Edge Node Deployment and Inference Scheduling The system deploys memory shards for the layers to be processed based on optimal sharding parameters, assigns corresponding hardware operator scheduling priorities to each shard, and prioritizes high-real-time and high-affinity shards for scheduling. A dynamic inference scheduling strategy is generated based on the optimal IO performance and latency parameters, driving multiple lightweight AI models to complete parallel and efficient inference on edge nodes, achieving multi-model layer shard sharing and reuse, precise adaptation of memory resources, and optimal inference performance.

[0026] In the application, several formulas are calculated by removing dimensions and taking their numerical values. The formulas are established by collecting a large amount of data and simulating the most recent real situation. Some coefficients or weights in the formulas are set by those skilled in the art according to the actual situation, so they will not be elaborated here.

[0027] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0028] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lightweight AI model inference deployment system for edge nodes, characterized in that, include: The model parsing and layer identification module is used to import multiple lightweight deep learning models to be deployed in batches, and parse the network structure, operator type, weight parameters and inter-layer dependencies of each model layer by layer. Calculate hash fingerprints for the weights of each network layer, construct a cross-model inter-layer shared relationship graph by fingerprint matching, and divide the shared layer blocks that can be reused across models into private layer blocks that are unique to a single model. A unified layer block description table is generated by integrating the hardware parameters of edge nodes; The performance profile generation module is used to receive the unified layer block description table, conduct edge hardware benchmark tests on each layer block, obtain the layer block operator affinity score and single inference execution latency, calculate the external storage loading IO time of private layer blocks, and collect and generate layer block computing performance profiles. The memory quota joint optimization module is used to parameterize the shared memory into a residence area and a working area, with the total capacity of the shared memory of the edge nodes as a constraint. It generates multiple sets of candidate quota pairs that correspond one-to-one between the residence area and the working area through parameter traversal and iteration. The resident block selection and filling module is used to traverse each group of candidate quota pairs, combine the unified block description table and the block computing performance profile to calculate the reuse value score of each shared block, and fill the resident area according to the score to determine the optimal resident block set corresponding to each candidate quota. The work area sharding merging and adaptation module is used to filter out unprocessed layer blocks, complete the initial memory sharding based on operator affinity, iteratively merge the shards in combination with work area memory constraints, adapt the memory quota, and record the sharding parameters and model prediction inference latency. The comprehensive performance evaluation and scheme selection module is used to summarize the deployment parameters and inference latency corresponding to each candidate quota, calculate the comprehensive performance evaluation score of each candidate scheme, select the scheme with the highest score as the optimal memory partitioning and sharding scheduling scheme, and complete the edge model inference deployment configuration.

2. The lightweight AI model inference deployment system for edge nodes according to claim 1, characterized in that, The process by which the model parsing and block identification module performs block splitting, parameter acquisition, and generates a unified block description table includes: Import multiple lightweight deep learning model files in batches, disassemble the network topology structure of each model and layer, identify the types of convolution, pooling, normalization and activation operators of each network layer, extract the weights and bias weights of each layer, and sort out the relationship between the pre-dependencies and post-calls between layers to complete the full model structure analysis. For each parsed layer weight and bias weight parameter, a fixed-length hash operation is performed to generate a unique hash fingerprint corresponding to each layer. The weight parameter vectors of different model network layers are dimensionally aligned, and the cosine similarity algorithm is used to calculate the similarity value of the weight vectors of two layers. Through a dual screening mechanism of global fingerprint traversal matching and similarity threshold, the weight similarity of different model network layers is compared, and a cross-model inter-layer shared relationship graph containing layer block reuse association, call frequency, and weight similarity value is constructed. The cosine similarity calculation method is as follows: the weights of the two network layers are mapped to high-dimensional vectors, and the cosine value of the angle between the vectors is used to quantify the similarity of the weight distribution. The closer the cosine similarity value is to 1, the higher the similarity of the weight distribution and feature extraction logic of the two network layers. When the similarity is greater than or equal to the preset similarity threshold, the two network layers are determined to be reusable. Based on the shared relationship graph, network layers that have successfully matched fingerprints among multiple models, have reached the cosine similarity threshold, and are reusable are aggregated into shared layer blocks, while network layers that are unique to a single model, have no matching reuse relationship, and have similarity below the threshold are divided into private layer blocks, thus completing the classification and splitting of all model layer blocks. Collect the total shared memory capacity, external storage read / write bandwidth, operator hardware adaptation parameters, and computing power frequency hardware parameters of edge nodes. Bind the hierarchical attributes, parameter sizes, dependencies, and reuse attributes of each shared and private layer block. Integrate all data to generate a standardized and unified layer block description table.

3. The lightweight AI model inference deployment system for edge nodes according to claim 2, characterized in that, The process by which the performance profile generation module performs benchmark testing, I / O time calculation, and constructs a layer-block computing performance profile includes: Based on the native hardware environment of the edge node, an interference-free benchmark test environment is built. The hardware underlying interface is called to encapsulate and test all layers in the unified layer block description table one by one. According to the compatibility between each layer operator and the edge hardware core, the operator affinity score of 0-10 is quantified and generated. At the same time, the execution latency data of each layer block for a single complete inference is recorded. The method for setting up the interference-free benchmark testing environment is as follows: close the background processes of edge nodes, block the automatic scheduling and background update tasks of the system, lock the hardware computing power frequency and memory scheduling strategy, close peripheral read and write interference, and build an independent and isolated hardware test sandbox. For all private layers, extract the layer weight volume parameters from the unified layer description table, combine them with the transmission bandwidth and read / write latency parameters from the external storage to the shared memory preset by the edge nodes, and calculate the single load IO time of each private layer using the private layer IO time calculation formula. The formula for calculating the I / O time consumption of a single load of a private layer block is as follows: ； in, For the I / O time of a single load of a private layer block, For the private layer block weight data volume, Store transmission bandwidth for edge nodes. Store the base latency for read and write operations at edge nodes; Collect operator affinity scores, single inference execution latency, IO loading time, and layer block reuse frequency parameters for all shared and private layers, establish layer block performance index tags, and integrate all performance data to generate a structured layer block computing performance profile.

4. The lightweight AI model inference deployment system for edge nodes according to claim 3, characterized in that, The process by which the memory quota joint optimization module partitions memory and generates a set of candidate quota pairs includes: Read the total capacity parameter of the shared memory of the edge nodes and divide the shared memory into two major functional areas: a resident area for resident cache shared blocks and a working area for dynamically loading and running various blocks. The minimum step size for memory partitioning and the legal range of capacity for the resident area and the working area are preset to ensure that the sum of the quota for the resident area and the quota for the working area is equal to the total capacity of the shared memory, and that the quotas for both areas are greater than 0, thus satisfying the edge memory operation constraints. Using a preset segmentation step size as the iteration increment, the quota values ​​of the residence area are traversed and adjusted within the legal interval, and the corresponding quota values ​​of the work area are generated synchronously. All quota combinations that meet the constraints are collected one by one to generate a complete set of candidate quota pairs.

5. A lightweight AI model inference deployment system for edge nodes according to claim 4, characterized in that, The process by which the preferred filling module for resident blocks calculates the reuse value score of shared blocks and completes the filling of blocks includes: Extract three core metrics from the layer block computing performance profile: cross-model reuse count of each shared layer block, single inference execution latency, and single load IO time. The reuse value scoring formula is constructed by pre-setting the reuse frequency weight coefficient, delay benefit weight coefficient, and IO saving weight coefficient, and using a weighted fusion algorithm to quantify the reuse value score of each shared layer block. The formula for calculating the reuse value score of shared layer blocks is as follows: ； in, For the value score of shared layer block reuse, The number of times a layer is reused across models. For the single inference execution delay of the layer, The IO time saved by loading a layer block per load. These are a preset reuse count weighting system, a latency benefit weighting coefficient, and an I / O saving weighting coefficient, respectively, and the weighting coefficients satisfy normalization constraints: ; Sort all shared blocks in descending order of reuse value score, and fill the blocks sequentially according to the current candidate resident area quota limit until the memory is full. Iterate through the filling of blocks for all candidate quota pairs to obtain the optimal resident block set corresponding to each quota group.

6. A lightweight AI model inference deployment system for edge nodes according to claim 5, characterized in that, The process of layer block segmentation, iterative merging and adaptation by the working area segmentation and merging adaptation module includes: Filter out non-resident private blocks and low-reuse-value shared blocks under the current candidate quota and collect them into a set of blocks to be processed; retrieve the operator affinity score of each block to be processed, with a preset affinity score ≥7 as the high affinity threshold; at the same time, determine the operator association strength by the call dependency frequency of operators before and after the block and the data interaction dimension; the data interaction volume between the blocks before and after the block is greater than the preset interaction threshold and there is a strong temporal dependency relationship, and the blocks with high affinity and strong association are preferentially divided into the same initial memory shard, and the initial sharding of the blocks to be processed is completed; The total memory usage of all initial shards is calculated and compared with the current candidate workspace quota. If the total memory requirement is less than or equal to the workspace quota, the initial sharding result is directly retained; if the total memory requirement exceeds the workspace quota, the shard merging iteration mechanism is initiated. Calculate the cross-shard data transfer volume between each pair of adjacent shards, sort them according to the priority of the transfer volume from large to small, and prioritize merging adjacent shards with frequent data interaction to reduce cross-shard transmission overhead; After each shard merge is completed, the total memory requirement is recalculated. The merging operation is repeated until the total memory requirement meets the current working area memory constraint. Finally, the number, range, memory usage, transmission overhead parameters, and corresponding model prediction inference latency of all shards are recorded.

7. A lightweight AI model inference deployment system for edge nodes according to claim 6, characterized in that, The process by which the comprehensive performance evaluation and scheme selection module calculates the comprehensive score of the schemes and selects the optimal deployment scheme includes: Based on the set of resident blocks corresponding to each group of candidate quotas, the cumulative IO time saved by the resident blocks not needing to be reloaded is calculated as the IO saving benefit indicator of the solution. Based on the final sharding parameters corresponding to each group of candidate quotas, the cumulative inference latency increment caused by the reduction in operator parallelism after sharding merging is statistically analyzed as a performance loss indicator of the scheme. A comprehensive performance evaluation score calculation model for candidate solutions is constructed, and the calculation formula is as follows: ； in, The comprehensive performance evaluation score of the candidate scheme. The total IO savings resulting from a hit in the resident block. This refers to the latency increment caused by the total parallelism loss due to partitioning and merging. The comprehensive scores of all candidate schemes are sorted, and the scheme with the highest score and the best overall inference performance is selected. Its memory partitioning quota, resident layer block configuration, and sharding scheduling parameters are output as the optimal scheme.

8. A lightweight AI model inference deployment system for edge nodes according to claim 7, characterized in that, After the optimal memory partitioning and sharding scheduling scheme is output, the edge node inference deployment process includes: The memory sharding deployment of the layers to be processed is completed according to the optimal sharding parameters, and the hardware operator scheduling priority of each shard is bound. Based on the optimal solution's latency and IO performance parameters, an inference scheduling strategy is generated to drive lightweight AI models to complete efficient parallel inference deployment on edge nodes, achieving multi-model sharing and reuse, and optimal adaptation of memory resources.

Images