Edge computing-based model partitioning application inference deployment method

Abstract

A kind of application inference deployment method based on model partition for edge computing, in performance test stage, the delay and bandwidth of network link and the time consumption of each layer of neural network (DNN) are obtained by network performance test, client inference task delay test and server inference task delay test respectively, and the best split point configuration is obtained by split point decision method;In local inference stage, after obtaining the image, the inference task before split point is executed by the current client according to the best split point configuration, and the relay point data is transmitted to the server located at the edge side after int8 feature quantization processing;In edge inference stage, the server executes edge inference after split point according to the same best split point configuration, and executes inference task, and the result is transmitted back to the client, and the classification result and detection frame are output by the client after format processing. The application is realized through the practical available system of application deployment for edge computing, and the performance improvement verification of actual system in image classification and target detection two kinds of AI applications is completed through the split point decision method of dynamic adjustment.

Description

Technical Field

[0001] This invention relates to a technology in the field of distributed information processing, specifically an application inference deployment method based on model segmentation for edge computing. Background Technology

[0002] Edge computing is a distributed computing framework that pushes cloud services from the network core to the network edge. Its goal is to host computing tasks as close as possible to data sources and end-users, making applications closer to data sources such as IoT devices, local terminal devices, or edge servers. Current edge inference technologies lack methods for deployment and performance verification in real-world systems. Performance testing is limited to simulation testing in simulated environments, lacking methods for deploying AI applications and performing edge segmentation inference on actual hardware and software platforms. Furthermore, there is a lack of methods for verifying the performance of these methods on real-world platforms. Meanwhile, the optimization of feature map data transmission based on model segmentation is inadequate: the DNN models that can be optimized for segmentation in edge inference still require excessively large amounts of feature map data for communication transmission, making it difficult to reduce transmission latency. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes an application inference deployment method based on model segmentation for edge computing. Specifically, in dynamic network environments, this method combines model segmentation and feature quantization techniques. Through a practical, usable system deployed for edge computing applications, and using a dynamically adjusted segmentation point decision method, the performance improvement of the actual system in two AI applications—image classification and object detection—is verified.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to an application inference deployment method based on model segmentation for edge computing. In the performance testing phase, network link latency and bandwidth, as well as the execution time of each layer of the deep neural network (DNN), are obtained through network performance testing, client-side inference task latency testing, and server-side inference task latency testing. An optimal segmentation point configuration is then obtained through a segmentation point decision method. In the local inference phase, after acquiring the image, the current client executes the inference task before the segmentation point according to the optimal segmentation point configuration, and the relay point data is transmitted to the server located at the edge after int8 feature quantization. In the edge inference phase, the server executes edge inference after the segmentation point according to the same optimal segmentation point configuration and, after executing the inference task, transmits the results back to the client, which performs formatting processing and outputs classification results and detection boxes.

[0006] The optimal split point configuration refers to the logical layer position of the DNN model where the DNN inference split decision point, obtained through the split point decision method with the goal of achieving optimal latency performance, is located.

[0007] The aforementioned client-side inference task latency test and server-side inference task latency test refer to: the client and server running the DNN inference task of the AI ​​application on a trial basis, timing the execution time of each layer to measure the execution time of each layer.

[0008] Preferably, to avoid errors caused by fluctuations in host task load, client inference task latency tests and server inference task latency tests will be run multiple times.

[0009] The execution time of each layer of the DNN refers to the time difference in completing the same layer of inference under different performance conditions of the two hosts by testing and statistically analyzing the hardware computing performance and network performance information of the inference module on both the client and the server.

[0010] The network performance test mentioned above refers to comparing the network link round-trip time (TN) with the transmission delay (D). i The total network latency is obtained by superimposing the bandwidth (BW) and the transmission latency (D), where: the transmission latency is based on the real-time network bandwidth (BW) and the transmission data volume (D). i The earlier the segmentation point, the larger the feature map needs to be transmitted, which will introduce a longer transmission time.

[0011] The optimal split point configuration in both the local and edge inference stages is obtained using an improved DNN. This improved DNN includes an initialization module, a sequence transformation module, and a forward propagation logic module. Specifically, the initialization module inherits the feature extraction block, average convergence block, and feature classification block from the PyTorch official standard DNN and outputs them to the sequence transformation module. The sequence transformation module uses the ModuleList container provided by PyTorch to flatten all network layer blocks inherited from the original DNN by the initialization module. After flattening, all layer-level sub-modules are placed into a ModuleList container in sequence. The forward propagation logic module controls the ModuleList container to stop inference at the specified DNN layer sub-module based on the split point configuration information and returns the output tensor of that module.

[0012] The aforementioned flattening process refers to: splitting the network modules encapsulated by the PyTorch official standard DNN into layers and re-encapsulating them sequentially into the ModuleList container of the sequence transformation module, so that the output of any network layer can be obtained when performing model segmentation during the inference stage.

[0013] The aforementioned segmentation point configuration information refers to: the sub-modules {L1, L2, ..., L...} of the layer granularity after flattening. N} and the dividing point L i , where: L1 to Li The inference task of the layer network is executed locally on the client. i+1 Layer to L N Layer 1 networking executes at the server edge, specifically: when the client runs to layer L... i When layer L, i The output tensor of the layer is the relay feature data in this segmentation task. After int8 feature quantization, it is sent to the server, making L i The reasoning tasks after the layer are completed on the server side.

[0014] The aforementioned split point decision method specifically includes:

[0015] Step a: Input parameters include: the total number of DNN layers, and the set of DNN network layers {L}. i |0, 1, ..., i, ..., N}, the data volume of each layer's output after int8 feature quantization is {D}. i |0, 1, ..., i, ..., N}, the execution time of each layer on the client side {TC i |0, 1, ..., i, ..., N}, the execution time of each layer on the server side {TS i |0, 1, ..., i, ..., N}, the round-trip latency of the network link between the client and the server, the bandwidth of the network between the client and the server, and the data volume constant (default value is 8).

[0016] Step b: Assume each layer i of the DNN as a split point and iteratively test its E2E latency performance. The latency performance is expressed as Latency = TN + TC. i +TS i +S·D i / BW. After looping N+1 times, the split point with the lowest E2E delay is recorded as the optimal split point, and the optimal split point configuration is output.

[0017] Technical effect

[0018] This invention addresses the issue of excessively large feature map data volume requiring communication transmission after model segmentation by employing int8 feature quantization technology. Compared to existing edge inference deployment simulation technologies, it enables segmentation inference deployment in actual hardware and software systems, making it possible to verify inference performance in real dynamic network and computing environments. It allows for the evaluation of network communication performance and client-server computing performance in dynamic environments. By adjusting model segmentation points, it effectively reduces E2E inference latency in image classification applications, reduces the data size of relay feature maps to one-quarter of the original size, effectively reduces communication transmission latency, and improves the average inspection frame rate of target detection applications. Attached Figure Description

[0019] Figure 1This is a schematic diagram of the system of the present invention;

[0020] Figure 2 This is a flowchart of the present invention;

[0021] Figure 3 Here is a flowchart of the process for obtaining model segmentation points in an example embodiment;

[0022] Figure 4 This is an example of a client-side flowchart breakdown.

[0023] Figure 5 This is an example of a server-side process breakdown diagram;

[0024] Figure 6 The diagram shows the E2E inference latency effect of image recognition in the example.

[0025] Figure 7 The image shows the average detection frame rate of the target detection in the example. Detailed Implementation

[0026] like Figure 1 As shown in this embodiment, an application inference deployment system based on model segmentation for edge computing is included. It comprises: a control module, an image acquisition module, a local inference module, a local communication module, and an output processing module located on the client side; and an edge communication module and an edge inference module located on the server side. Specifically: during the testing phase, the control module obtains the execution time of each layer of the DNN through the local and edge inference modules, and obtains the network link latency and bandwidth through the local communication module. It then dynamically outputs the optimal segmentation point configuration to the inference module using a segmentation point decision method. During the inference phase, the image acquisition module acquires images, converts them into tensor matrix form, and outputs them to the inference module. The local inference module located on the client side includes a pre-processing module... The DNN with trained weights performs partial DNN layer processing on the tensor matrix image based on the optimal segmentation point configuration, extracts relay feature data, and outputs it to the local communication module. After int8 feature quantization, the data is then output to the server. The output processing module receives the server-side inference results from the communication module and presents them in different ways depending on the task type. The edge inference module located on the server receives and performs inverse int8 feature quantization on the relay feature data generated by the local inference module through the edge communication module. Based on the optimal segmentation point configuration generated by the control module, the built-in pre-trained DNN inputs the relay feature data into the specified network layer, performs inference, and returns the output results to the client through the edge communication module.

[0027] The aforementioned image acquisition refers to: directly decoding images from image files or acquiring frame images through frame-by-frame or timed sampling by a camera device.

[0028] Preferably, the local inference module and the edge inference module perform hardware computing performance tests in advance and output the execution time of each layer of the DNN on the client and server to the control module for optimizing the split point decision.

[0029] The local communication module, after completing local inference, performs int8 feature quantization on the relay feature data output by the client's DNN, formats it together with the optimal split point configuration to generate a pickle-formatted binary bit stream, and outputs it to the server via an HTTP request.

[0030] The int8 feature quantization mentioned above refers to: asymmetric quantization based on linear quantization, that is, based on the feature map matrix R. m The maximum tensor in R m Minimum tensor min(R) m After determining the quantization coefficients s and the zero point z, and then performing a scaling transformation with rounding and a truncation function, the original 4-byte feature map matrix R of type float32 is obtained. m The element x is compressed into a feature parameter x of type int8, which is 1 byte in size. q This forms the quantized feature map matrix R. q Specifically, this refers to: quantization coefficient Zero point z = -round(min(R) map )·s)-2 7 , characteristic parameter x q =quantize(x,s,z)=clip(round(s·x+z),-2 7 ,2 7 -1), where the truncation function is: l represents the upper bound of truncation, and u represents the lower bound of truncation.

[0031] Preferably, during the performance testing phase, the local communication module pre-measures the network communication quality between the client and the server, and outputs the obtained network link latency and bandwidth to the control module.

[0032] The aforementioned inverse int8 feature quantization processing refers to: the feature map matrix R received by the server... q Each element x q It will be transformed one by one into a 4-byte float32 type using the dequantization formula. Restore to the feature matrix Specifically: s and z are the quantization coefficients and zeros obtained from the client-side int8 feature quantization process.

[0033] The control module includes a network segmentation control unit and an image acquisition control unit. During the performance testing phase, the network segmentation control unit processes the segmentation point decision method based on the execution time of each layer of the DNN fed back by the local and edge inference modules, as well as the network link latency and bandwidth test results fed back by the local communication module, to obtain the optimal segmentation point configuration. The image acquisition control unit controls image decoding or video frame acquisition based on the information type obtained by the image acquisition module, thereby enabling the selection of image preprocessing methods and video frame acquisition frame rates.

[0034] The image acquisition module, through its built-in image preprocessing unit, and based on the file input from the control module and the camera acquisition control, directly decodes and acquires images from image files, and acquires frame images from video or camera devices by sampling frame by frame or at regular intervals, thereby acquiring images from files or cameras.

[0035] The local inference module includes a client-side computational performance testing unit and a local model segmentation inference unit. The client-side computational performance testing unit performs hardware computational performance testing according to the commands of the test phase control module and outputs the execution time of each layer of the DNN on the client to the control module. The local model segmentation inference unit performs inference of the network layers before the segmentation point of the DNN on the tensor matrix image according to the optimal segmentation point configuration output by the control module and extracts relay feature data, which is then sent to the server through the local communication module.

[0036] The local communication module includes: a network performance testing unit, an int8 feature quantization unit, and a relay point data transmission unit. Specifically: the network performance testing unit performs network performance testing according to commands from the testing phase control module, obtains the real-time network link latency and bandwidth, and outputs it to the control module; the int8 feature quantization unit quantizes the relay point data and transmits it to the relay point data transmission unit; the relay point data transmission unit formats the relay point data and the optimal split point configuration together into a pickle-formatted binary bitstream, and then outputs it to the server via an HTTP request.

[0037] The edge inference module includes a server-side computing performance testing unit and an edge model segmentation inference unit. The server-side computing performance testing unit performs hardware computing performance testing according to the commands of the testing phase control module and outputs the execution time of each layer of the DNN on the server to the control module. The edge model segmentation inference unit performs inference of the network layers after the segmentation point of the DNN according to the optimal segmentation point configuration and the relay point data after inverse int8 feature quantization received by the edge communication module and outputs the inference results to the client through the edge communication module.

[0038] The edge communication module includes an inverse int8 feature quantization unit and an inference result transmission unit, wherein: the inverse int8 feature quantization unit performs inverse int8 feature quantization processing on the relay point data output by the client and transmits it to the edge inference module; the inference result transmission unit formats the inference result output by the edge inference module and outputs it to the client in the form of an HTTP request.

[0039] The output processing module, through its built-in inference result display unit, formats and presents the data to the user in different ways according to the task type based on the server-side inference results obtained from the local communication module. For classification tasks, it displays image categories, and for object detection tasks, it draws object detection boxes.

[0040] Through practical experiments, a high-performance x86-based desktop computer was used as the server, and a low-performance ARM-based laptop was used as the client. The two devices were connected wirelessly via Wi-Fi (802.11ac, 5GHz). The `tc` tool on a Linux system was used to limit the basic latency and bandwidth between the server and client. To simulate a 5G scenario, the uplink bandwidth was set to 100Mbps, the random packet loss rate to 0.2%, and the RTT latency to 10ms. Tests showed that the optimal E2E inference latency for image classification applications based on the VGG11 model was 123.6ms; the optimal average detection frame rate for object detection applications based on the SSD model was 7.3 FPS.

[0041] like Figures 2 to 5 As shown, this embodiment illustrates the edge computing-based model segmentation-based application inference deployment method for the aforementioned system. During the performance testing phase, network link latency and bandwidth, as well as the execution time of each DNN layer, are obtained through network performance testing, client inference task latency testing, and server inference task latency testing. The optimal segmentation point configuration is then obtained through a segmentation point decision method. During the local inference phase, after acquiring the image, the current client executes the inference task before segmentation and transmits the relay point data to the server located on the edge side for edge inference according to the optimal segmentation point configuration after int8 feature quantization. After the server executes the inference task and returns the results, the client further formats the data and outputs the classification results and detection boxes.

[0042] This embodiment employs the following methods for inference deployment and performance verification: An image classification application based on the VGG11 model and an object detection application based on the SSD model are selected. The weights for both are pre-trained using the ImageNet and COCO datasets, respectively. The selectable segmentation points are layers 0 to 29 and layers 0 to 23. Layer 0 as a segmentation point indicates that the DNN does not segment, and the inference task is completely offloaded to the server for computation. The last layer as a segmentation point indicates that the DNN does not segment, and the inference task is entirely computed locally on the client side. The quantization value for relay data features is set to 8 bits, using non-linear quantization and dequantization methods.

[0043] When the application is deployed for image classification based on the VGG11 model, this invention effectively reduces end-to-end inference latency and improves the user experience for client users. Figure 6 As shown in Table 1, when the split point is 21, the total inference latency is reduced by 24.8% compared to the case where the task is completely offloaded to the server for computation (split point 0). The main latency gain comes from the reduction in transmission latency and the coordinated use of computing power between the client and the server.

[0044] Table 1. Impact of Segmentation Point Decision on Image Recognition Inference Performance

[0045]

[0046] When the application is deployed for target recognition based on the SSD model, this invention effectively improves the average inspection frame rate of the video and optimizes the update frequency of the bounding boxes in the detected video. Figure 7 This diagram illustrates the impact of each layer as a segmentation point on the inspection frame rate when using int8 feature quantization. Combined with the analysis in Table 2, it shows that the segmentation decision method accurately selects layer 17 as the segmentation point. When layer 17 is the optimal segmentation point and feature quantization is not used, the average video detection frame rate is 4.84, a 19.8% improvement compared to the case where the task is completely offloaded to the server for computation. Furthermore, by using int8 feature quantization to optimize relay data transmission, the data size can be reduced to one-quarter of the original, and the corresponding transmission latency is also reduced to approximately one-quarter, resulting in a further 50.8% increase in the average detection frame rate. Overall, compared to the case where the task is completely offloaded to the server for computation, the average inspection frame rate is improved by 81%.

[0047] Table 2. Impact of Segmentation Point Decision and Feature Quantization on Object Detection Inference Performance

[0048]

[0049] Compared to existing technologies, this method reduces the size of relay feature data to one-quarter of its original size through int8 feature quantization. It considers the impact of fluctuating task loads on the computational latency of each layer of the DNN due to the hardware computing power of both the client and server in a dynamic environment, as well as the impact of network latency and bandwidth between the client and server on the transmission latency of data at different relay points. Furthermore, it uses an traversal optimization method to obtain the segmentation point with the lowest E2E inference latency as the objective, thus effectively improving the latency performance of the inference task. In summary, under the same network link performance, this invention can theoretically improve transmission latency performance by 4 times, effectively increasing the average detection frame rate.

[0050] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A model-segmentation-based application inference deployment method for edge computing, characterized in that, During the performance testing phase, network link latency and bandwidth, as well as the execution time of each layer of the neural network, were obtained through network performance testing, client-side inference task latency testing, and server-side inference task latency testing. The optimal segmentation point configuration was obtained through a segmentation point decision method. In the local inference phase, after acquiring the image, the current client executes the inference task before the segmentation point according to the optimal segmentation point configuration, and transmits the relay point data to the server located on the edge side after int8 feature quantization. In the edge inference phase, the server executes the edge inference after the segmentation point according to the same optimal segmentation point configuration and then transmits the results back to the client, which performs formatting processing and outputs the classification results and detection boxes. The optimal split point configuration refers to the logical layer position of the DNN model where the DNN inference split decision point, obtained through the split point decision method with the goal of achieving optimal latency performance, is located. The optimal split point configuration in both the local and edge inference stages is obtained using an improved DNN. This improved DNN includes an initialization module, a sequence transformation module, and a forward propagation logic module. Specifically: the initialization module inherits the feature extraction block, average convergence block, and feature classification block from the PyTorch official standard DNN and outputs them to the sequence transformation module; the sequence transformation module uses the ModuleList container provided by PyTorch to flatten all network layer blocks inherited from the original DNN by the initialization module. After flattening, all layer-level sub-modules are sequentially placed into a ModuleList container; the forward propagation logic module, based on the split point configuration information, controls the ModuleList container to stop inference at the specified DNN layer sub-module and returns the output tensor of that module. The aforementioned flattening process refers to splitting the network modules encapsulated in the PyTorch official standard DNN into layers and re-encapsulating them sequentially into the ModuleList container of the sequence transformation module, thereby enabling the acquisition of arbitrary network layer outputs during model segmentation in the inference phase.

2. The application inference deployment method based on model segmentation for edge computing according to claim 1, characterized in that, The aforementioned client-side inference task latency test and server-side inference task latency test refer to: the client and server running the DNN inference task of the AI ​​application in a trial run, and timing the execution time of each layer to measure the execution time of each layer; The execution time of each layer refers to the time taken to complete the same layer of inference under different performance conditions of the two hosts by testing and statistically analyzing the execution time of each layer of the DNN on both the client and server sides. The network performance test mentioned above refers to: measuring the round-trip latency of network links. The total network latency is obtained by adding the transmission latency to the total network latency, where the transmission latency is based on the real-time network bandwidth. With the volume of transmitted data The earlier the segmentation point, the larger the feature map needs to be transmitted, which will introduce a longer transmission time.

3. The application inference deployment method based on model segmentation for edge computing according to claim 1, characterized in that, The segmentation point configuration information refers to: sub-modules with layer granularity after flattening. and dividing points , of which: to The inference task of the layer network is executed locally on the client. Layer to Layer 1 networking executes at the server edge, specifically: when the client runs to the... When the layer is reached, the first The output tensor of the layer is the relay feature data in this segmentation task. After int8 feature quantization, it is sent to the server, making... The reasoning tasks after the layer are completed on the server side.

4. The application inference deployment method based on model segmentation for edge computing according to claim 1, characterized in that, The aforementioned split point decision method specifically includes: Step a, Input parameters include: the total number of layers in the DNN DNN network layer set The data volume of each layer's output after int8 feature quantization The execution time of each layer on the client side The execution time of each layer on the server side Round-trip latency of the network link between the client and the server Network bandwidth between the client and the server Data volume constant ; Step b: assemble each layer of the DNN Assuming the E2E delay performance is tested cyclically at the split point, the delay performance is expressed as follows: , loop through After +1 iterations, the split point with the lowest E2E latency is recorded as the optimal split point. The optimal split point configuration is output, where: For transmission delay, The execution time of each layer on the client side. The execution time of each layer on the server side. For data volume.

5. A system for implementing the application inference deployment method based on model segmentation for edge computing as described in any one of claims 1-4, characterized in that, include: The system comprises a control module, an image acquisition module, a local inference module, a local communication module, and an output processing module located on the client side, and an edge communication module and an edge inference module located on the server side. Specifically: During the testing phase, the control module obtains the execution time of each layer of the DNN through the local and edge inference modules, and obtains the network link latency and bandwidth through the local communication module. It then dynamically outputs the optimal segmentation point configuration to the inference module using a segmentation point decision method. The image acquisition module acquires images during the inference phase, converts them into tensor matrix form, and outputs them to the inference module. The local inference module on the client side contains a DNN with pre-trained weights, configured according to the optimal segmentation point. The DNN performs partial layer processing on the tensor matrix image and extracts relay feature data, which is then output to the local communication module. After int8 feature quantization, the data is output to the server. The output processing module receives the server-side inference results from the communication module and presents them in different ways depending on the task type. The edge inference module on the server receives and performs inverse int8 feature quantization on the relay feature data generated by the local inference module through the edge communication module. Based on the optimal segmentation point configuration generated by the control module, the built-in pre-trained DNN inputs the relay feature data into the specified network layer, performs inference, and returns the output results to the client through the edge communication module. The local inference module and the edge inference module are tested for hardware computing performance in advance and output the execution time of each layer of DNN on the client and server to the control module for optimizing the split point decision; The local communication module, after completing local inference, performs int8 feature quantization on the relay feature data output by the client's DNN, and together with the optimal split point configuration, formats it to generate a pickle-formatted binary bit stream, which is then output to the server via an HTTP request.

6. The system according to claim 5, characterized in that, The int8 feature quantization mentioned above refers to: asymmetric quantization based on linear quantization, i.e., based on the feature map matrix. The largest tensor in Minimum Tensor Determine the quantization coefficient and zero point Then rounded up After scaling and truncation functions, the original 4-byte feature map matrix of type float32 is reduced. Middle elements Feature parameters compressed to 1 byte of int8 type This forms a quantized feature map matrix. Specifically, this refers to: quantization coefficient midnight Feature parameters The truncation function is: , This indicates that the upper bound has been truncated. This indicates that the lower bound has been truncated. The aforementioned inverse int8 feature quantization processing refers to: the feature map matrix received by the server. Elements It will go through the inverse quantization formula Transform them one by one into 4-byte float32 type. , restore to feature matrix Specifically: , and The quantization coefficients and zeros obtained from the int8 feature quantization process on the same client side.

7. The system according to claim 5, characterized in that, The control module includes a network segmentation control unit and an image acquisition control unit. During the performance testing phase, the network segmentation control unit processes the segmentation point decision method based on the execution time of each layer of the DNN fed back by the local and edge inference modules, as well as the network link latency and bandwidth test results fed back by the local communication module, to obtain the optimal segmentation point configuration. The image acquisition control unit controls image decoding or video frame acquisition based on the information type obtained by the image acquisition module, thereby enabling the selection of image preprocessing methods and video acquisition frame rates.

8. The system according to claim 5, characterized in that, The local inference module includes a client-side computational performance testing unit and a local model segmentation inference unit. The client-side computational performance testing unit performs hardware computational performance testing according to the commands of the test phase control module and outputs the execution time of each layer of the DNN on the client to the control module. The local model segmentation inference unit performs inference of the network layers before the segmentation point of the DNN on the tensor matrix image according to the optimal segmentation point configuration output by the control module and extracts relay feature data, which is then sent to the server through the local communication module.

9. The system according to claim 5, characterized in that, The edge inference module includes a server-side computing performance testing unit and an edge model segmentation inference unit. The server-side computing performance testing unit performs hardware computing performance testing according to the commands of the testing phase control module and outputs the execution time of each layer of the DNN on the server to the control module. The edge model segmentation inference unit performs inference of the network layers after the segmentation point of the DNN according to the optimal segmentation point configuration and the relay point data after inverse int8 feature quantization received by the edge communication module and outputs the inference results to the client through the edge communication module.

Images