A Real-Time Instance Segmentation Method and System Based on Computational Offloading Mechanism

Abstract

This invention discloses a real-time instance segmentation method and system based on a computation offloading mechanism. The method includes: constructing a target network model; obtaining the current network bandwidth; determining the computation offloading point and feature map compression ratio based on the structure of the target network model and the size and computational cost of the feature maps output by each module; dividing the target network model into two parts using the computation offloading point as the splitting point; inputting data collected by a visual sensor on an IoT device into the front-end network model for computation to obtain intermediate result data; compressing the data according to a set compression ratio and downsampling method, and sending the compressed data to an edge server; the edge server decompressing the received data and inputting it into the back-end network model for computation to obtain inference results, which are then returned to the IoT device. This real-time instance segmentation method based on a computation offloading mechanism can run in real-time on resource-constrained devices.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and smart Internet of Things (IoT) technology, and in particular to a real-time instance segmentation method and system based on a computation offloading mechanism. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence in fields such as computer vision, speech recognition, and natural language processing, more and more intelligent information processing technologies have been integrated into traditional Internet of Things (IoT) systems, greatly stimulating the potential of IoT environmental perception and understanding. Instance segmentation, as one of the fundamental visual understanding tasks, can be widely applied in fields such as autonomous driving, robot control, assisted medical care, and remote sensing imagery.

[0003] In the early stages of instance segmentation research, researchers focused on improving inference accuracy. They continuously improved accuracy by deepening the backbone network, adding feature pyramid structures, and designing complex prediction modules, resulting in notable works such as FCIS, DeepMask, and Mask R-CNN. However, while improving inference accuracy, the computational complexity of instance segmentation methods also increased significantly, making their inference time much longer than other simpler visual understanding tasks, such as image classification and object detection. Until 2018, Mask R-CNN remained one of the fastest instance segmentation methods, achieving an inference speed of only 13.5 FPS (Frames Per Second) on a high-performance GPU (graphics processing unit) Titan Xp. In recent years, considering the high real-time requirements of many instance segmentation applications, more and more work has shifted its focus to researching real-time instance segmentation methods. In 2019, Bolya et al. proposed the first real-time instance segmentation method, which could only achieve real-time performance on high-performance GPU processors. However, due to factors such as cost, size, load capacity, and power consumption, conventional IoT devices simply cannot handle such bulky GPU processing units. Therefore, real-time instance segmentation for resource-constrained IoT devices remains a challenge.

[0004] Currently, most studies attempt to compress models using techniques such as pruning, distillation, and quantization to accelerate inference. However, this approach often sacrifices accuracy for faster inference, making it unsuitable for scenarios with high accuracy requirements and significantly limiting its application scope. In our previous work [Xie Y, Guo Y, Chen Y, et al. Real-Time Instance Segmentation for Low-Cost Mobile Robot SystemsBased on Computation Offloading[C] / / 2021 International Conference on Communications, Computing, Cybersecurity, and Informatics(CCCI).IEEE,2021:1-5], we first adopted a computation offloading mechanism to solve the real-time inference problem of instance segmentation networks on low-cost mobile robots, improving inference speed with almost no loss of accuracy. However, two problems remain: first, the data transmission volume during computation offloading is still large, resulting in high data communication costs; second, the impact of dynamic network environments on computation offloading strategies is not considered, making it impossible to meet latency requirements when network bandwidth is low. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time instance segmentation method, apparatus, and electronic device based on a computation offloading mechanism, in order to solve the problem that existing instance segmentation algorithms have high processor requirements and cannot run in real time on resource-constrained devices.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] A real-time instance segmentation method based on a computational offloading mechanism, wherein the method includes: adding an attention mechanism module before the downsampling layer in the backbone network of the target instance segmentation network, and training the target instance segmentation network with an input image training set to obtain a target network model;

[0008] Obtain the current network bandwidth, and determine the computation offload point and feature map compression ratio based on the structure of the target network model and the size and computational cost of the feature map output by each module;

[0009] Based on the determined computational offloading scheme, the target network model is divided into two parts with the computational offloading point as the dividing point. The front-end model is deployed on IoT devices, and the back-end model is deployed on edge servers. A communication link is established between the IoT devices and the edge servers.

[0010] The data collected by the visual sensor on the IoT device is input into the front-end network model for calculation to obtain intermediate result data;

[0011] The data is compressed using a preset downsampling compression method at a set compression ratio, and the compressed data is then sent to the edge server.

[0012] The edge server decompresses the received data and inputs the decompressed data into the backend network model for calculation to obtain inference results.

[0013] The edge server returns the inference result to the IoT device.

[0014] The steps of obtaining the current network bandwidth and determining the offload point and feature map compression ratio based on the structure of the target network model and the size and computational cost of the feature map output by each module include:

[0015] All downsampling layers of the backbone network in the instance segmentation network are selected as candidate computational unloading points;

[0016] Based on the current network bandwidth, the unloading point and feature map compression ratio are jointly optimized to maximize inference accuracy while meeting latency requirements.

[0017] The step of jointly optimizing the offload point and feature map compression ratio based on the current network bandwidth to maximize inference accuracy while meeting latency requirements includes:

[0018] Set the initial channel attention threshold and spatial attention threshold;

[0019] Calculate the data transfer volume at each candidate computation unloading point;

[0020] Calculate the data transmission time and total inference latency of the computation unloading scheme with each candidate computation unloading point as the final unloading point;

[0021] Determine whether the minimum inference latency among all current candidate computation offloading schemes meets the latency requirement;

[0022] If satisfied, output the current computational unloading point, channel attention threshold, and spatial attention threshold.

[0023] The method further includes, after the step of determining whether the minimum inference latency among all current candidate computation offloading schemes meets the latency requirement:

[0024] If not, the channel attention threshold is increased by the first preset time interval, and the total inference delay of each candidate computation offloading scheme is recalculated until the inference delay of one scheme meets the latency requirement.

[0025] When the channel attention threshold is greater than or equal to the maximum channel attention threshold, if no computation offloading scheme that meets the latency requirements is determined, the computation offloading scheme with the minimum current inference latency will be used as the temporary computation offloading scheme.

[0026] The spatial attention threshold is increased by the first preset time interval, and the total inference delay of the current temporary computation unloading scheme is recalculated until the scheme meets the delay requirements and the computation unloading point, channel attention threshold and spatial attention threshold that meet the delay requirements are output.

[0027] The step of compressing the data using a preset downsampling compression method at a set compression ratio includes:

[0028] The attention module is used to obtain the channel attention weights and spatial attention weights of the feature map;

[0029] Based on the activation function, activate certain channels and spatial regions of the feature map;

[0030] Perform a preset downsampling operation on the active channels in the feature map to obtain a compressed feature map.

[0031] The step of performing a preset downsampling operation on the activated channels in the feature map to obtain a compressed feature map includes:

[0032] For each active channel, extract the active spatial region and perform the first downsampling operation to obtain compressed active region information;

[0033] The feature values ​​of the active spatial region are replaced with 0, and a second downsampling operation with a double step size is performed on the entire active channel to obtain compressed information of the non-active region.

[0034] The step of the edge server decompressing the received data includes:

[0035] The edge server uses pixel interpolation to restore the width and height of the feature map to their original dimensions before compression.

[0036] The inactive channels are reconstructed with zero padding, and then inserted according to the inactive channel index to restore the number of channels in the feature map to the uncompressed dimension.

[0037] The step of restoring the width and height of the feature map to their original dimensions using pixel interpolation by the edge server includes:

[0038] The compressed inactive region information is restored to matrix form;

[0039] The matrix width and height are restored to their original dimensions using nearest neighbor interpolation.

[0040] Based on the activation region mask, the compressed activation region information is overlaid on the activation region to obtain the restored activation channel.

[0041] A real-time instance segmentation system based on a computation offloading mechanism, the system comprising IoT devices and edge servers;

[0042] The IoT device adds an attention mechanism module before the downsampling layer in the backbone network of the target instance segmentation network, and trains the target instance segmentation network by inputting an image training set to obtain the target network model.

[0043] Obtain the current network bandwidth, and determine the computation offload point and feature map compression ratio based on the structure of the target network model and the size and computational cost of the feature map output by each module;

[0044] Based on the determined computational offloading scheme, the target network model is divided into two parts with the computational offloading point as the dividing point. The front-end model is deployed on IoT devices, and the back-end model is deployed on edge servers. A communication link is established between the IoT devices and the edge servers.

[0045] The data collected by the visual sensor on the IoT device is input into the front-end network model for calculation to obtain intermediate result data;

[0046] The data is compressed using a preset downsampling compression method at a set compression ratio, and the compressed data is then sent to the edge server.

[0047] The edge server decompresses the received data and inputs the decompressed data into the backend network model for calculation to obtain inference results.

[0048] The edge server returns the inference result to the IoT device.

[0049] The real-time instance segmentation method based on a computation offloading mechanism provided in this invention has several advantages. First, it effectively leverages the powerful computing and storage resources of edge servers to solve the problem of real-time operation of instance segmentation networks on resource-constrained IoT devices by utilizing the computation offloading mechanism. Second, it designs efficient and easily integrated data compression and decompression methods based on the instance segmentation network structure to optimize data transmission volume. By jointly optimizing the computation offloading point and data compression ratio, it better balances computation offloading volume and data communication cost. Third, it fully considers the impact of dynamic network environment on the computation offloading scheme and adaptively adjusts the computation offloading scheme according to the real-time network bandwidth to maximize inference accuracy while meeting latency requirements. Attached Figure Description

[0050] Figure 1 A flowchart illustrating the steps of a real-time instance segmentation method based on a computational offloading mechanism provided in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of candidate computation offloading points for the Resnet-50 backbone network provided in an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of the compression method based on improved downsampling operation provided in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the improved downsampling operation process provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the decompression method based on pixel and channel interpolation provided in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the pixel interpolation operation process provided in an embodiment of the present invention;

[0056] Figure 7 This is a schematic diagram illustrating an example of instance segmentation results provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0058] The real-time instance segmentation scheme based on the computational unloading mechanism provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0059] The real-time instance segmentation method based on a computational offloading mechanism in this application includes the following steps:

[0060] Step 101: Add an attention mechanism module before the downsampling layer in the backbone network of the target instance segmentation network, and train the target instance segmentation network using the input image training set to obtain the target network model.

[0061] In practical implementation, Yolac can be selected as the target instance segmentation network. The number of images in the image test set can be flexibly set by those skilled in the art, such as 100, 150, or 200 images, etc., and no specific limitation is made in this embodiment.

[0062] Step 102: Obtain the current network bandwidth, and determine the computation offload point and feature map compression ratio based on the structure of the target network model and the size and computational cost of the feature map output by each module.

[0063] A feasible way to obtain the current network bandwidth and determine the computation offload point and feature map compression ratio based on the structure of the target network model and the size and computational cost of the feature map output by each module can be as follows:

[0064] First, all downsampling layers of the backbone network in the instance segmentation network are selected as candidate computational offload points;

[0065] The backbone network can use ResNet-50, and all its candidate computation offload points are as follows: Figure 2 As shown.

[0066] Secondly, based on the current network bandwidth, the unloading point and feature map compression ratio are jointly optimized to maximize inference accuracy while meeting latency requirements.

[0067] Furthermore, based on the current network bandwidth, jointly optimizing the offload point and feature map compression ratio to maximize inference accuracy while meeting latency requirements can include the following steps:

[0068] S1: Set the initial channel attention threshold and spatial attention threshold;

[0069] Initial channel attention threshold in the embodiments of this application Spatial attention threshold Set all to 0.

[0070] S2: Calculate the data transfer volume at each candidate computation unloading point;

[0071] In this embodiment of the invention, the data transmission volume at each candidate computational offload point is calculated using the following formula:

[0072]

[0073] Here, we estimate the data transmission volume using 100 test images. Specifically, we calculate the average data transmission volume of these 100 test images at each candidate computational unloading point, and use this average as the estimated data transmission volume. Where d... F0 A is the size of the original feature map. c and A s These are the channel attention values ​​and spatial attention values ​​of the feature map, respectively, f c (·) and f s (·) represent the channel activation function and the spatial activation function, respectively, as shown in formulas (2) and (3), where τ c and τ s These are the channel attention threshold and the spatial attention threshold, respectively.

[0074]

[0075]

[0076] S3: Calculate the data transmission time and total inference latency of the computation unloading scheme with each candidate computation unloading point as the final unloading point;

[0077] Data transmission time can be calculated using formula (4), where B is the current network bandwidth.

[0078]

[0079] Total inference latency can be calculated using the following formula:

[0080]

[0081] in, This refers to the execution time of each module in the front-end network model on the IoT device. This represents the execution time of each module in the backend network model on the edge server.

[0082] S4: Determine whether the minimum inference latency among all candidate computation offloading schemes meets the latency requirement; if it does, proceed to S5; if not, proceed to S6.

[0083] S5: If satisfied, output the current calculation unloading point, channel attention threshold, and spatial attention threshold;

[0084] S6: If not satisfied, increase the channel attention threshold by the first preset time interval, recalculate the total inference delay of each candidate computation offloading scheme, until there is a scheme whose inference delay meets the delay requirement.

[0085] The first preset time interval can be represented as Δτ c ,Δτ cThe specific value can be flexibly set by those skilled in the art, for example, set to 0.05.

[0086] S7: When the channel attention threshold is greater than or equal to the maximum channel attention threshold If no computational offloading scheme that meets the latency requirements is determined, the computational offloading scheme with the lowest current inference latency will be used as the temporary computational offloading scheme.

[0087] In the embodiments of this application, and Δτ s They were set to 0.5 and 0.05 respectively.

[0088] S8: Increase the spatial attention threshold by the first preset time interval, recalculate the total inference delay of the current temporary computation unloading scheme, until the scheme meets the delay requirements and outputs the computation unloading point, channel attention threshold and spatial attention threshold that meet the delay requirements.

[0089] Step 103: Based on the determined computational unloading scheme, the target network model is divided into two parts using the computational unloading point as the dividing point.

[0090] The front-end model is deployed on IoT devices, and the back-end model is deployed on edge servers, with a communication link established between the IoT devices and the edge servers.

[0091] Step 104: Input the data collected by the visual sensor on the IoT device into the front-end network model for calculation to obtain intermediate result data.

[0092] Step 105: Compress the data using a preset downsampling compression method at a set compression ratio, and then send the compressed data to the edge server.

[0093] The preset sampling compression method is an improved downsampling compression method. The process of compressing data based on the improved downsampling compression method is as follows: Figure 3 As shown.

[0094] The steps of compressing data using a preset downsampling compression method at a set compression ratio include:

[0095] S1: Use the attention module to obtain the channel attention weights and spatial attention weights of the feature map;

[0096] In this embodiment of the invention, the attention module used is a Convolutional Block Attention Module (CBAM).

[0097] S2: Activate some channels and spatial regions of the feature map according to the activation function;

[0098] The activation function is the same as in formulas (2) and (3).

[0099] S3: Perform a preset downsampling operation on the active channels in the feature map to obtain a compressed feature map.

[0100] The compressed feature map includes compressed active and inactive region information, inactive channel index, and active region mask.

[0101] Furthermore, referring to Figure 4 The improved downsampling operation procedure shown includes:

[0102] S1: For each active channel, extract the active spatial region and perform the first downsampling operation to obtain compressed active region information;

[0103] S2: Replace the feature values ​​of the active spatial region with 0, and perform a second downsampling operation with a double step size on the entire active channel to obtain compressed non-active region information.

[0104] One feasible approach is to further compress the sparse matrix using Compressed Sparse Row (CSR) technology to ultimately obtain compressed inactive region information.

[0105] Step 106: The edge server decompresses the received data and inputs the decompressed data into the backend network model for calculation to obtain the inference result.

[0106] Reference Figure 5 The flowchart of the pixel- and channel-based decompression method is shown in the figure. The decompression operation of the received data includes the following steps:

[0107] S1: The edge server uses pixel interpolation to restore the width and height of the feature map to their original dimensions before compression;

[0108] S2: Zero-padding reconstruction of inactive channels, and insertion of inactive channels according to the inactive channel index, restoring the number of channels of the feature map to the uncompressed dimension.

[0109] Reference Figure 6 As shown, the edge server uses pixel interpolation to restore the width and height of the feature map to their original dimensions before compression in the following ways:

[0110] First, the compressed inactive region information is restored to matrix form; second, the width and height of the matrix are restored to their original dimensions using nearest neighbor interpolation; finally, the compressed active region information is overlaid on the active region according to the active region mask to obtain the restored active channel.

[0111] Step 107: The edge server returns the inference results to the IoT device.

[0112] In practical implementation, the real-time instance segmentation method described in this application was deployed on the commonly used IoT platform Nvidia Jetson TX2. The experimental results shown in Table 1 show that the real-time instance segmentation method based on the computation offloading mechanism proposed in this application can achieve a faster inference speed (70.59ms) with a lower accuracy loss (<6%) compared to other existing methods. Among them, YOLACT is the current mainstream instance segmentation method, YolactEdge is a competitive instance segmentation method that can run on mobile devices, MobileNetv2-YOLACT and MobileNetv2-YolactEdge are instance segmentation methods after replacing the backbone network of YOLACT and YolactEdge with the lightweight network MobileNetv2, respectively, Yolact_allEdge is a method that offloads the entire instance segmentation network to the edge server for execution, YolactCOS is the work in [Xie Y, Guo Y, Chen Y, et al. Real-Time Instance Segmentation for Low-Cost Mobile Robot Systems Based on Computation Offloading[C] / / 2021 International Conference on Communications, Computing, Cybersecurity, and Informatics(CCCI).IEEE,2021:1-5], and YolactACOS is the real-time instance segmentation method based on the computation offloading mechanism proposed in this application.

[0113] Table 1 compares the inference latency and accuracy of multiple instance segmentation methods.

[0114]

[0115] Figure 7The examples are the original YOLACT network, the network after lightweight processing (MobileNetv2-YOLACT), and the instance segmentation results of the real-time instance segmentation method based on the computation offloading mechanism proposed in this invention (YolactACOS), which correspond to (1), (2), and (3) in the figure, respectively. It can be seen that, unlike the traditional lightweight processing method, the method proposed in this invention does not lose much accuracy and achieves a performance comparable to the original YOLACT network. The real-time instance segmentation method based on the computation offloading mechanism provided in this application has the following advantages: First, it effectively utilizes the powerful computing and storage resources on the edge server to solve the problem of real-time operation of the instance segmentation network on resource-constrained IoT devices by leveraging the computation offloading mechanism. Second, it designs efficient and easily integrated data compression and decompression methods based on the instance segmentation network structure to optimize data transmission volume and better balance computation offloading volume and data communication cost by jointly optimizing the computation offloading point and data compression ratio. Third, it fully considers the impact of the dynamic network environment on the computation offloading scheme and adaptively adjusts the computation offloading scheme according to the real-time network bandwidth to maximize inference accuracy while meeting latency requirements.

[0116] This application also provides a real-time instance segmentation system based on a computation offloading mechanism, which includes IoT devices and edge servers;

[0117] The IoT device adds an attention mechanism module before the downsampling layer in the backbone network of the target instance segmentation network, and trains the target instance segmentation network by inputting an image training set to obtain the target network model.

[0118] Obtain the current network bandwidth, and determine the computation offload point and feature map compression ratio based on the structure of the target network model and the size and computational cost of the feature map output by each module;

[0119] Based on the determined computational offloading scheme, the target network model is divided into two parts with the computational offloading point as the dividing point. The front-end model is deployed on IoT devices, and the back-end model is deployed on edge servers. A communication link is established between the IoT devices and the edge servers.

[0120] The data collected by the visual sensor on the IoT device is input into the front-end network model for calculation to obtain intermediate result data;

[0121] The data is compressed using a preset downsampling compression method at a set compression ratio, and the compressed data is then sent to the edge server.

[0122] The edge server decompresses the received data and inputs the decompressed data into the backend network model for calculation to obtain inference results.

[0123] The edge server returns the inference result to the IoT device.

[0124] Preferably, when the IoT device determines the computation offload point and feature map compression ratio based on the structure of the target network model and the size and computational load of the feature map output by each module, the acquisition of the current network bandwidth is specifically used for:

[0125] All downsampling layers of the backbone network in the instance segmentation network are selected as candidate computational unloading points;

[0126] Based on the current network bandwidth, the unloading point and feature map compression ratio are jointly optimized to maximize inference accuracy while meeting latency requirements.

[0127] Preferably, the IoT device, based on the current network bandwidth, jointly optimizes the calculation of the offload point and the feature map compression ratio, so as to maximize inference accuracy while meeting latency requirements. Specifically, it is used for:

[0128] Set the initial channel attention threshold and spatial attention threshold;

[0129] Calculate the data transfer volume at each candidate computation unloading point;

[0130] Calculate the data transmission time and total inference latency of the computation unloading scheme with each candidate computation unloading point as the final unloading point;

[0131] Determine whether the minimum inference latency among all current candidate computation offloading schemes meets the latency requirement;

[0132] If satisfied, output the current computational unloading point, channel attention threshold, and spatial attention threshold.

[0133] Preferably, after determining whether the minimum inference latency among all current candidate computation offloading schemes meets the latency requirement, the IoT device is further configured to:

[0134] If not, the channel attention threshold is increased by the first preset time interval, and the total inference delay of each candidate computation offloading scheme is recalculated until the inference delay of one scheme meets the latency requirement.

[0135] When the channel attention threshold is greater than or equal to the maximum channel attention threshold, if no computation offloading scheme that meets the latency requirements is determined, the computation offloading scheme with the minimum current inference latency will be used as the temporary computation offloading scheme.

[0136] The spatial attention threshold is increased by the first preset time interval, and the total inference delay of the current temporary computation unloading scheme is recalculated until the scheme meets the delay requirements and the computation unloading point, channel attention threshold and spatial attention threshold that meet the delay requirements are output.

[0137] Preferably, when the IoT device compresses data using a preset downsampling compression method at a set compression ratio, it is specifically used for:

[0138] The attention module is used to obtain the channel attention weights and spatial attention weights of the feature map;

[0139] Based on the activation function, activate certain channels and spatial regions of the feature map;

[0140] Perform a preset downsampling operation on the active channels in the feature map to obtain a compressed feature map.

[0141] Preferably, when the IoT device performs a preset downsampling operation on the activated channels in the feature map to obtain a compressed feature map, it is specifically used for:

[0142] For each active channel, extract the active spatial region and perform the first downsampling operation to obtain compressed active region information;

[0143] The feature values ​​of the active spatial region are replaced with 0, and a second downsampling operation with a double step size is performed on the entire active channel to obtain compressed information of the non-active region.

[0144] Preferably, when the edge server performs decompression on the received data, it specifically uses the following methods:

[0145] The edge server uses pixel interpolation to restore the width and height of the feature map to their original dimensions before compression.

[0146] The inactive channels are reconstructed with zero padding, and then inserted according to the inactive channel index to restore the number of channels in the feature map to the uncompressed dimension.

[0147] Optionally, when the edge server uses pixel interpolation to restore the width and height of the feature map to their original dimensions before compression, it is specifically used for:

[0148] The compressed inactive region information is restored to matrix form;

[0149] The matrix width and height are restored to their original dimensions using nearest neighbor interpolation.

[0150] Based on the activation region mask, the compressed activation region information is overlaid on the activation region to obtain the restored activation channel.

[0151] The real-time instance segmentation system based on a computation offloading mechanism provided in this application addresses the real-time operation of instance segmentation networks on resource-constrained IoT devices by leveraging the powerful computing and storage resources of edge servers. Secondly, it designs efficient and easily integrated data compression and decompression methods based on the instance segmentation network structure to optimize data transmission volume. Furthermore, it better balances computation offloading volume and data communication costs by jointly optimizing computation offloading points and data compression ratios. Thirdly, it fully considers the impact of dynamic network environments on the computation offloading scheme, adaptively adjusting the scheme based on real-time network bandwidth to maximize inference accuracy while meeting latency requirements.

[0152] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0153] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A real-time instance segmentation method based on a compute offload mechanism, characterized in that, The method comprises: adding an attention mechanism module before a down-sampling layer in a backbone network of a target instance segmentation network, and inputting a picture training set to train the target instance segmentation network to obtain a target network model; obtaining a current network bandwidth, determining a calculation offloading point and a feature map compression ratio according to a structure of the target network model and a feature map size and a calculation amount output by each module, comprising: selecting all down-sampling layers of the backbone network in the instance segmentation network as candidate calculation offloading points; jointly optimizing the calculation offloading point and the feature map compression ratio according to the current network bandwidth, comprising: S1: setting an initial channel attention threshold and a spatial attention threshold; wherein the initial channel attention threshold and the spatial attention threshold are both set to 0; S2: calculating data transmission amounts at the candidate calculation offloading points; wherein, is the original feature map size, A c and A s are the channel attention value and the spatial attention value of the feature map, respectively, f c (·) and f s (·) are the channel activation function and the spatial activation function, respectively, as shown in equations (2) and (3), wherein τ c and τ s are the channel attention threshold and the spatial attention threshold, respectively; S3: calculating data transmission times and total inference delays of the calculation offloading schemes with the candidate calculation offloading points as the final offloading points; the data transmission time is calculated by formula (4), wherein B is the current network bandwidth; the total inference delay is calculated by the following formula: wherein, is the execution time of each module in the front network model on the Internet of Things device, is the execution time of each module in the back network model on the edge server; S4: determining whether the smallest inference delay in all candidate calculation offloading schemes meets a delay requirement; if yes, performing S5; S5: outputting the current calculation offloading point, the channel attention threshold and the spatial attention threshold; according to the determined calculation offloading scheme, the target network model is divided into two parts with the calculation offloading point as a split point, wherein a front model is deployed on an Internet of Things device, a rear model is deployed on an edge server, and a communication link is established between the Internet of Things device and the edge server; data collected by a visual sensor on the Internet of Things device is transmitted to the front network model for calculation to obtain intermediate result data; a preset down-sampling compression method is used to compress the data according to a set compression ratio, and the compressed data is sent to the edge server; the edge server decompresses the received data, transmits the decompressed data to the rear network model for calculation to obtain an inference result; the edge server returns the inference result to the Internet of Things device.

2. The method of claim 1, wherein, After the step of determining whether the smallest inference delay in all candidate calculation offloading schemes meets a delay requirement, the method further comprises: if not, increasing the channel attention threshold by a first preset time interval, recalculating the total inference delays of the candidate calculation offloading schemes, until there is a scheme whose inference delay meets the delay requirement; when the channel attention threshold is greater than or equal to a maximum channel attention threshold, if a calculation offloading scheme meeting the delay requirement is not determined, a calculation offloading scheme with the smallest current inference delay is taken as a temporary calculation offloading scheme; increasing the spatial attention threshold by the first preset time interval, recalculating the total inference delay of the current temporary calculation offloading scheme, until the scheme meets the delay requirement and outputs the calculation offloading point, the channel attention threshold and the spatial attention threshold meeting the delay requirement.

3. The method of claim 1, wherein, the step of using a preset down-sampling compression method to compress data according to a set compression ratio, comprising: using an attention module to obtain channel attention weights and spatial attention weights of a feature map; According to the activation function, activate the partial channel and spatial region of the feature map; Perform a preset downsampling operation on the activated channel in the feature map to obtain a compressed feature map.

4. The method of claim 3, wherein, The step of performing a preset downsampling operation on the activated channel in the feature map to obtain a compressed feature map comprises: For each activated channel, extract the activated spatial region and perform a first downsampling operation to obtain compressed activated region information; Replace the feature values of the activated spatial region with 0, and perform a second downsampling operation with double step size on the entire activated channel to obtain compressed non-activated region information.

5. The method of claim 1, wherein, The step of decompressing the received data by the edge server comprises: The edge server restores the width and height of the feature map to the dimension before compression by pixel interpolation; Zero padding reconstructs the non-activated channel, and inserts the non-activated channel according to the non-activated channel index to restore the number of channels of the feature map to the dimension before compression.

6. The method of claim 5, wherein, The step of restoring the width and height of the feature map to the dimension before compression by pixel interpolation by the edge server comprises: Restore the compressed non-activated region information to matrix form; Restore the width and height of the matrix to the dimension before compression by the nearest neighbor interpolation method; According to the activated region mask, cover the compressed activated region information in the activated region to obtain the restored activated channel.

7. A real-time instance segmentation system based on a compute offload mechanism, configured to implement the method of any one of claims 1-6, characterized in that, The system comprises an Internet of Things device and an edge server; The Internet of Things device adds an attention mechanism module before the downsampling layer in the backbone network of the target instance segmentation network, and inputs a picture training set to train the target instance segmentation network to obtain a target network model; Obtain the current network bandwidth, and determine the calculation offloading point and the feature map compression ratio according to the structure of the target network model and the feature map size and calculation amount output by each module; According to the determined calculation offloading scheme, the target network model is divided into two parts at the calculation offloading point, wherein the front-end model is deployed on the Internet of Things device, and the rear-end model is deployed on the edge server, and a communication link is established between the Internet of Things device and the edge server; The data collected by the visual sensor on the Internet of Things device is transmitted into the front-end network model for calculation to obtain intermediate result data; A preset downsampling compression method is used to compress the data according to the set compression ratio, and the compressed data is sent to the edge server; The edge server decompresses the received data, transmits the decompressed data into the rear-end network model for calculation to obtain an inference result; The edge server returns the inference result to the Internet of Things device.

Images