Deep learning task execution system and method based on mobile phone cooperative embedded device

Abstract

The application relates to a deep learning task execution system and method based on a mobile phone and a collaborative embedded device, the system comprising an embedded device (used as a task subject) and a smart phone (used as a collaborative device) which are connected in communication, and based on a preset pipeline collaboration scheme, parallel collaborative processing of a deep learning task is realized between the two. The method comprises the following steps: a deep learning network model is constructed in a training set pruning manner; communication time, data processing time of the embedded device and the smart phone in a current task scene are determined, and a pipeline scheme is determined in combination with a set judgment condition; according to the pipeline scheme, the deep learning network model is carried on the embedded device or the smart phone, input data is received by the embedded device, and the deep learning task is collaboratively executed by the embedded device and the smart phone. Compared with the prior art, the application is suitable for various deep learning task scenes, can accelerate the task execution speed, ensure real-time performance and improve the task execution effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning task scheduling, and in particular to a deep learning task execution system and method based on mobile phone collaborative embedded devices. BACKGROUND

[0002] With the continuous development of deep learning technology, its application scenarios in real life are becoming more and more extensive. However, it is worth noting that due to the relatively lagging development of hardware platforms, when facing expensive embedded platform computing resources, deep learning tasks are seriously hindered in actual application. At present, in engineering practice, the way of reducing model calculation complexity is often adopted, but it is easy to lead to the fact that deep learning tasks cannot obtain accurate and reliable results.

[0003] Nowadays, deep learning tasks have been widely used on embedded devices. The main embedded devices for realizing fast model inference mainly have the following three architectures: 1) device-side calculation, in which the model runs in the embedded device; 2) edge server-based architecture, in which the data in the embedded device is sent to one or more edge servers for operation; and 3) joint calculation among the embedded device, the edge server and the cloud. The three architectures are as shown in Figures 1a-1c

[0004] In the device-side calculation architecture, researchers mainly realize the deployment of the model on the embedded device and reduce the inference delay from the aspects of algorithm and hardware. In terms of algorithm, researchers mainly focus on how to compress the storage of the parameters of the model and reduce the number of model parameters to accelerate the inference speed of the model. At the beginning of the design of the model, considering the embedded application scenario, researchers usually take the calculation time complexity and the model accuracy as two very important standards. Such models typically include MobileNets, SqueezeNet and YOLO, and such lightweight models are still under continuous research. MobileNets decomposes the convolution filter into two simpler operations, thereby reducing the number of calculations required. YOLO is a classic one-stage object detection model, which can jointly predict the position and category of an object in a single detection. SqueezeNet reduces the number of parameters by replacing the convolution kernel size and reducing the number of convolution channels. In addition to artificially designing lightweight neural network models, researchers are also exploring how to compress existing networks to realize running on embedded devices. According to the redundancy of neural networks in different aspects, methods such as network branch pruning, reducing the number of bits occupied by network weights, low-rank decomposition of convolution kernels, and knowledge distillation are used to compress existing network models. These methods can reduce the requirements for space and computing power, and realize inference on edge devices.

[0005] ​In hardware, there are studies that propose integrating special-purpose hardware accelerators for neural processing in embedded edge devices. As special-purpose processing units, accelerators provide faster and more energy-efficient computing solutions for neural network model applications. Typical neural network accelerators found in embedded devices include the tensor cores of NVIDIA Jetson AGX Xavier GPUs, the VLIW vector processors in Intel Myriad X VPUs, the multiply-accumulate (MAC) units of Tesla FSD chips, and the systolic arrays of Google Edge TPUs. The execution of these neural accelerators generally follows the host-device model, where a host CPU initiates a series of neural computation kernels for an application and then offloads them to a device accelerator that implements a neural network optimized data path. There are also researchers who explore the parallel capabilities within the computing platform, splitting the workload into pipeline stages and using multiple processing units to overlap their execution. For example, Yang et al. proposed a pipelined execution method to increase the system throughput of computer vision applications using multiple GPUs. PipeDream attempts to use multiple GPUs in HPC environments to utilize inner-slice and network pipelining methods for neural network training. However, this multi-GPU solution has limited effectiveness as an acceleration solution for neural network models in resource-constrained embedded devices.

[0006] In edge server computing architectures, all computations are offloaded from embedded devices to edge servers. In this architecture, embedded devices send their data to nearby edge servers and receive the corresponding results after server processing. For example, Wang et al. offload models to edge servers (IoT gateways) to analyze wireless signals. At this time, the embedded device is mainly responsible for data preprocessing. In Glimpse, all model computations are offloaded to edge servers, and the embedded device filters out some useless frames by detection. Liu et al. constructed a food recognition system through two preprocessing steps: first, discard blurry images; second, crop images to only include the region of interest. Although this architecture can reduce the computational and storage pressure on embedded devices to the greatest extent, it only transfers the cost to edge servers. Especially when edge servers handle the demands of several embedded devices at the same time, real-time requirements cannot be guaranteed.

[0007] In the architecture of combining cloud for deep learning computing, the superior computing in the cloud server is fully utilized. Li et al. divide the DNN model into two parts, the edge server computes the initial layer of the DNN model, and the cloud computes the higher layer of the DNN. After the edge server receives the data, it processes the lower layer of the DNN, and then sends the intermediate result to the cloud. After the upper layer of the cloud, the final result is sent back to the terminal device. DDNN also distributes the computation in the hierarchy of cloud, edge server and terminal device, and combines it with the idea of fast exit, so the computing request does not always reach the cloud. When using this framework, there are fewer communication methods available, and the communication environment is required to be higher. When the cloud receives a large number of model computing tasks, the load of the cloud server is too high, which easily causes the server to be down, and cannot guarantee the real-time performance of the model running task on the embedded device. SUMMARY

[0008] The purpose of the present application is to overcome the defects of the prior art and provide a deep learning task execution system and method based on mobile phone cooperation with embedded device, which can speed up task execution, ensure real-time performance and improve task execution effect.

[0009] The purpose of the present application can be achieved by the following technical scheme: a deep learning task execution system based on mobile phone cooperation with embedded device, comprising an embedded device and a smart phone connected with each other for communication, the embedded device is used as a task subject, and the smart phone is used as a cooperation device, and the embedded device and the smart phone realize parallel cooperative processing of deep learning tasks based on a preset pipeline cooperation scheme.

[0010] Further, the communication connection mode between the embedded device and the smart phone includes but is not limited to local area network, Bluetooth, USB (Universal Serial Bus, Universal Serial Bus).

[0011] Further, the preset pipeline cooperation scheme includes a first scheme to a fourth scheme, the first scheme specifically is that the embedded device pre-processes input data, the smart phone performs model inference and post-processing;

[0012] The second scheme specifically is that the embedded device pre-processes and infers the input data, and the smart phone performs post-processing;

[0013] The third scheme specifically is that the embedded device directly transmits the input data to the smart phone for pre-processing, and then the embedded device performs model inference and post-processing;

[0014] The fourth scheme specifically is that the embedded device directly transmits the input data to the smart phone for pre-processing and model inference, and then the embedded device performs post-processing.

[0015] A deep learning task execution method based on mobile phone collaborative embedded device, comprising the following steps:

[0016] S1, a deep learning network model is constructed by pruning the training set;

[0017] S2, determine the communication time, data processing time of the embedded device and the smart phone in the current task scene, and determine the pipeline scheme combined with the set judgment condition;

[0018] S3, according to the determined pipeline scheme, the deep learning network model is carried on the embedded device or the smart phone, the input data is received by the embedded device, and the deep learning task is executed cooperatively with the smart phone.

[0019] Further, the step S1 specifically comprises the following steps:

[0020] S11, train the initialized model and make it reach basic convergence;

[0021] S12, input the entire training set to the current model and calculate the activation value of each layer of neurons under the training set;

[0022] S13, based on the set pruning ratio, calculate the threshold value of neuron pruning for each layer structure in the model, delete the neurons with activation value lower than the corresponding threshold value, and correspondingly delete the weights of the upper and lower layers;

[0023] S14, further train the pruned model until convergence.

[0024] Further, the step S13 specifically prunes the full connection layer structure and the convolution layer structure in the model.

[0025] Further, the communication time in the step S2 under the current task scene includes the transmission time t p of the preprocessed data, the transmission time t output of the model output result, the sending time t img of the original data and the receiving time t rf of the task result.

[0026] Further, the data processing time of the embedded device in the step S2 includes the preprocessing time t ep of the embedded device, the post-processing time t ef of the embedded device and the model inference time t ec of the embedded device.

[0027] Further, the data processing time of the smart phone in the step S2 includes the preprocessing time t mp, the post-processing time t of the smart phone mf , the model inference time t of the smart phone mc .

[0028] Further, the judgment condition set in the step S2 is specifically:

[0029] If t ep ≥ t p + t rf + t mc + t mf , the pipeline scheme is determined as the first scheme, that is, the embedded device pre-processes the input data, the smart phone performs model inference and post-processing;

[0030] If t ep + t ec ≥ t output + t rf + t mf , the pipeline scheme is determined as the second scheme, that is, the embedded device pre-processes and performs model inference on the input data, and the smart phone performs post-processing;

[0031] If t ec + t ef ≥ t img + t p + t mp , the pipeline scheme is determined as the third scheme, that is, the embedded device directly transmits the input data to the smart phone for pre-processing, and then the embedded device performs model inference and post-processing;

[0032] If t ef ≥ t img + t output + t mp + t mc , the pipeline scheme is determined as the fourth scheme, that is, the embedded device directly transmits the input data to the smart phone for pre-processing and model inference, and then the embedded device performs post-processing.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] One, the present application takes into account that the current smart phone has become one of the most widely used and rich in computing resources in society, its unique private property also makes it have the characteristics of fast updating speed and idle computing resources. Therefore, the design of the present application cooperates the smart phone and the embedded device which have private property at the same time, which can improve the task execution effect of the embedded device without affecting the normal use of the smart phone. The present application uses the embedded device as the task subject and the smart phone as the cooperative device, and makes the embedded device and the smart phone based on the preset pipeline cooperation scheme to realize parallel collaborative processing of deep learning tasks, which can speed up the task execution speed, ensure real-time performance and improve the task execution effect.

[0035] Two, the present application proposes a scheme for parallel cooperation reasoning of mobile phone and embedded device in pipeline mode, and in order to make the pipeline technology more widely applied to various deep learning models, a model pruning method is also proposed, which can prune specific convolutional layers and fully connected layers in the model in the training stage only relying on the training set, which can effectively reduce the model size and reduce the model reasoning time, and further speed up the task execution speed.

[0036] Three, the present application divides the task into three stages: preprocessing, model reasoning and model result post-processing, and fully considers that the embedded device and the smart phone have different processing speeds for each stage of the task, and the calculation time needs to be allocated to the calculation task. Therefore, the present application calculates the communication time and the data processing time of different stages in the task, and combines the set judgment condition to formulate the pipeline scheme design standard, that is, by extracting the key factors of the pipeline, and then making reasonable pipeline scheme according to these factors, it is ensured that the formulated pipeline scheme can be well applied to the current task scene and demand, and the task execution effect is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1a For the existing device end computing architecture;

[0038] Figure 1b For the existing edge server based architecture

[0039] Figure 1c For the existing embedded device, edge server and cloud collaborative computing architecture;

[0040] Figure 2 For the system architecture diagram of the second scheme in the present application;

[0041] Figure 3 For the method flow diagram of the present application;

[0042] Figure 4 For the model pruning process diagram in the present application;

[0043] Figure 5 A timing diagram for pipelined computation of mobile phone-embedded devices;

[0044] Figure 6 This is a schematic diagram of the lane line detection task process in the embodiment;

[0045] Figure 7 This is a schematic diagram comparing the data processing time of each stage of the lane detection task in different devices in the embodiment.

[0046] Figure 8 This is a schematic diagram comparing the communication time of the lane line detection task under different communication environments in the embodiment;

[0047] Figure 9 This is a schematic diagram comparing the execution time of lane detection tasks under different pipeline schemes in the embodiment. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0049] Example

[0050] A deep learning task execution system based on a mobile phone and an embedded device includes an embedded device and a smartphone that are interconnected (the communication connection methods include, but are not limited to, local area network, Bluetooth, and USB). The embedded device is used as the task subject, and the smartphone is used as the collaborating device. The embedded device and the smartphone achieve parallel collaborative processing of deep learning tasks based on a preset pipeline collaboration scheme.

[0051] The preset pipeline collaboration scheme includes the first scheme to the fourth scheme. The first scheme is specifically: the embedded device preprocesses the input data, and the smartphone performs model inference and post-processing.

[0052] The second approach involves: embedded devices performing preprocessing of input data and model inference, and smartphones performing post-processing (e.g., ...). Figure 2 (as shown);

[0053] The third approach involves the embedded device directly transmitting the input data to the smartphone for preprocessing, followed by model inference and post-processing by the embedded device.

[0054] The fourth approach involves the embedded device directly transmitting the input data to the smartphone for preprocessing and model inference, followed by post-processing by the embedded device.

[0055] The above system is applied in practice to realize a method for executing deep learning tasks based on mobile phones and embedded devices, such as... Figure 3 As shown, it includes the following steps:

[0056] S1, a deep learning network model is constructed in a training set pruning manner;

[0057] S2, the communication time, the data processing time of the embedded device and the smart phone in the current task scene are determined, and the pipeline scheme is determined in combination with the set judgment condition;

[0058] S3, according to the determined pipeline scheme, the deep learning network model is carried on the embedded device or the smart phone, the input data is received by the embedded device, and the deep learning task is cooperatively executed with the smart phone.

[0059] The technical scheme considers that the pipeline technology has strict requirements on time. Therefore, in order to reduce the inference time of the model, a training set pruning method is proposed, which can consider model acceleration and compression. In addition, since the communication time is often related to the size of the transmitted data, the communication speed of different schemes is studied in the calculation of the communication time of different pipeline schemes. Furthermore, since the embedded device and the mobile phone have different processing speeds for each stage of the task, the calculation time is also needed to allocate the calculation task.

[0060] Specifically:

[0061] I. Pruning the model only relying on the training set

[0062] With the increasingly wide range of application of neural network models, the characteristics of high storage and high calculation of the neural network models are increasingly limited on embedded devices. In addition to improving the performance of hardware devices as soon as possible, it is more important to train a small and fast neural network. Unrestricted model structure design leads to a large number of redundant neurons in the neural network model. By pruning these redundant neurons, not only can the storage compression be effectively achieved, but also the running acceleration can be achieved. The technical scheme proposes a method of pruning the model structure during training using the training set: 1) train the initialized model and make it reach basic convergence; 2) feed the entire training set to the current model and calculate the activation value of each layer of neurons under the training set; 3) set the pruning ratio, calculate the threshold for pruning neurons in each layer, delete the neurons with activation values below the threshold, and correspondingly delete the weights of the upper and lower layers; 4) further train the pruned model until convergence. This way of deleting redundant neurons not only greatly reduces the parameters in the model, but also reduces the computational complexity as the number of neurons decreases, thereby achieving model acceleration.

[0063] 1.1, pruning of fully connected layers

[0064] In this pruning method, if a certain fully connected layer is pruned, the entire training set needs to be fed in, and then the activation value of each neuron in the layer is recorded. Note that the activation value of the neuron must be calculated, because it is the real input value to the next layer, and its size can reflect the importance of the neuron. The activation value of a certain neuron is accumulated on the entire training set, and then divided by the size of the training set. After setting the proportion of neurons to be deleted, only the activation value of each neuron needs to be compared with the threshold value, and the neurons less than the threshold value are deleted. In addition to deleting neurons, this method also needs to delete the weights of the upper and lower layers corresponding to them. Assume that the current layer h (i) has n neurons, the previous layer h (i-1) has c neurons, and the next layer h (i+1) has k neurons. The weight matrix between the previous layer and the current layer is W (i) (size c x k), and the weight matrix between the current layer and the next layer is W (i+1) (size s x c). The calculation formula of the neurons of the current layer and the upper and lower weight matrices is as follows:

[0065]

[0066]

[0067] wherein, represents the jth neuron in the ith fully connected layer h i , and similarly represents the jth neuron in the i+1th fully connected layer h i+1 . represents the jth column of the transpose of the matrix W (i) , and similarly represents the jth column of the transpose of the matrix W (i+1) . If we delete the jth neuron in the h (i) layer, we need to delete the jth row in the matrix W (i) and the jth column in the matrix W (i+1) .

[0068] 1.2, pruning of convolutional layers

[0069] This method refers to the normalized method, and records the average value of each channel on the entire training set. First, count the sum of the activation values of each channel on the training set. Second, calculate the sum of all pixel values in each channel and divide it by the length and width of the channel. Third, divide the result by the size of the dataset. The entire calculation process can be represented by the following formula.

[0070]

[0071]

[0072]

[0073] where C [i] is the i-th channel in a certain convolutional layer in the current model, K is the total number of samples in the training set, and W and H are the length and width of the current channel, respectively.

[0074] After the process of threshold calculation is determined, it is also necessary to determine how to delete the convolution kernel related to the deleted channel. Similar to the fully connected layer, this step needs to understand the calculation process between convolution layers. It is assumed that the current layer h (i) has c (i) channels, the previous layer h (i-1) has c (i-1) channels, and the next layer h (i+1) has c (i+1) channels. Then the parameter matrix W (i) between the previous layer and the current layer has a size of [c (i) , c (i-1) , H, W], and the parameter matrix W (i+1) between the current layer i and the next layer has a size of [c (i+1) , c (i) , H, W]. The calculation formula of the neurons in the current layer and the parameters of the upper and lower layers is as follows.

[0075]

[0076]

[0077] where [:] represents all values of the current latitude, represents the convolution operation. As can be seen from the above formula, after the pruning of the current layer h (i) , all corresponding elements in the dimension of 0 of the current layer parameter W (i) need to be deleted, and all element values of the next layer parameter with the latitude of 1 need to be deleted.

[0078] Training a neural network model suitable for a corresponding task is an iterative process, and the iterative process is as shown in Figure 4 After multiple iterations, the most important neurons can be found. Through continuous iterative training, the compression and inference acceleration of several times compared with the original model are realized, and the cost is extremely low.

[0079] II. Calculate the communication time in different pipeline schemes

[0080] The communication methods available to deep learning tasks vary in different scenarios. By listing the current mainstream communication methods and their rates, distances, etc., it can be seen that communication speed, as one of the important factors in forming the pipeline speed, must be accurately measured. Although the communication speed in different scenarios has its calibrated speed in the industry, it often cannot be achieved in actual use. Therefore, this embodiment measures the communication speed by transmitting data packets. In order to simulate the real embedded and mobile phone collaborative computing as much as possible, the data packet size is defined as 1MB. The embedded device sends 1MB of data to the mobile phone, which immediately returns the original data to the embedded device without any processing after receiving the data. And considering that this research is aimed at continuous deep learning tasks, this embodiment will continuously send and receive 100 data packets, and calculate the relative stable communication speed by calculating the acceptance and sending time delay of the 100 data packets. In addition, the stability of the communication environment needs to be calculated according to the congestion of the communication, and different emergency frequencies need to be developed. This emergency frequency can be used to guide the system to detect the communication congestion at what frequency to ensure the real-time requirements of the task.

[0081] After obtaining the average communication speed in the current scenario, it is necessary to further calculate the communication time in different embedded and mobile pipeline scheme designs. At present, this technical solution divides the pipeline scheme into four types, although four different pipeline schemes are proposed, but the communication time to be calculated is only: the sending (receiving) time of the preprocessed data t p , the sending (receiving) time of the model output result t output , the sending time of the original data t img , and the receiving time of the task result t rf .

[0082] III. Calculation of the calculation time of different parts of the task

[0083] When the embedded device and the mobile phone are initially connected, a reasonable pipeline scheme should be developed according to the scenario of the task. In order to develop a reasonable pipeline scheme, the calculation time of each part must also be accurately calculated. The calculation process of the deep learning task has been divided into three stages: preprocessing stage, model inference stage, and model result post-processing stage. Since the computing power of the embedded platform and the mobile phone is different, the calculation time of each stage in the embedded device and the mobile phone needs to be calculated respectively.

[0084] For the calculation time of the preprocessing stage and the model result post-processing stage, the average of multiple calculations can be taken to obtain a relatively accurate result. The preprocessing time in the embedded platform is denoted as t ep , and the post-processing time in the embedded platform is denoted as t efIn the same way, the pre-processing time and post-processing time t mp on the mobile phone are recalculated mf .

[0085] Although the time for the pre-processing stage and the time for the post-processing stage of the model result can be obtained at a small cost, the time for calculating the model inference on the mobile phone is often costly. Since the weights of the model are relatively large, if only the time for calculating the model inference on the mobile phone is downloaded to the mobile phone, it is actually a waste. Existing research has proposed a predictor for estimating the model inference time in an edge device based on the operator level. By inputting the model structure and the neural network framework in the mobile phone into the predictor, the estimated time for the model inference on the mobile phone can be quickly obtained. Although the work of this research may not be very accurate, considering the factors such as mobile phone storage and stability, if the time advantage is not particularly obvious, the model is not considered to be transplanted to the mobile phone for cooperation. The model inference time in the embedded (mobile phone) is denoted as t ec (t mc ).

[0086] Four, pipeline scheme is established

[0087] The embedded is the main body of the deep learning task, and the result of the final task will guide the next judgment of the embedded. Therefore, in order to perform the deep learning task in a pipeline manner, the condition in formula (1) must be met.

[0088] t e ≥ t c + t m (1)

[0089] Where t e represents the total calculation time of the embedded, t c represents the passing time, and t m represents the total calculation time of the mobile phone.

[0090] After considering the calculation time and the communication time, the present technical solution designs four different pipeline acceleration schemes according to the time relationship. The specific scheme allocation can be seen in Table 2.

[0091] Table 2 Pipeline scheme establishment

[0092] Scheme Embedded device Mobile phone Conditions Scheme one Preprocessing Model inference + postprocessing t ep ≥t p +t rf +t mc +t mf ]]> Scheme two Preprocessing + model inference Postprocessing t ep t ec ≥t output t rf t mf ]]> Scheme three Model inference + postprocessing Preprocessing t ec t ef ≥t img t p t mp ]]> Scheme four Postprocessing Preprocessing + model inference [CAT ef ≥t img +t output +t mp +t mc ]]>

[0093] The stage allocation of different pipeline schemes in the table mainly depends on the time relationship of each part. Although the preconditions for using different schemes are given, in reality, there may be situations where several schemes can be used simultaneously. However, considering factors such as mobile device storage and stability, the schemes are prioritized as follows: Scheme 2, Scheme 3, Scheme 1, Scheme 4. It is also worth noting that, theoretically, t... c +t m The closer to t e The more efficient the acceleration, the better. However, when considering this factor, the task duration and the stability of the current communication environment must be taken into account. Long task durations and unstable communication can lead to pipeline congestion, preventing full pipeline implementation. Furthermore, the communication pipeline must be periodically checked during the pipeline process. When communication congestion occurs, the local computing resources of the embedded device should be used to obtain the required results promptly and clear the pipeline. The final overall pipeline timing diagram is as follows: Figure 5 As shown.

[0094] This embodiment applies the above-described technical solution to perform collaborative processing for lane detection tasks, and its main contents include:

[0095] 1. Experimental Model – Lane Detection Task

[0096] Lane detection is crucial in the field of autonomous driving, supporting vehicle path planning and decision-making. It remains a vital component in the foreseeable future development of autonomous driving. Lane detection is a real-time computer vision task, primarily deployed on vehicle-side devices, exhibiting strong proprietary characteristics. This makes it a suitable candidate for the mobile-embedded system-driven pipeline acceleration method described in this embodiment. The lane detection model used in this embodiment is a self-developed model. Its backbone and neck components reference the CenterNet model structure, with the backbone being ResNet18 and the neck being CTResNeck. The model's head consists of three parts, predicting the starting point of each lane line, the lane line length, and the offset between each lane keypoint and the next keypoint. The overall model structure and lane detection performance are shown below. Figure 6 As shown.

[0097] To practically verify the acceleration effect of this technical solution on lane detection tasks on embedded devices, the Tusimple dataset was used to create a video, producing a 5-minute video taken by the vehicle's forward-facing camera at a frame rate of 20 frames per second.

[0098] 2. Experimental equipment

[0099] The main equipment used in this experiment is NVIDIA Jetson TX1, Samsung Galaxy S8, OnePlus 10T, etc.

[0100] Among them, NVIDIA Jetson TX1 is an embedded device produced by NVIDIA for deep learning scenarios. They are all equipped with NVIDIA graphics cards inside and have a complete development environment and kit, which is a powerful tool for deep learning task deployment and learning. Jetson TX1 is a SoM (system-on-module) solution for computer vision systems, which combines the latest NVIDIA Maxwell GPU architecture. It has an ARM Cortex-A57MPCore (Quad-Core) CPU family, which maximizes performance and power consumption, meeting the requirements of next-generation industrial vision computing. The size of the Jetson Nano module is only 70x45mm, smaller than a credit card. When deploying deep learning models for edge devices in multiple industries, including smart cities, smart factories, and agriculture and robotics, this production-ready module system (SOM) can provide strong support.

[0101] Samsung Galaxy S8 and OnePlus 10T are both popular smartphones on the market today. They have strong computing power and market share, and can represent the vast majority of smartphones on the market.

[0102] For the scenarios involved in this embodiment, the use of the above-mentioned embedded devices and mobile phones is sufficient to simulate various possible situations of mobile phone cooperation with embedded devices for deep learning task acceleration in actual scenarios.

[0103] 3. Experimental results

[0104] Firstly, the feasibility of the proposed model pruning method is verified. The backbone model used in the lane line detection task is ResNet, which is a classic feature extraction model. This type of model was originally proposed as an image classification task model. In order to verify the universality of this method for this type of model, VGG-16 and ResNet-50 were respectively tested on CIFAR-10 and CIFAR-100 to study the relationship between pruning and accuracy reduction.

[0105] VGG-16 has 13 convolutional layers and 3 fully connected layers. First, the VGG-16 model is trained on the CIFAR-10 dataset. CIFAR-10 contains 10 classes, with 50,000 training images and 10,000 test images. During the training, the model is iteratively pruned. The VGG-16 model trained using the pruning method has 4 times compression and 5 times inference acceleration compared to the original model.

[0106] The method is also applied to train the VGG-16 model on CIFAR-100. CIFAR-100 has the same number of images as CIFAR-10, but it has 100 classes. This task is more complex and difficult than CIFAR-10, but still achieves 2 times compression and 2 times acceleration. The compression and acceleration effects are not as good as the results on CIFAR-10. However, this further illustrates the point of the present solution, that the structure of the neural network model needed for different tasks is different, and the present solution needs to train not only the model weights but also the appropriate model structure.

[0107] The model accuracy and compression effect of VGG-16 on CIFAR-10 and CIFAR-100 datasets are shown in Table 3.

[0108] Table 3

[0109]

[0110] For the ResNet-50 model training process, the parameter settings used in this embodiment are basically the same as those of VGG-16. However, from the experimental results, it can be seen that the model needs to be enlarged to 224x224 to obtain better model accuracy. At the same time, the Resnet-50 model only needs 20 epochs to converge. ResNet-50 has more convolutional layers and more complex structure than VGG-16. In particular, the residual structure in ResNet is a big challenge for many model pruning methods, but using the pruning method of the present solution, important neurons are selected during training, which is not affected by the model structure. Without additional calculations, the compressed ResNet-50 model can be obtained. The pruning effect of the method on the ResNet-50 model is shown in Table 4.

[0111] Table 4

[0112]

[0113]

[0114] From the above experimental results, it can be concluded that the pruning method proposed in the present scheme can be well adapted to various classification models, and the precision decrease is controllable.

[0115] The present embodiment runs the lane line monitoring detection model on TX1 (embedded device), and simulates the scene of the actual vehicle driving on the road through the video converted by the Tusimple dataset. The present embodiment mainly tests two pipeline schemes, uses two different mobile phones, and simulates the use effect of the pipeline scheme in two different scenes. In Figure 7 and 8 show the calculation of each part of the pipeline in different scenarios, using different mobile devices and communication methods. In the experiment, the pictures in the video made of the Tusimple dataset are read frame by frame, and the speedup ratio of using the pipeline scheme is calculated.

[0116] TX1 is an embedded device designed for visual tasks, and its GPU computing resources are very powerful. Figure 7 and Figure 8 show the calculation time of each part of the lane line detection task in different devices and the communication time of different data in different communication environments. At this time, a variety of pipeline schemes can be selected, and the present embodiment tests the task execution speed of different pipeline schemes under the condition of using Wi-Fi as the communication method. From Figure 9 (wherein baseline corresponds to the case of not using the pipeline scheme, scheme1~4 correspond to the first scheme~fourth scheme, respectively) can be known that only the second pipeline scheme is suitable for this scenario. Without processing the pipeline congestion, using an inappropriate pipeline scheme will slow down the running speed. Using the second pipeline scheme will provide about 1.3 times acceleration for task execution. In the case of using a suitable pipeline, both the number of pictures that can be detected per second and the speed of task execution on a fixed time video have been significantly improved. This improvement can be achieved at a very small cost, and the deployment is simple and widely applicable.

[0117] In summary, the present technical scheme takes into account that the computing power of the smartphones that people can use is often very limited, which leads to a large amount of computing resources in society being idle for a long time. The continuous development of smart home and smart car fields is still subject to the computing resources of expensive embedded platforms. In such fields with a large number of deep learning model application scenarios, there is often a certain privacy. Therefore, the present technical scheme cooperates the mobile phone and the embedded device with private attributes, improves the task execution effect of the embedded device without affecting the normal use of the mobile phone.

[0118] The appropriate selection of the cooperation mode between the mobile phone and the embedded device can effectively utilize the computing resources of the mobile phone. The application scenario of the scheme is a deep learning task, and the workflow has the following characteristics. First, there are quite a few continuous deep learning tasks, such as video processing in computer vision, which requires continuous frames in a period of video to be calculated to obtain results. Second, a deep learning task often has a fixed computing process. These characteristics are important characteristics of forming a pipeline. Based on this, a pipeline mode is proposed to realize the parallel reasoning of the mobile phone and the embedded device. And in order to make the pipeline technology more widely used in various deep learning models, a model pruning method is also proposed to reduce the model size. For example, in the above embodiment, the lane line detection of computer vision is set. In this experiment, it is proposed to perform part of the visual task of a frame of picture on the embedded device, and perform another part of the task of the frame of picture on the mobile phone. Therefore, when the embedded device is processing n frames of pictures, the mobile phone is processing n-1 or n+1 frames of pictures, thereby forming the parallel collaborative acceleration effect of the pipeline.

[0119] In order to form a pipeline cooperation mode between the mobile phone and the embedded device, it is necessary to prune the model in the deep learning task deployed in the embedded device and slice the task or model. According to the task characteristics of deep learning, the task is divided into three stages: preprocessing stage, model reasoning stage and model result post-processing stage. Since the embedded device is the task subject and the mobile phone is the cooperative device, the prerequisite for forming a pipeline is to ensure that the computing time on the mobile phone is less than that on the embedded device. However, there is a very important and non-negligible factor, which is the communication problem. Through investigation, it is found that in the past, the collaborative computing mode from the cloud to the edge device is often affected by the communication speed. However, with the continuous development of communication technology today, we have 5G communication mode. In the 5G communication mode, the data transmission speed will reach more than 100 Mbps. Such communication speed can almost meet the communication delay requirements in most current researches. And in this scheme, the cooperation between the mobile phone and the embedded device is mainly aimed at the scene with private attributes. In this scene, there can be various flexible and relatively stable communication modes, such as USB, local area network communication and Bluetooth, etc., but the communication time must be reasonably calculated to select the best pipeline cooperation scheme.

[0120] In general, when the calculation in the embedded device is completed, another process is started to take charge of communication, and the current process continues to execute the calculation of the next picture. Thus, a method of collaborative calculation between a smartphone and an embedded device in a pipeline manner is proposed, and this method can optimize the model inference effect in the embedded device; in order to cooperate with the collaborative calculation of the smartphone and the embedded device, a pruning method for specific convolutional layers and fully connected layers only relying on the training set is proposed in the training stage; a reasonable pipeline scheme design standard is formulated, the key factors of realizing the pipeline are extracted, and a reasonable pipeline scheme is formulated according to these factors; the proposed pipeline scheme can be applied to many deep learning scenes, and better user experience can be brought to users through the interaction between the smartphone and the embedded device.

Claims

1. A method for performing a deep learning task based on a mobile phone cooperating with an embedded device, characterized in that, The method comprises the following steps: S1, constructing a deep learning network model in a pruning manner of a training set; S2, determining the communication time, data processing time of the embedded device and the smart phone in the current task scenario, and combining the set judgment condition to determine the pipeline scheme; S3, according to the determined pipeline scheme, the deep learning network model is carried on the embedded device or the smart phone, the embedded device receives the input data, and the deep learning task is executed in cooperation with the smart phone; The communication time in the current task scene in step S2 includes the difference transmission time of the preprocessed data , the transmission time of the model output result , the sending time of the original data , and the receiving time of the task result ; The data processing time of the embedded device in step S2 includes a pre-processing time of the embedded device , a post-processing time of the embedded device , a model inference time of the embedded device ; The data processing time of the smartphone in step S2 includes a pre-processing time of the smartphone , a post-processing time of the smartphone , a model inference time of the smartphone ; The judgment condition set in step S2 is specifically: If , the pipeline scheme is determined as the first scheme, i.e., the embedded device pre-processes the input data, the smartphone performs model inference and post-processing. If , the pipeline scheme is determined as the second scheme, i.e., the embedded device performs pre-processing and model inference on the input data, and the smartphone performs post-processing. If , the pipelining scheme is determined as the third scheme, i.e., the embedded device directly transmits the input data to the smartphone for preprocessing, and then performs model inference and post-processing by the embedded device. If , the pipelining scheme is determined as the fourth scheme, i.e., the embedded device directly transmits the input data to the smartphone for pre-processing and model inference, and then performs post-processing by the embedded device. 2.The method of claim 1, wherein, The step S1 specifically comprises the following steps: S11, training the initialized model and making it reach basic convergence; S12, inputting the entire training set into the current model and calculating the activation value of each layer of neurons under the training set; S13, based on the set pruning ratio, calculating the threshold value of neuron pruning for each layer structure in the model, deleting the neurons with activation values lower than the corresponding threshold value, and deleting the weights of the upper and lower layers correspondingly; S14, further training the pruned model until convergence. 3.The method of claim 2, wherein, The step S13 specifically prunes the full connection layer structure and the convolution layer structure in the model.

4. A deep learning task execution system based on a mobile phone cooperating with an embedded device, for implementing the deep learning task execution method based on a mobile phone cooperating with an embedded device according to claim 1, characterized in that, The embedded device and the smart phone are connected in communication, the embedded device is used as the task subject, the smart phone is used as the cooperative device, and the parallel cooperative processing of the deep learning task is realized between the embedded device and the smart phone based on the preset pipeline cooperation scheme. 5.The deep learning task execution system based on mobile phone cooperative embedded device according to claim 4, wherein, The communication connection mode between the embedded device and the smart phone includes but is not limited to local area network, Bluetooth and USB. 6.The deep learning task execution system based on mobile phone cooperative embedded device according to claim 4, wherein, The preset pipeline cooperation scheme includes a first scheme, a second scheme, a third scheme and a fourth scheme, the first scheme is specifically that the embedded device preprocesses the input data, the smart phone performs model inference and post-processing; The second scheme is specifically that the embedded device preprocesses and performs model inference on the input data, and the smart phone performs post-processing; The third scheme is specifically that the embedded device directly transmits the input data to the smart phone for preprocessing, and then performs model inference and post-processing by the embedded device; The fourth scheme is specifically that the embedded device directly transmits the input data to the smart phone for preprocessing and model inference, and then performs post-processing by the embedded device.

Images