A video target detection method based on multi-mobile terminal-edge terminal collaborative inference

Abstract

The application discloses a video target detection method based on multi-mobile terminal-edge terminal cooperative reasoning, which comprises the following steps: collecting a video through a mobile terminal device, taking the size of a detected target bounding box in a previous frame in the video as an original bounding box size; expanding the original bounding box size to obtain a first target boundary expansion box of a current frame by cutting the video frame; determining a new target region appearing in the current frame, expanding the size of the bounding box of the new target region, and obtaining a second target boundary expansion box of the current frame by cutting; taking the union of the first target boundary expansion box and the second target boundary expansion box as a task set of the current frame; an edge server simultaneously receives task sets sent by multiple mobile terminal devices, splices the boundary expansion boxes contained in multiple task sets to obtain a spliced image; and the edge server performs target detection on the spliced image, maps the obtained detection result to a corresponding position of a current frame of a mobile terminal video, and sends the detection result to the mobile terminal.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of edge computing and video target detection, and particularly relates to a video target detection method based on multi-mobile terminal-edge terminal collaborative reasoning. BACKGROUND

[0002] The main task of video target detection is to identify and locate targets in each video frame, and label the position of the target object using a bounding box. Video target detection technology is widely used in the fields of unmanned driving, intelligent security, etc., and has always been a research hotspot in artificial intelligence and related disciplines. With the development of deep learning technology, various deep neural networks, such as Faster-R-CNN, SSD, YOLO, etc. models, are being widely used in target detection.

[0003] The existing video target detection technology mainly includes frame-by-frame static image target detection and a method based on "detection + tracking". For the frame-by-frame detection method, a static target detector is executed in each frame of the video, and the detection results of all video frames are obtained. The method based on "detection + tracking" executes the target detector in each frame while using the context information between frames to track and correct the detection results, which can further improve the detection accuracy of the target.

[0004] In addition to the accuracy of the detection results, the time cost should also be considered in the application of video target detection. With the landing application of artificial intelligence, various mobile devices are used to deploy target detection programs. However, due to the highly limited computing resources in mobile devices and the complexity of network model calculation, it is difficult to achieve a balance between speed and accuracy on the mobile terminal. Edge computing is a feasible solution. If the computing task is offloaded to the edge terminal close to the terminal device, the powerful computing capability of the edge server can be utilized to alleviate the high delay problem caused by video target detection, and it is easier to realize high-performance model calculation and real-time service.

[0005] Although the current video target detection and edge computing technology have undergone multiple iterations and updates, and have achieved good results, there are still some key problems to be discussed. First, in each frame of video, only a small part of the foreground area needs to be located and identified, and most of the background area does not need to be calculated, so we can improve and optimize the existing video target detection algorithm for spatial redundancy in the video. Secondly, the existing methods all assume that data is generated and model reasoning is performed on a single mobile terminal device, while in reality, the number of terminal devices is quite large, and multiple users may request similar computing services at the same time. In this case, it is particularly important to ensure reasonable task scheduling and computing resource allocation. SUMMARY

[0006] The application aims to provide a video target detection method based on multi-mobile terminal-edge terminal collaborative reasoning, which uses a content-based cropping and splicing method and can improve the execution efficiency of video target detection application in a multi-mobile terminal environment.

[0007] The technical scheme provided by the application is:

[0008] A video target detection method based on multi-mobile terminal-edge terminal collaborative reasoning comprises:

[0009] A mobile terminal device collects a video, and the size of a target bounding box detected in a previous frame in the video is used as the size of an original bounding box; the size of the original bounding box is expanded, and a foreground region of a current frame is cropped according to the size of the expanded original bounding box to obtain a first target boundary expansion box of the current frame;

[0010] A new target region appearing in the current frame is determined, and the size of the bounding box of the new target region is expanded to obtain a second target boundary expansion box of the current frame;

[0011] The union of the first target boundary expansion box and the second target boundary expansion box is used as a task set of the current frame;

[0012] After the edge server receives the task sets sent by the plurality of mobile terminal devices, the first target boundary expansion box and the second target boundary expansion box contained in the plurality of task sets are spliced to obtain a spliced image;

[0013] The edge server performs target detection on the spliced image, maps the obtained detection result back to the corresponding position of the current frame of the mobile terminal video, and sends the detection result to the mobile terminal.

[0014] Preferably, the video target detection method based on multi-mobile terminal-edge terminal collaborative reasoning further comprises:

[0015] After the edge server receives the task sets sent by the plurality of mobile terminal devices, the task sets are sequentially put into a current round of calculation queue in the order of increasing remaining idle time until the maximum capacity of single image reasoning of the edge server is reached;

[0016] The remaining idle time represents the remaining time from the current time to the expected task completion deadline of the mobile terminal user.

[0017] Preferably, if the offset of the detected target in a plurality of historical frames is zero, the detected target is regarded as a static target, and the static target is unloaded to the edge server every several frames.

[0018] Preferably, when the original bounding box size is expanded, the expansion rates of the original bounding box in the width and height directions are respectively:

[0019]

[0020] wherein w and h are the width and height of the original bounding box, K is the number of reference history frames, Aw and Ah represent the width and height of the bounding box of the target in the i-th frame compared with the (i-1)-th frame, i , Ah i , and and represent the maximum of the width and height of the target in the previous K frames, C w and C h are the basic expansion rates in the width and height directions, respectively.

[0021] Preferably, C w and C h are in the range of 0.1-0.3.

[0022] Preferably, the method for determining the new target region in the current frame is as follows:

[0023] Subtract the pixel value of the current frame from the pixel value of the previous frame, and convert the obtained difference value map into a gray scale map;

[0024] Perform density-based clustering on the pixel values in the gray scale map to obtain a plurality of pixel point classes;

[0025] Construct the smallest circumscribed rectangular frame based on each of the pixel point classes;

[0026] Match the plurality of circumscribed rectangular frames with the first target boundary expansion frame respectively; if the intersection-over-union of the circumscribed rectangular frame and the first target boundary expansion frame is less than a threshold value, retain the circumscribed rectangular frame, and take the circumscribed rectangular frame as the new target region.

[0027] Preferably, when expanding the size of the bounding box of the new target region, the expansion rate is set to be in the range of 0.1-0.5.

[0028] Preferably, the method for splicing the first target boundary expansion frame and the second target boundary expansion frame in the plurality of task sets is as follows:

[0029] Create a blank image, arrange the expansion frames in descending order of the height of the image, and place the expansion frames in order from the top left corner of the blank image;

[0030] wherein the placement rule is that the top left corner of the subsequent expansion frame closely fits the top right corner of the previous expansion frame, and the expansion frames are arranged from left to right; if there is not enough free area in the current row, turn to the next row and continue to place the remaining expansion frames from the left bottom coordinate of the first image in the previous row until all the expansion frames are placed.

[0031] Preferably, before target detection is performed, a training set of target detection is reconstructed, and the target detection model in the edge server is trained through the reconstructed training set of target detection.

[0032] The method for reconstructing the training set of target detection is:

[0033] The target bounding box information in the image annotation file of the original training set is used to crop all target bounding boxes in the original training set, a plurality of target bounding boxes are randomly extracted each time, and are spliced into a new blank image, the position and category information of the target in the spliced image are recorded, a plurality of new training pictures and annotation files are formed, and are used as a supplementary training set.

[0034] The supplementary training set is added to the original training set to obtain a reconstructed training set.

[0035] Preferably, the data quantity of the supplementary training set is 20% of the data quantity of the original training set.

[0036] The beneficial effects of the present application are:

[0037] The video target detection method based on multi-mobile terminal-edge terminal collaborative reasoning provided by the present application can obviously reduce the spatial redundancy in the video frame to be reasoned, greatly reduce the processing time delay of the video target detection, especially the target detection task in a low-density foreground environment, and increase the throughput of model reasoning, and can play a significant role in real-time scenes and offline scenes. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The figure is a framework diagram of the video target detection method based on multi-mobile terminal-edge terminal collaborative reasoning.

[0039] Figure 2 The figure is a schematic diagram of the expansion of the original target bounding box.

[0040] Figure 3 The figure is a schematic diagram of the data transmission process of the multi-mobile terminal and the edge server. DETAILED DESCRIPTION

[0041] The present application will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement the present application according to the description and drawings.

[0042] As Figure 1As shown, the application provides a video target detection method based on multi-mobile terminal-edge terminal collaborative reasoning. For a specific video target detection scene, after each frame of the video is obtained on the mobile terminal, the possible foreground region of the current frame of the video is first cropped, which includes the region of the original target previously stored in the cache and the region of the current new target; then, a multi-device task scheduling module is designed on the edge terminal, and when the computing tasks of each mobile device arrive, the task amount of the edge server in a single calculation is determined according to the overall scale and urgency of the task. The edge server will splice the foreground region of the selected mobile terminal together to assemble a spliced image and then execute the target detection model. After the computing task is completed, the reasoning result obtained in the spliced image is mapped back to the corresponding result of the original mobile terminal, and finally the final result is output after the correction and update on the mobile terminal.

[0043] The application assumes that the camera collecting the video is in a fixed state, and does not consider the detection situation of the video in the camera shaking state. The main technical process realized by the application is as follows:

[0044] I. Target region cropping based on historical information. In the current video frame, the possible foreground region needs to be located, at which time the detection result of the previous frame needs to be used. We set a local cache table in the mobile device to record the latest K frame reasoning result, and the reasoning information of each frame includes the category, confidence, and position coordinates of the target. Since the position and state of the target to be detected between adjacent frames will change slightly, the original bounding box of the target in the last frame in the cache table cannot be directly used in the current frame, but should be expanded to a certain length based on the original box, and then the foreground region is cropped to obtain the boundary expansion box of the target. For a target in a video frame, the expansion rates applied to the original target boundary box in the width and height directions of the t-th frame are respectively:

[0045]

[0046] That is, the width and height of the expanded target boundary box are:

[0047]

[0048] Wherein, w and h are the width and height of the previous boundary box of the target, K is the number of reference historical frames, which is set by the user, Δw i , Δh i represents the offset of the boundary box width and height of the i-th frame of the target compared with the i-1-th frame, and represent the maximum value of the width and height direction offset of the target in the previous K frames, C w , C hare the base expansion rates in width and height directions respectively, w' and h' are the width and height of the expanded target bounding box respectively. The base expansion rate is set to reserve additional foreground area for the bounding box. Through experimental analysis, the best setting value is between 0.1 and 0.3. The expanded box obtained from the original bounding box eliminates the influence of the position offset of the current frame object relative to the previous frame. In addition, since there is not enough number of reference historical frames from the initial first frame to the Kth frame, it is necessary to unload these complete video frames, execute the target detection model at the edge server, and return the bounding box calculated thereby for use in subsequent video frames.

[0049] II. Detection of new targets in video frames. The detection results based on historical frames can only extract the foreground regions that have appeared in the current video frame, but cannot extract new targets that may appear in the video frame. Therefore, it is necessary to deploy a new target region detector on the mobile terminal. When the camera is in a fixed state, the frame difference method can be used to detect the approximate range of the new target. First, the pixel values of the current frame and the previous frame are subtracted, and the obtained difference value graph is converted into a gray scale graph. Then, the pixel values in the gray scale graph are clustered based on density, which can divide these pixel points into different classes. Then, the smallest circumscribed rectangle is constructed based on each pixel class. These circumscribed rectangles can be used as candidate position regions of new targets. Finally, these circumscribed rectangles are matched with the object boundary expansion box obtained in the first step in turn. If the intersection over union of the two is less than a given threshold (such as 0.5), it indicates that a new target may appear in the rectangle, which needs to be retained for further processing. If the intersection over union is greater than the threshold, it indicates that the region is highly overlapped with the original target region, and it can be excluded. We expand the range of the finally retained circumscribed rectangle, and obtain the boundary expansion box of the new target. Since the boundary expansion rate of the new target has no historical frame data as a reference, the specific value can be specified by the user.

[0050] III. Determining the target set to be unloaded. The union of the new target and the original target boundary expansion box is selected as the inference task set to be unloaded. Considering that the offset of part of the targets in the recent historical frames is always zero, i.e. static targets that may appear in the video frame, we will not repeatedly unload these targets, but will re-unload these targets to the server at intervals to obtain the latest inference information. This can reduce the additional computational load in the detection work.

[0051] IV. Task scheduling for multiple mobile terminals. When multiple mobile terminal users send computing tasks to the edge server at the same time, the server needs to perform reasonable task scheduling and computing resource allocation. In order to meet the computing needs of each user, the minimum idle time first algorithm is used here to determine the mobile device that is given priority to execute. The idle time refers to the time the user wants to complete the task. The initial idle time of the task is specified by the mobile terminal and is generally set to the time the user expects to complete each frame. During the task offloading process, the idle time will dynamically change due to the transmission delay and propagation delay. When the task reaches the edge server, the system will preferentially select the mobile terminal with the least remaining idle time. The current remaining idle time can be expressed as T = W + I, where W and I represent the waiting time and model inference time of the task, respectively. Specifically, we first put each task into the waiting queue in order of increasing remaining idle time. After the edge server completes the model inference of the previous round, the user task with the smallest current remaining idle time is selected from the waiting queue to join the current round of computation until the maximum capacity of the edge server for single image inference is reached. The reason why multiple mobile terminals can execute computing tasks at the same time is that each mobile terminal sends a number of boundary expansion boxes for foreground regions, which can be spliced to form a spliced image for inference. Through this method, the number of mobile devices that can be processed by the edge server at a time can be significantly increased, and the average response time of the mobile terminal can be reduced.

[0052] V. Splicing of expansion boxes. The multiple boundary expansion boxes generated by multiple mobile terminal devices are spliced on the edge server to form a new image. The specific method is as follows: first, create a blank image, and the image size can be dynamically adjusted according to the situation, then place these expansion boxes in the corresponding positions of the blank image in a certain order. The placement method and order of the expansion boxes need to be reasonably optimized. This problem belongs to the two-dimensional packing problem, i.e., loading multiple small rectangles into an empty large rectangle and making the number of small rectangles as large as possible. Considering the time complexity and execution performance of the algorithm, we choose to use the next-fit algorithm to solve this problem. First, arrange the images of these expansion boxes in descending order of height to reduce the size of the unused free area in the large image; then, we start placing these images from the top left corner of the empty image. The principle of placement is that the top left corner of the next image is tightly fitted with the right top corner of the previous image, and the images are arranged from left to right. If there is not enough free area in the current row, it will move to the next row and continue to store the remaining images from the left bottom coordinate of the first image in the previous row until all the images are placed.

[0053] VI. Execute the remote target detection model. Since the training set and the actual test set do not completely come from the same data distribution due to the use of the cropping and splicing method, directly using the original data set to train the detection model can easily cause the detection accuracy to decrease. Therefore, we reconstructed the training set of target detection. First, using the target bounding box information in the image annotation file in the original training set, all target bounding boxes in the original training set were cropped, and then a number of target bounding boxes were randomly extracted from them each time and spliced into a new blank image. Finally, the position and class information of the target in the spliced image were recorded to form new training pictures and annotation files. We used this method to increase 20% new data based on the original training set to form a new training set. Practice has proved that this method can improve the accuracy of the model in the test stage. In the model inference stage, the server executes the trained target detection model, and then maps the obtained results back to the corresponding position of the mobile terminal video frame and sends it to the mobile terminal. In a single target original area, since other targets with similar distances to the target in the video frame will also be included in the expanded box, multiple inference results may appear, and we also return the results of these targets to the mobile terminal.

[0054] VII. Mobile terminal receives remote computing results and updates local cache table. The mobile terminal receives the computing results returned by the edge server, removes the targets that have been repeatedly inferred by the server, and adds the final results into the local cache table. Thus, the target detection of a single frame is completed. Similarly, the inference of the next frame can continue.

[0055] Mobile terminal level implementation:

[0056] The mobile terminal uses mobile devices such as mobile phones and Raspberry Pi for actual deployment and operation. The mobile terminal reads real-time video streams or offline video files in sequence, where the initial frame needs to unload the complete original image and execute on the edge server. When reading the t-th video frame F t , it is first scaled to a fixed size for target detection network input, such as adjusting the resolution to 640x640 when using YOLO-V5, and then the target bounding box of F t-1 is extracted based on the inference result of F t , and the region is expanded, as shown in Figure 2 , the foreground bounding box of the video frame is expanded by a certain length in the width and height directions according to the situation of the historical frame.

[0057] Then, the new target boundary box set is obtained using the inter-frame difference method, and the boundary expansion of the new target boundary box is also needed. Since the boundary expansion rate of the new target has no historical frame data as a reference, the specific value can be specified by the user, and the general range is between 0.1 and 0.5. After merging the two sets, the data is encoded and sent to the server side. The offset of some targets in the historical frame is close to zero, and we do not unload these targets, but reload them by unloading every few frames. For this purpose, a data cache table recording the detailed information of each target in the video must be established in advance to facilitate the analysis and processing of some special targets.

[0058] Edge-side layer implementation:

[0059] When multiple mobile ends arrive at the same time, the server selects the device that will reach the deadline from the device queue according to the user's required deadline. The images in these selected mobile ends are spliced using the Next-fit algorithm to form a spliced image for inference. First, the size of the blank image to be stored needs to be specified, for example, the standard input size of the current detection network can be used, such as 640x640, and the maximum workload of the mobile end that can be processed at one time is estimated from the device queue, so that the appropriate number of mobile ends can be selected, where the workload can be approximately estimated as the sum of the areas of the boundary expansion boxes. Considering that there will still be a small gap in the spliced image, the workload of the currently selected mobile end should be slightly smaller than the area of the blank image. After the expansion boxes of these selected mobile ends are spliced, the spliced image is input into the target detection network for inference. After the inference is completed, the boundary box coordinates in the spliced image are mapped back to the corresponding positions in the original frame, and then returned to the mobile end. As shown in FIG. 3, in one task scheduling, the server selects device p and device q, combines their computing tasks, and then inputs them into the target detector for inference. The detector gives the target annotation box (annotated with a dashed line in the middle) and other information, and finally the data is mapped back to the mobile end. Figure 3

[0060] ​The application utilizes the content correlation of the video frame context, uses the detection result of the previous video frame buffer to cut out the key region of the original target that may be located in the current frame as a candidate calculation region. Then, the traditional computer vision method (interframe difference method) is applied on the mobile terminal to perceive the possible appearing region of the new target in the current video frame. The boundary box of the new target is sent to the edge server together with the original target boundary box, and then the splicing method is used to input the network to complete the model reasoning, which greatly reduces the redundant calculation amount of the background region in the video and reduces the transmission amount of the mobile terminal data. On the other hand, since each mobile terminal device sends the image region fragments to be processed to the edge server, the application combines the calculation tasks of multiple mobile terminals to further optimize the calculation efficiency of the system and the average response time of each mobile terminal.

[0061] The video target detection method supporting the collaborative reasoning of multiple mobile terminal devices provided by the application uses the content-based cutting and splicing technology to improve the execution efficiency of the video target detection application in the multiple mobile terminal environment and ensures that the precision loss is within a reasonable range. The application can greatly improve the efficiency of the video target detection, especially the target detection task in the low-density foreground environment, and can play its role in real-time scenes and offline scenes.

[0062] Although the embodiments of the application have been disclosed as above, they are not limited to the application listed in the specification and the embodiments, and can be fully applied to various fields suitable for the application. For those skilled in the art, other modifications can be easily realized, and therefore the application is not limited to the specific details and the figures shown and described herein, without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A method for video object detection based on multi-mobile edge collaborative inference, characterized in that, The method comprises the following steps: Collecting a video through a mobile terminal device, taking the size of a target bounding box detected in a previous frame in the video as an original bounding box size; Expanding the original bounding box size, and cropping a foreground region of a current frame according to the expanded original bounding box size to obtain a first target bounding expansion box of the current frame; Determining a new target region appearing in the current frame, and expanding the size of the bounding box of the new target region to obtain a second target bounding expansion box of the current frame; Taking the union of the first target bounding expansion box and the second target bounding expansion box as a task set of the current frame; After receiving the task sets sent by multiple mobile terminal devices, the edge server splices the first target bounding expansion box and the second target bounding expansion box in the multiple task sets to obtain a spliced image; The edge server performs target detection on the spliced image, maps the obtained detection result to a corresponding position of a current frame of the mobile terminal video, and sends the detection result to the mobile terminal; When the original bounding box size is expanded, the expansion rates of the original bounding box in the width and height directions are respectively: where w and h are the width and height of the original bounding box, K is the number of reference history frames, Aw i , and Ah i represent the offset of the target in the width and height direction of the i-th frame compared to the (i-1)-th frame of the history frames, and represent the maximum value of the width and height direction offset of the target in the previous K frames, C w and C h are the basic expansion rates in the width and height direction, respectively. The method for determining the new target region appearing in the current frame comprises the following steps: Subtracting the pixel value of the current frame from the pixel value of a previous frame, and converting the obtained difference value image into a grayscale image; Performing density-based clustering on the pixel values in the grayscale image to obtain multiple pixel point classes; Constructing a minimum circumscribed rectangle for each pixel point class; Matching the multiple circumscribed rectangles with the first target bounding expansion box respectively; if the intersection-over-union ratio of the circumscribed rectangle and the first target bounding expansion box is less than a threshold value, the circumscribed rectangle is retained, and the circumscribed rectangle is taken as the new target region.

2. The method of claim 1, wherein the method further comprises: The method further comprises the following steps: After receiving the task sets sent by multiple mobile terminal devices, the edge server sequentially puts the task sets into a current round of calculation queue in the order of increasing remaining idle time until the maximum capacity of the edge server for executing single image inference is reached. The remaining idle time represents the remaining time from the current time to the deadline expected by the mobile terminal user. 3.The method of claim 2, wherein, If the offset of the detected target in multiple continuous historical frames is zero, the detected target is regarded as a static target, and the static target is unloaded to the edge server every interval of a plurality of frames.

4. The method of claim 2 or 3, wherein the method further comprises: C w and C h The value range of each of A, B, C and D is 0.1-0.

3.

5. The method of claim 4, wherein the method further comprises: When the size of the bounding box of the new target region is expanded, the expansion rate is set to 0.1-0.

5.

6. The method of claim 5, wherein the method further comprises: The method for splicing the first target bounding expansion box and the second target bounding expansion box in the multiple task sets comprises the following steps: Creating a blank image, arranging the expansion boxes in descending order of the height of the image, and placing the expansion boxes from the top left corner of the blank image in sequence; The placement rule is that the top left corner of a subsequent expansion box is tightly fitted with the top right corner of a previous expansion box, and the expansion boxes are arranged from left to right; if there is not enough free area in the current row, the next row is turned to, and the remaining expansion boxes are placed from the bottom left coordinate of the first image of the previous row, until all the expansion boxes are placed.

7. The method of claim 6, wherein the method further comprises: Before target detection, the method further comprises the following steps of: Reconstructing a training set of target detection, and training a target detection model in the edge server through the reconstructed training set. The method for reconstructing the training set of the target detection is as follows: The target bounding box information in the image annotation file of the original training set is used to crop all the target bounding boxes in the original training set, a plurality of target bounding boxes are randomly extracted each time, and the plurality of target bounding boxes are spliced into a new blank image, the positions and category information of the targets in the spliced image are recorded, a plurality of new training pictures and annotation files are formed, and the plurality of new training pictures and annotation files are used as a supplementary training set; The original training set is added to the supplementary training set to obtain a reconstructed training set. 8.The method of claim 7, wherein, The data quantity of the supplementary training set is 20% of the data quantity of the original training set.

Images