Edge computing unloading method for visual application

Abstract

The invention discloses a visual application-oriented edge calculation unloading method, which comprises the following steps of: extracting a region of interest, correspondingly dividing a video frame into fragments, and regularly unloading a low-resolution video image to an edge server for deploying a target counting task for reasoning to obtain a density map; dynamically adjusting the region-of-interest extraction range of the video frame according to the obtained density map, identifying the dense regions in the video image, further dividing the dense regions possibly existing in the video image, splicing the dense regions into fragments, and realizing balanced unloading on common fragments and dense fragments, so as to obtain an unloaded region; and executing the target detection task, returning all detection results after reasoning is completed, and integrating the detection results from different edge computing servers. According to the method, the visual application problem is solved by utilizing a video analysis framework based on edge computing unloading, and the computing precision is considered while low time delay is guaranteed.

Description

Technical Field

[0001] The present application belongs to the field of mobile edge computing technology, and in particular relates to an edge computing offloading method for visual applications. Background Art

[0002] Video analytics is the process of automatically extracting useful information from video data using computer vision and deep learning techniques. With the rapid development of deep learning technology, breakthroughs have been made in the field of computer vision, leading to the emergence of a range of advanced visual applications, such as object detection, object tracking, object counting, instance segmentation, and human pose recognition. These technologies have significantly improved the ability to extract key information from video streams, providing strong support for intelligent decision-making. Currently, video analytics technology has been widely applied in various fields, including public security monitoring, traffic management, retail analytics, industrial automation, sports event analysis, and healthcare, providing precise and efficient intelligent services for various industries.

[0003] However, despite significant progress in deep learning-driven vision tasks, their high computational resource requirements pose a significant challenge to resource-constrained Internet of Things (IoT) devices. Traditional computing paradigms often rely on cloud computing, which offloads computational tasks to remote data centers. While cloud computing offers powerful computing capabilities, it often suffers from high communication latency due to the long physical distance between end devices and cloud servers and potential network congestion.

[0004] To address the high latency issues inherent in cloud computing, edge computing, as an emerging computing model, is increasingly being recognized as an effective complement or even a viable alternative to cloud computing. Edge computing utilizes a distributed architecture, offloading computing tasks from the remote cloud to locations closer to the data source, such as base stations, routers, or local servers. This significantly reduces data transmission latency and improves computational response speed. In recent years, the integration of edge computing and video analytics, known as edge video analytics, has garnered widespread research attention in both academia and industry. By offloading video streams captured by resource-constrained IoT devices, such as cameras, to edge servers for visual reasoning, this approach not only optimizes computing resource allocation but also significantly reduces latency while ensuring computational accuracy. Consequently, edge video analytics is playing an increasingly important role in numerous application scenarios, including smart security, smart transportation, and smart manufacturing, and is expected to become a core computing model for large-scale video surveillance and analysis systems in the future.

[0005] However, current video analysis technologies, particularly in edge computing environments, face numerous challenges. To adapt to the limited computing power of edge devices, lightweight models are widely used. While these models can effectively reduce computing resource consumption, they suffer from low detection accuracy in complex scenarios and are prone to missing small targets or misdetecting background noise, resulting in inaccurate detection results. In particular, in densely populated surveillance scenarios, small targets (such as pedestrians' heads) may be obscured by background noise or larger objects, reducing detection accuracy. Furthermore, in high-density target scenes, traditional detection models may misdetect multiple targets in a dense area as one large target or miss some targets, leading to inaccurate detection results. Furthermore, edge computing environments typically include a variety of edge servers with varying computing power and network bandwidth. However, many existing offloading strategies simply distribute video analysis tasks evenly across all edge servers or impose fixed allocations based on server type, without dynamically adjusting task allocation based on each server's actual computing power. In these situations, high-performance servers may remain idle, while low-performance servers may become overloaded, increasing processing latency and failing to meet the requirements of low latency and efficient utilization of computing resources. Summary of the Invention

[0006] The purpose of this application is to provide an edge computing offloading method for visual applications, so as to efficiently offload video analysis tasks to edge computing nodes, while ensuring low latency and taking into account computing accuracy.

[0007] In order to achieve the above objectives, the technical solutions of this application are as follows: An edge computing offloading method for visual applications, applied to terminal devices, comprising: Collect the original video image, downsample it to obtain a low-resolution video image, send the low-resolution video image to the edge computing server deployed with the multimodal open-world counting model for target counting processing, and obtain the returned density map; A background model is established based on the historical color distribution of pixels, and regions of interest are extracted from the original video image to obtain a first set of regions of interest. Then, the regions of interest in the irrelevant background are filtered out from the first set of regions of interest by combining the density map to obtain a second set of regions of interest. Divide the original video image into blocks, match each region of interest in the second region of interest set to each block, and generate common slices based on the blocks containing the region of interest; Based on the density map, we identify the dense areas in the common slices and combine all the dense areas into a dense slice. Normal sharding and dense sharding are offloaded to different edge computing servers for target detection.

[0008] Furthermore, the background model is established based on the historical color distribution of the pixels, and the regions of interest are extracted from the original video image to obtain a first set of regions of interest, including: Match each pixel in the original video image with each Gaussian distribution in the historical color distribution, and then update the parameters of each Gaussian distribution; A Gaussian distribution with a total weight greater than the background ratio threshold is selected as the background model. If a pixel in the original video image belongs to the background model, it is considered background; otherwise, it is considered foreground, and a foreground mask is generated. Based on the foreground mask, a first set of regions of interest is obtained.

[0009] Furthermore, after obtaining the first set of regions of interest based on the foreground mask, the method further includes: Small noise is filtered out by setting a minimum area threshold to extract the final first set of regions of interest.

[0010] Furthermore, the combined density map filters out regions of interest in irrelevant background from the first region of interest set to obtain a second region of interest set, including: The original video image is vertically divided into multiple regions, and the upper limit value of each region is determined based on the density map; The region to which each ROI in the first ROI set belongs is determined. For any ROI, if its bottom boundary is higher than the previous value of the region to which it belongs, the ROI is removed from the first ROI set to obtain a second ROI set.

[0011] Furthermore, generating common slices based on the blocks containing the region of interest includes: Calculate the minimum bounding rectangle of each block containing the region of interest; Each minimum bounding rectangle is expanded, and the corresponding area is cropped from the original video image using the expanded bounding rectangle as a common slice.

[0012] Furthermore, the process of offloading the common shards and the dense shards to different edge computing servers for target detection includes: Offload common shards to edge computing servers deployed with lightweight YOLOv8s; Offload dense sharding to edge computing servers where optimized YOLOv8x is deployed.

[0013] Furthermore, the optimized YOLOv8x includes a backbone network, a neck network, and a detection head, wherein: Add the CBAM module before the SPPF module in the backbone network; Insert the Involution module after the SPPF module in the backbone network and between it and the neck network; A small target detection head is added to the detection head, and a feature fusion branch that fuses higher-resolution feature maps is added to the neck network and output to the small target detection head for detection.

[0014] This application proposes an edge computing offloading method for visual applications, which has the following beneficial effects: 1. Utilize background modeling technology to extract regions of interest (ROIs) from video frames and adaptively segment them to accommodate dynamic scene changes. Combined with the density map generated by the target counting model CountGD, the algorithm dynamically filters out irrelevant areas, reducing interference from background changes on ROI extraction and improving the stability and accuracy of ROI extraction.

[0015] 2. A balanced offloading strategy dynamically allocates shards based on the computing power of edge computing servers, offloading them in parallel to a heterogeneous edge server cluster for target detection inference. This fully utilizes computing resources, optimizes end-to-end latency, and improves detection efficiency. Furthermore, by optimizing the offloading strategy based on model heterogeneity, shards in common areas are allocated to lightweight models, while shards in dense areas are allocated to high-performance models. This effectively balances detection accuracy and system latency in dense scenarios. This demonstrates excellent adaptability in high-resolution video analysis tasks.

[0016] 3. Combined with the density map generated by CountGD, dense areas are accurately identified and combined and optimized according to the minimization principle to form more reasonable fragments for unloading, which improves the detection effect of dense areas and significantly improves the detection performance in dense scenarios.

[0017] 4. By uniformly scaling all shards before inference, the computational graph reconstruction overhead is reduced, further optimizing inference stability and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of the edge computing offloading method for visual applications in this application.

[0019] Figure 2 This is a network structure diagram of YOLOv8x-crowd, an embodiment of the present application.

[0020] Figure 3 A schematic diagram showing the comparison of accuracy of different scenarios and comparison methods.

[0021] Figure 4 Schematic diagram of latency comparison in different scenarios and comparison methods. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0023] An embodiment of the present application provides an edge computing offloading method for mobile vision applications, which is applied to terminal devices, such as Figure 1 Shown, including: Step S1: Collect the original video image, downsample the original video image to obtain a low-resolution video image, send the low-resolution video image to the edge computing server deployed with the multimodal open world counting model for target counting processing, and obtain the returned density map.

[0024] The devices that make up the edge computing system usually include terminal devices and multiple edge computing servers. In the edge computing offloading task, the terminal device will assign the task to each edge server, which will perform distributed processing and then return the processing results to the terminal device. Finally, the terminal device will integrate the processing results from different edge servers to generate the final video analysis results.

[0025] Wireless communication technology between terminal devices and edge computing servers enables high-speed, low-latency data exchange. For example, WiFi or 5G mobile communication technology can be used, which will not be discussed here.

[0026] In the subsequent steps, it is necessary to extract the region of interest based on the original video image captured by the terminal device. In order to prevent noise from being considered as the region of interest, this application introduces a multimodal open world counting model (CountGD) and deploys the CountGD model in some edge computing servers to obtain the density map. ,Then the density map is used to distinguish irrelevant background from relevant background to obtain a more accurate region of interest.

[0027] This embodiment downsamples the original video image to a low-resolution image and sends it to an edge computing server deployed with the CountGD model to obtain a density map. Transmitting low-resolution video images between the terminal device and the edge server can reduce transmission latency. For example, if the original video image is ultra-high resolution (26,753 × 15,052), the resulting low-resolution video image after downsampling has a resolution of 3,840 × 2,160 (i.e., 4K resolution).

[0028] The low-resolution video image is then sent to the edge computing server where the multimodal open world counting model is deployed for target counting processing, and the returned density map is obtained. The multimodal open world counting model (CountGD) is a relatively mature technology in this field. The density map obtained by the CountGD model It directly reflects the distribution of targets in the image. Bright areas indicate high target density, while dark areas indicate sparse or no target areas.

[0029] Step S2: Establish a background model based on the historical color distribution of pixel points, extract regions of interest from the original video image, and obtain a first set of regions of interest. Then, combine the density map to filter out regions of interest in irrelevant background from the first set of regions of interest to obtain a second set of regions of interest.

[0030] There are many methods for extracting regions of interest from original video images in the art, such as those based on historical target detection results, target tracking technology, and heat map generation. This embodiment establishes a background model based on the historical color distribution of pixels, and then extracts regions of interest from the original video image, including: Step 2.1.1: Match each pixel in the original video image with each Gaussian distribution in the historical color distribution, and then update the parameters of each Gaussian distribution.

[0031] In this embodiment, the terminal device collects a fixed video image of a certain scene, and its environmental background remains unchanged. Therefore, the historical color distribution can be obtained through the historical video image. The historical color distribution of each pixel is given by It is composed of Gaussian distributions and will not be described here.

[0032] The historical color distribution of each pixel is given by Gaussian distribution, then the pixel The probability density function can be expressed as:

[0033] in is the number of Gaussian mixture distributions, usually taken as ; For the The weight of a Gaussian distribution represents the contribution of the distribution to the background and satisfies the normalization condition ; For the A Gaussian distribution, whose probability density function is defined as follows:

[0034] in is the mean vector (RGB three channels); is the covariance matrix, assuming that each channel is independent, it can be simplified to a diagonal matrix .

[0035] Processing a new frame of video image When each pixel value Need to be consistent with the historical color distribution Gaussian distributions are matched to determine their ownership. The matching steps are as follows: First calculate Mahalanobis distance to each Gaussian distribution:

[0036] If satisfied , then the pixel is considered to belong to the Gaussian distribution, and the mean, variance and weight of the distribution are updated:

[0037] in is the adaptive learning rate, defined as follows:

[0038] in is the time window size, which is used to control the update speed of the model.

[0039] It should be noted that if the pixel value Does not match any existing Gaussian distribution. In order to dynamically adapt to new data points, the Gaussian distribution with the smallest current weight is selected. To change the parameters of the Gaussian distribution.

[0040] After the parameters of each Gaussian distribution are updated, the weights of the Gaussian distributions need to be normalized so that the sum of the weights of the Gaussian distributions is equal to 1. This will not be described in detail here.

[0041] Step 2.1.2: Select a Gaussian distribution whose total weight is greater than the background ratio threshold as the background model. If the pixel in the original video image belongs to the background model, it is the background; otherwise, it is the foreground, and a foreground mask is generated.

[0042] The region of interest extraction module uses a dynamic weight accumulation method to select the background model and set the background ratio threshold. (usually 0.7 to 0.9), select the sum of weights greater than Before A Gaussian distribution is used as the background model:

[0043] If the pixel Belong to this If the value matches one of the background Gaussian distributions, it is considered to be the background, otherwise it is the foreground.

[0044] Through foreground segmentation, a binary foreground mask is obtained ,in represents the foreground (region of interest); Represents background or shadow.

[0045] Step 2.1.3: Based on the foreground mask, obtain the first set of regions of interest.

[0046] In order to ensure that a stable and complete region of interest is extracted, the foreground mask Perform morphological opening and closing operations to remove noise and fill holes, and use contour detection to obtain the bounding box of each contour , each bounding box is a region of interest, thereby obtaining a first region of interest set.

[0047] In a specific embodiment, by setting a minimum area threshold Filter small noise to extract the final first set of regions of interest:

[0048] Finally, the first set of regions of interest is obtained: .

[0049] Although the minimum area threshold has been set To filter small noise, but in high-resolution video frames, the target pixels are smaller in the area closer to the top, so If the setting is too large, the area of ​​interest in this area will be missed; If the value is set too small, noise will also be considered as the region of interest. Therefore, this embodiment also combines the density map to filter out the regions of interest in irrelevant background from the first region of interest set to obtain the second region of interest set.

[0050] The target counting model CountGD is introduced to solve the above problems. The edge computing server deploying the CountGD model regularly performs target counting inference on the low-resolution video frames sent by the terminal device, and then calculates the density map. Returns the terminal device. Density map It directly reflects the distribution of targets in the image. Bright areas indicate high target density, while dark areas indicate sparse or no target areas.

[0051] In a specific embodiment, the terminal device filters out the regions of interest in irrelevant background from the first region of interest set in combination with the density map to obtain a second region of interest set, including: Step 2.2.1: Divide the original video image vertically into multiple regions, and determine the upper limit value of each region based on the density map.

[0052] In this embodiment, the original video image Vertically divided into regions, each with a width of:

[0053] in Original video image width.

[0054] For each area , traverse all columns (pixel columns) in the area, find the first pixel with non-zero density value from top to bottom in the vertical direction of each column, and then select the pixel with the largest y-axis coordinate value from these non-zero density pixels, and use its y-axis coordinate value as the upper limit of the area.

[0055] structure The set of upper bound values ​​for a region:

[0056] in It is The upper bound of the region.

[0057] It should be noted that in high-resolution outdoor video images, the top area usually does not contain the target to be detected, and is mainly composed of irrelevant background such as buildings, sky, and trees. Therefore, it is necessary to filter out the area of ​​interest in the irrelevant background to avoid redundant calculations. The density map generated by CountGD can more accurately reflect the target distribution in the video image, and the upper bound set can be calculated through the density map. To distinguish irrelevant background from relevant background, each area The corresponding upper bound value Indicates the dividing line between irrelevant background and relevant background in the area.

[0058] Step 2.2.2: Determine the region to which each ROI in the first ROI set belongs. For any ROI, if its bottom boundary is higher than the previous value of the region to which it belongs, remove the ROI from the first ROI set to obtain a second ROI set.

[0059] This step traverses all areas of interest , first determine The region to which you belong , to determine whether its bottom boundary exceeds the upper boundary of the area If it exceeds, it will be removed from Remove it from the list to get the updated set of regions of interest , which is the second set of regions of interest.

[0060] Step S3: Divide the original video image into blocks, match each ROI in the second ROI set to each block, and generate common slices based on the blocks containing the ROI.

[0061] After filtering out the area of ​​interest from the irrelevant background, the video image needs to be Further divided into multiple chunks to facilitate parallel offloading.

[0062] First set the number of levels for the blocks (ie the number of elements in the upper bound set), determines the vertical number of blocks , then based on the block shape Sum value set , the video image Divided into blocks, denoted as , the width of each block is , the height is .

[0063] It should be noted that the upper limit value is deducted when calculating the height of each block. , the irrelevant background can be excluded, that is, the blocks can be directly divided based on the relevant background, thereby reducing invalid redundant calculations.

[0064] After dividing the video image into blocks, the filtered area of ​​interest needs to be Assigned to First, traverse all the areas of interest ,use The function calculates its relationship with all blocks The overlapping area , then Assigned to the block with the largest overlapping area , and The collection of regions of interest added to this block In this way, each block containing the region of interest can be obtained, and the blocks that do not contain the region of interest can be discarded.

[0065] In a specific embodiment, generating common slices based on blocks containing regions of interest includes: Step 3.1: Calculate the minimum bounding rectangle of each block containing the region of interest.

[0066] In order to ensure that each block contains all the assigned regions of interest and excludes the background as much as possible, its minimum bounding rectangle needs to be calculated. , calculate the minimum bounding rectangle of all regions of interest in the block , ensuring that the background area is minimized to optimize computational efficiency.

[0067] Step 3.2: Expand each minimum bounding rectangle, and use the expanded bounding rectangle to crop the corresponding area from the original video image as a common slice.

[0068] Since the region of interest extracted based on Gaussian mixture distribution may have errors (such as target shadows and subtle movements cannot be detected), the extracted region of interest may not be able to fully represent the entire target. This application also adopts an expansion strategy to adjust the block size.

[0069] Set the expansion ratio , then use Function extends the minimum enclosing rectangle , and get the expanded bounding rectangle.

[0070] Then based on the expanded bounding rectangle, in the original video image The corresponding area is cut out to obtain the normal slice corresponding to each circumscribed rectangle.

[0071] Step S4: Based on the density map, identify the dense areas in the common slices, and combine all the dense areas into a dense slice.

[0072] Based on density map Further identify the dense areas that may exist in ordinary slices, and splice and divide the dense areas of the video frame.

[0073] First, based on the density map, we identify dense areas in the common slices. That is, we define the areas with extremely high target density as dense areas and set the density threshold. To quantify the dense area. Through the thresholding operation, the dense area is extracted from the density map:

[0074] in It is a binary mask of dense areas, where a pixel value of 255 represents a dense area and a pixel value of 0 represents a non-dense area.

[0075] Next, in the binary mask Apply contour detection on it to get the boundaries of dense regions:

[0076] in is the set of contours of all dense areas, each contour is a closed boundary that represents the shape of a dense area.

[0077] In order to obtain the spatial location information of the dense area, the bounding rectangle of each contour is calculated:

[0078] in is the minimum bounding rectangle of the dense area, is the coordinate of the upper left corner of the rectangle, is the width and height of the rectangle.

[0079] Furthermore, in order to ensure that the extracted dense area contains a sufficient number of targets, in a specific embodiment, a target number threshold is introduced. If the number of targets in a dense area is less than the threshold, the area is considered invalid and filtered.

[0080] Number of targets It can be estimated by integrating the density map over the region:

[0081] in Represents the target density value of each pixel in the density map; by integrating the density map, the number of targets in the area can be calculated.

[0082] if , then the area is retained; otherwise, the area is filtered out.

[0083] After filtering the number of targets, the final dense area set is obtained , containing all valid dense regions and their corresponding rectangular bounding boxes:

[0084] in Represents a bounding box The target number.

[0085] Finally, all dense areas are spliced ​​into a dense slice. Find a minimum dense slice size that can include all dense areas. By reducing the size of the slice, transmission resources and server resources can be saved. Use binary search to determine the size that can accommodate all dense areas. The side length of the smallest square patch First, for all dense areas Sort them in descending order according to their maximum dimension (i.e. the larger of the maximum width and maximum height). Specifically, calculate each dense region Width and height , then according to the formula Sort Area , to ensure that the largest regions are prioritized. Then, a binary search is used to find the appropriate dense shard size. At the beginning of the binary search, set For all dense areas The maximum value of the maximum dimension, that is, ,and All dense areas The square root of the sum of the areas of Next, by calling the helper function , check the current dense shard size Can it accommodate dense area collections? All regions in . It will determine whether all areas can be placed within the given square size. If so, it will indicate the current size. If it works, continue to narrow the search range; if not, expand the search range; if not, expand the search range until the appropriate minimum dense shard size is found . Finally, the obtained dense shard size is .

[0086] After determining the optimal dense shard size Then, each dense area Arrange to this The dense region is placed starting from the upper left corner of the dense region and filling the dense region row by row and column by column. Specifically, first check whether the current row can accommodate the dense region. If the width of the current line plus the dense area The width of the dense shards will exceed the width of the dense shards , it means that a new row needs to be opened and the dense area is placed at the beginning of the next row (update He Xing Gao If the current row can still accommodate the dense area, the dense area will be placed at the end of the current row and updated. is the horizontal coordinate of the current row. Every time a dense area is placed, the position offset of the dense area is recorded and stored in the offset When all dense regions are successfully arranged in dense slices, two results are returned: the first is the final arrangement containing all dense regions , which is the final arrangement layout of the dense area; the second is a set of records of the position offset of each dense area ,These offsets indicate the specific position of each dense region in the final dense shard.

[0087] Step S5: Unload the common shards and dense shards to different edge computing servers for target detection.

[0088] This embodiment offloads different shards to edge computing servers that deploy different detection models and ensures that servers with different computing capabilities can complete tasks efficiently, thereby minimizing the overall inference delay.

[0089] For example, the principle of matching computing power with task size is adopted, that is, servers with stronger computing power process shards with larger computing loads to reduce task execution time; servers with weaker computing power process shards with smaller computing loads to avoid low-performance servers becoming system bottlenecks.

[0090] Normal shards are offloaded to The target detection task is performed by an edge computing server deployed with lightweight YOLOv8s, while the dense sharding is offloaded to an edge computing server deployed with YOLOv8x that is specially optimized for dense targets to improve detection accuracy.

[0091] It should be noted that this application divides the original video image into ordinary slices and dense slices, and then sends them to different edge computing servers for recognition. For ordinary slices, only the edge computing server with lightweight YOLOv8s is deployed for target recognition, while for dense slices, the edge computing server with optimized YOLOv8x is selected for dense target recognition. The combination of the two improves the recognition ability of small and dense targets. And when there is no dense slice, only ordinary slices are needed for recognition, saving server resources.

[0092] In a specific embodiment, in order to improve the small target detection capability of YOLOv8x in dense slices, YOLOv8 is improved based on the HIC-YOLOv5 architecture and named YOLOv8x-crowd. Its network structure is as follows Figure 2 As shown in the figure, it includes a backbone network (Backbone), a neck network (Neck) and a detection head (Head). It introduces a small object detection head (SODH) to enhance the small target detection capability, and adopts a convolutional block attention module (CBAM) and an involution module to optimize feature extraction and multi-scale feature fusion.

[0093] First, a CBAM module is added before the SPPF module in the backbone network.

[0094] The attention mechanism can effectively enhance key features and suppress irrelevant background information. However, the Transformer architecture is computationally intensive, making it unsuitable for real-time detection tasks. Directly adding CBAM to the Neck layer also incurs high computational overhead. Therefore, we chose to add CBAM before the fast-forward spatial pyramid pooling module (SPPF) at the end of the Backbone layer to enhance feature extraction while maintaining computational efficiency. Specifically, CBAM consists of channel attention and spatial attention. The channel attention module calculates channel importance through max pooling and average pooling, and uses a fully connected network and sigmoid function to calculate weights, thereby strengthening key channels and suppressing irrelevant channels. The spatial attention module uses 3×3 convolution and sigmoid function to calculate a spatial attention map, enhancing the feature representation of the target area while suppressing background interference. Placing CBAM in the last layer of the Backbone layer, rather than in the Neck structure, avoids performing CBAM calculations on large-scale feature maps, thereby reducing computational overhead. This also enhances the features extracted by the Backbone and makes the features processed by the Neck more effective.

[0095] Secondly, the Involution module is inserted after the SPPF module in the backbone network and between it and the neck network.

[0096] YOLOv8's Neck structure uses FPN+PAN for multi-scale feature fusion. However, there is still room for improvement in channel information processing. The original YOLOv8 uses 1×1 convolution for channel compression in the early Neck stage. Although this reduces the amount of computation, it also loses some key information, affecting the feature fusion effect. To this end, an Involution module is inserted between Backbone and Neck, as shown in the following example. Figure 2 As shown in the figure, the Involution module is inserted after the SPPF module to improve the channel feature expression capability. Compared with the traditional 1×1 convolution, Involution allows independent convolution kernels to be used in different spatial locations, thereby improving the ability to extract local features and making the features of small objects clearer.

[0097] Finally, a small target detection head is added to the detection head, and a feature fusion branch that fuses higher resolution feature maps is added to the neck network and output to the small target detection head for detection.

[0098] YOLOv8 uses a single detection head for multi-scale target detection, but its default feature map resolution still has certain limitations for dense small target detection. In order to enhance the detection capability of extremely small targets (occupying 0.1% to 1% of the image area), a small object detection head (SODH) is added on the basis of the original detection head. At the same time, a feature fusion branch that integrates higher resolution feature maps is added to the neck network, namely Figure 2The top layer of the mid-neck network, consisting of the upsampling, concat, and C2f modules, outputs higher-resolution fused features to the small object detection head for object detection. YOLOv8 extracts features through the C2f modules in the backbone network, outputting 80*80, 40*40, and 20*20 features. In this embodiment, the backbone network outputs an additional 160*160 feature, which is fed into the feature fusion branch added to the neck network and then input into the small object detection head for object detection.

[0099] SODH uses additional high-resolution feature maps for prediction to ensure that small target features are not easily lost in the deep network, and regenerates anchor boxes adapted to small targets through K-means to improve detection accuracy.

[0100] Through the above optimization, we obtained the object detection model YOLOv8x-crowd, which is specially optimized for dense areas. Finally, we trained it using the training set to ensure its detection performance in high-density scenes.

[0101] In one specific embodiment, all slices from the terminal device are resized uniformly on the edge computing server before being fed into the object detection model for inference. This eliminates the impact of input size variations on inference efficiency, thereby optimizing end-to-end latency.

[0102] Over-reducing the shards may lead to loss of target features and reduce detection accuracy; over-enlarging the shards may increase the amount of computation and increase inference latency. Therefore, it is necessary to strike a balance between shard size and computational efficiency. Therefore, the fixed size is dynamically adjusted according to the sharding strategy. The specific calculation formula is as follows:

[0103]

[0104] in is the number of horizontal shards when sharding. is the number of vertical slices during slice processing, and 3840×1920 is the resolution of the original video frame.

[0105] Through this formula, all shards are scaled to the same size to adapt to the input requirements of the inference model, while reducing the dynamic adjustment of the computational graph, thereby improving inference stability and optimizing overall system performance.

[0106] After completing object detection, the edge computing server returns the detection results to the terminal device. After receiving all detection results, the terminal device integrates them. Because the shards are appropriately expanded during the segmentation process to ensure complete coverage of the target object, this may result in overlapping areas between adjacent shards. To address this issue, a two-step optimization strategy, box filtering and non-maximum suppression (NMS), was designed. Box filtering removes small boxes that are almost completely contained by other boxes to reduce redundant detections. NMS merges highly overlapping detection boxes to ensure uniqueness in the final detection results.

[0107] For each pair of detection boxes in the overlapping area and , their intersection area The calculation is shown as follows:

[0108] Set target frame The area is:

[0109] If the intersection area occupies the frame The area ratio exceeds the set threshold , then the box is considered redundant and should be deleted. That is:

[0110] Similarly, if the frame satisfy:

[0111] Box will also be deleted.

[0112] This process can effectively remove nested boxes, avoid multiple overlapping boxes representing the same target, and improve the accuracy of target detection results.

[0113] After frame filtering and NMS processing, the final target detection result is obtained. The merged detection frame is rendered back to the original video image , forming the final detection results and completing the complete video analysis process.

[0114] In order to verify the technical solution of this application, experiments were conducted on the constructed dataset. Figure 3The accuracy comparison charts for the present invention's technical solution and comparative methods in different scenarios are presented. PACOD is the edge computing offload method for vision applications proposed in the present invention; the Full Frame method directly offloads raw 4K resolution video frames to edge servers for sequential inference without preprocessing or modification; the Masked Frame method masks the portions of the video image that do not contain the region of interest, transmitting only the potential target areas at 4K resolution to the edge server to reduce computational burden and bandwidth requirements; and the ELF method divides the video frame into multiple smaller slices and offloads these slices in parallel to multiple edge servers for processing, thereby improving processing efficiency. In five scenarios, the present invention's accuracy only slightly decreases compared to the Full Frame method, and the decrease is within an acceptable range. In the remaining five scenarios, the present invention even surpasses the Full Frame method, demonstrating its robustness and ability to maintain high detection accuracy across a variety of scenarios. Compared to Masked Frame and ELF, the present invention leads in accuracy in most scenarios, both in sparsely populated and densely populated scenes, further validating the effectiveness of the method. In particular, this application achieved significant accuracy improvements in densely populated scenes (such as Scene_02, Scene_03, Scene_06, Scene_08, and Scene_10). Compared to Masked Frame, mAP increased by 8.1%, 9.2%, 1.3%, 18.8%, and 3.8%, respectively; and compared to ELF, mAP increased by 6.2%, 6.4%, 0.9%, 10.4%, and 2.5%, respectively. This demonstrates that this application significantly optimizes small object detection in complex, high-density scenes, effectively reducing the problem of missed small object detection.

[0115] Figure 4 The following chart shows the latency comparison of the technical solution of this application in different scenarios and the comparison method. This application achieves the lowest end-to-end latency in all scenarios. Compared with Full Frame, the latency is reduced by 60.2% to 76.8%, achieving a 2.5x to 4.3x acceleration. This application achieves a good balance between accuracy and latency in scenarios with different crowd densities and is suitable for deployment on resource-constrained edge devices.

[0116] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An edge computing offloading method for visual applications, applied to terminal devices, characterized in that: The edge computing offloading method for visual applications includes: Collect the original video image, downsample it to obtain a low-resolution video image, send the low-resolution video image to the edge computing server deployed with the multimodal open-world counting model for target counting processing, and obtain the returned density map; A background model is established based on the historical color distribution of pixels, and regions of interest are extracted from the original video image to obtain a first set of regions of interest. Then, the regions of interest in the irrelevant background are filtered out from the first set of regions of interest by combining the density map to obtain a second set of regions of interest. Divide the original video image into blocks, match each region of interest in the second region of interest set to each block, and generate common slices based on the blocks containing the region of interest; Based on the density map, we identify the dense areas in the common slices and combine all the dense areas into a dense slice. Normal sharding and dense sharding are offloaded to different edge computing servers for target detection.

2. The edge computing offloading method for visual applications according to claim 1 is characterized in that: The process of establishing a background model based on the historical color distribution of pixels and extracting regions of interest from the original video image to obtain a first set of regions of interest includes: Match each pixel in the original video image with each Gaussian distribution in the historical color distribution, and then update the parameters of each Gaussian distribution; A Gaussian distribution with a total weight greater than the background ratio threshold is selected as the background model. If a pixel in the original video image belongs to the background model, it is considered background; otherwise, it is considered foreground, and a foreground mask is generated. Based on the foreground mask, a first set of regions of interest is obtained.

3. The edge computing offloading method for visual applications according to claim 2 is characterized in that: After obtaining the first set of regions of interest based on the foreground mask, the method further includes: Small noise is filtered out by setting a minimum area threshold to extract the final first set of regions of interest.

4. The edge computing offloading method for visual applications according to claim 1 is characterized in that: The combined density map is used to filter out regions of interest in irrelevant background from the first region of interest set to obtain a second region of interest set, including: The original video image is vertically divided into multiple regions, and the upper limit value of each region is determined based on the density map; The region to which each ROI in the first ROI set belongs is determined. For any ROI, if its bottom boundary is higher than the previous value of the region to which it belongs, the ROI is removed from the first ROI set to obtain a second ROI set.

5. The edge computing offloading method for visual applications according to claim 1 is characterized in that: Generating common slices based on blocks containing regions of interest includes: Calculate the minimum bounding rectangle of each block containing the region of interest; Each minimum bounding rectangle is expanded, and the corresponding area is cropped from the original video image using the expanded bounding rectangle as a common slice.

6. The edge computing offloading method for visual applications according to claim 1, characterized in that: The method of offloading common shards and dense shards to different edge computing servers for target detection includes: Offload common shards to edge computing servers deployed with lightweight YOLOv8s; Offload dense sharding to edge computing servers where optimized YOLOv8x is deployed.

7. The edge computing offloading method for visual applications according to claim 1 is characterized in that: The optimized YOLOv8x includes a backbone network, a neck network, and a detection head, wherein: Add the CBAM module before the SPPF module in the backbone network; Insert the Involution module after the SPPF module in the backbone network and between it and the neck network; A small target detection head is added to the detection head, and a feature fusion branch that fuses higher-resolution feature maps is added to the neck network and output to the small target detection head for detection.

Images