A cloud-edge collaborative method, device and product for target detection

Abstract

The application provides a cloud-edge collaborative method, device and product for target detection, and relates to the field of target detection. The method comprises the following steps: collecting original pictures by using an edge device, equally dividing the original pictures into multiple sub-pictures, cutting the original pictures to generate cut pictures; uploading all the cut pictures and the service level targets corresponding to all the cut pictures to a cloud scheduler and a cloud server; deploying a target detection deep learning model functionless service on the cloud server; defining one or more canvases by the cloud scheduler according to the number of cut pictures and the video memory of the target detection deep learning model functionless service, and adopting a boxing algorithm to splice all the cut pictures into the canvases; and inputting the canvases spliced with the cut pictures into the target detection deep learning model functionless service for batch processing. The application can meet the service level targets of batch processing and realize processing of all batch requests.

Description

Technical Field

[0001] This invention relates to the field of target detection, and in particular to a cloud-edge collaborative method, apparatus, and product for target detection. Background Technology

[0002] Object detection technology using high-definition surveillance cameras is a key technology in computer vision and artificial intelligence research. It has important applications in many fields, including video surveillance, autonomous driving, and human-computer interaction. Currently, various countries and regions are deploying cameras for video analytics on a large scale. For example, in urban commercial facilities, residences, public spaces, and other facilities, high-definition video surveillance cameras are used for theft prevention, vandalism prevention, robbery prevention, and other security and protection measures. In streets, highways, parks, and other public places, high-definition video cameras are used to monitor traffic flow, vehicle speed, and traffic violations. Cameras are installed in corporate campuses (such as company offices and retail stores) to improve business operations and the shopping experience. In most video analytics applications, object detection plays a crucial role because it can identify and locate all specific categories of object instances in an image.

[0003] In recent years, with the development of deep learning, especially the rise of Convolutional Neural Networks (CNNs), object detection technology has made significant progress. Compared with traditional methods, deep learning can automatically learn complex and abstract features in data, enabling it to maintain good detection performance even in complex scenes. R-CNN, Fast R-CNN, Faster R-CNN, YOLO, and SSD are some representative works of deep learning in the field of object detection. In the context of high-resolution camera object detection, there is a challenge in detecting small objects in high-definition images: due to the extremely dense pixel distribution, small objects occupy only a small portion of the entire image, and the information content of small objects is far less than that of the background, making the model more susceptible to the influence of the background and causing it to ignore the existence of small objects. Processing high-definition images requires significant computational resources, such as memory, storage, and computing power. Under current hardware conditions, processing large numbers of high-definition images remains a challenge.

[0004] However, most high-definition cameras or edge computing devices on the market currently have limited performance and cannot directly apply deep learning models, mainly in the following aspects: 1) Processor performance: The processor (CPU or GPU) performance of most edge devices is usually weaker than that of high-end servers or personal computers. This limits the ability of these devices to run complex deep learning models, as these models often require a large amount of computing resources for forward propagation (prediction) or backpropagation (training). 2) Memory size: Deep learning models, especially large, high-precision models, require a large amount of memory to store model parameters, intermediate calculation results, etc. Edge devices usually have less memory than servers or personal computers, making it impossible to deploy deep learning models on them. 3) Power limitations: Compared to devices in data center or desktop environments, edge devices usually need to be more careful about power usage. Large deep learning models may consume a lot of power during operation, which may be impractical for edge devices that rely on battery power. 4) Storage space: The weight files of deep learning models often require a large amount of storage space, which edge devices often cannot provide.

[0005] Therefore, the computing power of edge devices alone cannot meet the requirements of high-definition video target detection tasks. In practical applications, to fully utilize the powerful computing capabilities of cloud computing and the low latency advantages of edge computing, researchers have begun to explore cloud-edge collaborative computing models. In this model, computing tasks are dynamically allocated between the cloud and the edge as needed to achieve an optimal balance between efficiency and real-time performance. Specifically, edge devices or cameras with certain computing capabilities are responsible for video encoding and transmission, while cloud servers are responsible for deep learning model inference on the video frames transmitted from the edge. This is the common cloud-edge collaborative computing method for high-definition camera target detection.

[0006] In cloud-edge collaborative computing, video frames typically need to be frequently transmitted between the cloud and edge devices. Secondly, calculations show that, assuming a common bitrate of 35Mbps, transmitting one minute of 4K 30fps video would require approximately 260MB, a significant overhead given the limited bandwidth resources of edge devices. Furthermore, high-definition video object detection technologies involve a trade-off between latency and accuracy. In deep learning algorithms for object detection, the execution speed and accuracy are related to video resolution: higher resolution input video frames generally result in higher accuracy, but longer inference time; conversely, smaller image resolutions decrease inference accuracy but reduce inference time. Achieving a balance between latency and accuracy while effectively reducing bandwidth consumption during transmission is a highly challenging problem.

[0007] In cloud computing, FaaS (Function as a Service) and IaaS (Infrastructure as a Service) are two main service models, each with its own characteristics and application scenarios. IaaS is the most basic form of cloud service, providing fundamental computing resources such as servers (virtual machines or bare metal), storage, and networks. In the IaaS model, users can rent infrastructure resources from cloud providers and install operating systems, runtime environments, middleware, applications, etc., on them. Users have complete control over most layers of the entire stack but are also responsible for management and maintenance (e.g., operating system maintenance, network configuration, security settings, etc.). FaaS is a service model that implements a serverless architecture. In the FaaS model, users only need to focus on their business logic, writing it as one or more functions and uploading it to the cloud platform. The cloud platform is responsible for running these functions, and charges are only incurred when the functions are executed; no charges are incurred when they are not in use. Users do not need to manage any underlying infrastructure or even care about the runtime environment. FaaS is well-suited for scenarios requiring rapid scaling and high elasticity, such as microservices, real-time file processing, and data stream processing.

[0008] In high-definition camera target detection cloud-edge collaborative computing, the workload is related to the number of cameras, the number of video targets, and the resolution. In reality, the workload is often highly dynamic. If an IaaS computing model is used, it is easy to cause problems of over-allocation or under-allocation of resources, resulting in excessive costs or failure to meet target requirements. FaaS can effectively solve the problem of dynamic workloads, and it also supports rapid expansion, making it very suitable for high-definition camera target detection cloud-edge collaborative computing scenarios. In addition, due to FaaS's special pay-as-you-go billing model, which is mainly based on function execution time and resource usage, it has lower overhead compared to renting servers. Specifically, the billing model usually includes the following aspects: 1) Execution time: FaaS providers usually charge according to the function's execution time, and it is usually calculated in time intervals of 100 milliseconds or less. As long as the function is running, a fee will be incurred, and no fee will be incurred when the function is idle. 2) Resource allocation: In addition to execution time, FaaS costs also depend on the amount of resources allocated to the function, such as CPU, memory, and network. Generally speaking, the more resources allocated, the higher the cost. 3) Number of requests: Some FaaS providers also charge based on the number of requests for a function, that is, the number of times the function is called.

[0009] In the cloud-edge collaborative computing model for high-definition camera object detection, deploying deep learning models on a serverless architecture allows for the use of batch requests to further reduce costs and the number of function calls, thereby reducing the overhead of starting functions and managing their lifecycles. Each function call requires some start and end time; merging multiple requests into a single function call can significantly improve efficiency. FaaS typically charges based on the number of function calls and execution time. Batch processing reduces the number of function calls and overall execution time, thus reducing costs. Furthermore, when performing inference on GPUs or other hardware supporting parallel computing, batch inference can fully utilize hardware resources, significantly improving computational efficiency and reducing the cost and time of a single image inference. However, excessively large batch sizes can cause request waiting, potentially violating Service Level Objectives (SLOs). Secondly, excessively large batch sizes consume more GPU memory; if this exceeds the GPU memory resources available to the unserved instances of the object detection deep learning model's service functions, the batch of requests cannot be processed. Summary of the Invention

[0010] The purpose of this invention is to provide a cloud-edge collaborative method, device, and product for target detection, in order to solve the problems that excessively large batch sizes may violate service level objectives and that excessively large batch sizes will occupy more video memory, resulting in batch requests being unable to be processed.

[0011] To achieve the above objectives, the present invention provides the following solution:

[0012] A cloud-edge collaborative method for object detection includes: acquiring an original image using an edge device and dividing the original image into multiple sub-images; segmenting the original image based on the sub-images to generate slices; uploading all slices and their corresponding service level targets to a cloud scheduler, and uploading the service level targets to a cloud server; deploying a service-free function of a deep learning model for object detection on the cloud server; the service-free function of the deep learning model for object detection satisfying the service level targets of all slices; defining one or more canvases based on the number of slices and the GPU memory of the service-free function of the deep learning model for object detection using the cloud scheduler, and stitching all slices into the canvases using a binning algorithm; and inputting the canvases with the stitched slices into the service-free function of the deep learning model for object detection for batch processing.

[0013] Optionally, the original image is acquired using an edge device, and the original image is divided into multiple sub-images. Specifically, this includes: performing a Gaussian mixture background modeling algorithm on the original image to identify foreground objects in the original image, and drawing a target rectangle for each target; the foreground object is a moving target; dividing the original image into multiple sub-images; initializing a sub-image list for each sub-image; and using the sub-image list to store target information.

[0014] Optionally, based on the sub-image, the original image is segmented to generate a slice, specifically including: determining the sub-image to which the foreground object belongs based on the bounding box of the target, and storing it in the corresponding sub-image list; based on the sub-image list, calculating the position of the largest bounding rectangle based on all target bounding boxes contained in each sub-image, and segmenting the largest bounding rectangle from the original image to generate a slice.

[0015] Optionally, the image-stitched canvas is input into the no-server function of the object detection deep learning model for batch processing. Prior to this, the process includes: in the offline phase, inputting a typical image-stitched canvas into the no-server function of the object detection deep learning model for batch processing, and recording the time spent processing images of each batch size by the no-server function; in the online phase, whenever the cloud scheduler receives an image slice sent from the edge device, it treats it as a batch request, saving all the slices to a cache queue, recording resolution information, request start time, and service level target; using a binning algorithm, the slices in the cache queue are stitched into a canvas. If the current canvas cannot meet the binning requirements, a new canvas is created, until... The system can stitch together all image slices from the cache queue and record the number of canvases. It calculates a relaxation time based on the number of canvases and the time taken for each batch of images to be processed by the no-server function of the object detection deep learning model. The relaxation time is the time taken to input all current canvases as a batch into the no-server function of the object detection deep learning model. It calculates the deadline for all requests based on the request start time and the service level target. It calculates the start execution time of the no-server function of the object detection deep learning model based on the deadline and the relaxation time. At the start execution time of the no-server function of the object detection deep learning model, the canvases stitched together with image slices are input into the no-server function of the object detection deep learning model for batch processing.

[0016] Optionally, the start execution time of the no-service function of the object detection deep learning model. for: ;in, The deadline is [date / time]. , To request a start time, For service level objectives; For relaxation time, , For the number of canvases, The time spent by the object detection deep learning model without a service function for processing images of each batch size.

[0017] Optionally, the canvas with stitched images is input into the no-service function of the object detection deep learning model for batch processing. Before this, the process further includes: when the cloud scheduler receives a new batch request, obtaining the current time of receiving the new batch request, but if the current time exceeds the start execution time of the no-service function of the object detection deep learning model, immediately inputting the previous batch requests into the no-service function of the object detection deep learning model for batch processing, and clearing the cache queue, retaining only the new batch requests.

[0018] Optionally, the process of inputting the spliced ​​canvases into the no-server function of the object detection deep learning model for batch processing includes the following steps: when the cloud scheduler receives a new batch request, but the number of canvases is greater than or equal to the maximum number of canvases that the no-server function of the object detection deep learning model can process simultaneously, the previous batch request is immediately input into the no-server function of the object detection deep learning model for batch processing, and the cache queue is cleared, retaining only the new batch request.

[0019] A cloud-edge collaborative device for object detection includes: an edge device for acquiring an original image and dividing the original image into multiple sub-images; based on the sub-images, cutting the original image to generate slices, uploading all slices to a cloud scheduler, and uploading the service level targets corresponding to all slices to a cloud server; the cloud server for deploying a service-free function of a deep learning model for object detection; the service-free function of the deep learning model for object detection satisfies the service level targets of all slices; the cloud scheduler for defining one or more canvases according to the number of slices and the GPU memory of the service-free function of the deep learning model for object detection, and using a binning algorithm to stitch all slices into the canvases; and inputting the canvases stitched with slices into the service-free function of the deep learning model for object detection for batch processing.

[0020] An electronic device includes a memory and a processor, the memory storing a computer program and the processor running the computer program to enable the electronic device to perform the cloud-edge collaborative method for target detection described above.

[0021] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned cloud-edge collaborative method for target detection.

[0022] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: The present invention provides a cloud-edge collaborative method, device and product for object detection, which cuts the original image into multiple slices, stitches them together on a canvas for batch processing, and deploys a no-service function of a deep learning model for object detection. The number of canvases is defined according to the number of slices and the video memory of the no-service function of the deep learning model for object detection. Then, the canvas with the slices stitched together is input into the no-service function of the deep learning model for object detection to process all batch requests. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the cloud-edge collaborative method for target detection provided by the present invention;

[0025] Figure 2 This is a schematic diagram of the cloud-edge collaborative device for target detection provided by the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] The purpose of this invention is to provide a cloud-edge collaborative method, device, and product for target detection that can meet the service level target of batch processing and achieve the processing of all batch requests.

[0028] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] In real-world high-definition surveillance camera (4K) videos, it can be observed that cameras are often placed in fixed positions to capture images from fixed angles and ranges, focusing only on target areas such as pedestrians and vehicles. Experimental studies show that in the PANDA high-definition video dataset, target (pedestrian) pixel areas account for less than 10% of the total image pixels. This means that most areas in the video, such as the sky, ground, and buildings—static backgrounds—are not regions of interest, and targets are unlikely to appear in areas with the sky or buildings as backgrounds. Based on these observations, a Gaussian Mixture Modeling (GMM) algorithm is used to extract the target regions of interest from the captured video frames. GMM is a commonly used method for building dynamic background models in video surveillance systems. This method assumes that the color change of each pixel over time can be described by a Gaussian mixture model, meaning that the color value of each pixel in the time series is generated by a combination of multiple Gaussian distributions. Therefore, the target regions extracted by GMM coarsely filter out most background areas, significantly reducing the network bandwidth consumed by transmitting video frames at the edge.

[0030] In practical applications, a Gaussian mixture model (GMM) is typically maintained for each pixel. Each Gaussian distribution corresponds to a possible color value and its probability of occurrence, while the GMM represents the overall distribution of these color values. When a new frame arrives, the GMM corresponding to each pixel is updated to adapt to changes in the background. Since using only Gaussian mixture background modeling is inaccurate for identifying extremely small pedestrian targets in the distance of the video, or because GMMs are prone to missing some targets, this invention designs an adaptive subgraph segmentation method based on GMM to preserve the dynamic foreground information in the video to the greatest extent possible while removing the static background.

[0031] In the cloud, leveraging the rapid scalability and highly elastic, pay-as-you-go nature of serverless systems, a state-of-the-art Yolov8x object detection deep learning model serverless function is deployed. Only the required resources, including the number of vCPUs, memory size, and GPU memory size, need to be set during function deployment. The function takes the image to be detected as input and outputs the target location on the image. As mentioned earlier, batch processing technology can further reduce costs and the number of function calls, thereby reducing the overhead of starting and managing function lifecycles; it fully utilizes hardware resources, significantly improves computational efficiency, and amortizes and reduces the cost and time of single image inference. However, excessively large batch sizes can cause requests to wait, potentially violating Service Level Objectives (SLOs), i.e., missing the deadline for a particular image in the batch. Secondly, excessively large batch sizes consume more GPU memory; if this exceeds the GPU memory resources available to the serverless function instance of the object detection deep learning model, the batch of requests cannot be processed. Therefore, how to reasonably batch requests is a challenging problem. This invention designs an SLO-aware algorithm on the serverless function instance side of the cloud-based object detection deep learning model, which can maximize the advantages of batch processing without violating the SLO of any single request.

[0032] To better leverage the advantages of batch processing and handle image slices of varying resolutions transmitted from edge devices, this invention employs a mosaicking method to replace large-scale batch operations. Specifically, a size is defined in the cloud. A blank (black) canvas of a specified size. Requested images from edge devices are stitched onto this canvas, ensuring they cannot be rotated or overlapped, and that the canvas can hold as many images as possible.

[0033] This problem is equivalent to the 2D bin packing problem, a combinatorial optimization problem belonging to the NP-hard category of computational complexity theory, for which there is currently no known multinomial-time solution. In the 2D bin packing problem, there are one or more fixed-size 2D "boxes" (or containers, usually rectangles), and a series of 2D "items" (also usually rectangles) to be placed into the boxes. Items can be rotated or not, but cannot overlap or extend beyond the box boundaries. The goal of this invention is to find a way to put all items into as few boxes as possible. If the current canvas cannot hold all the images, a new canvas of the same size is created. The blank canvas of the specified size is then used for further packing. This method effectively batch processes images sent from edge devices, with the batch size equal to the number of canvases. This approach significantly improves batch processing throughput and effectively avoids the impact on inference accuracy caused by resizing requested images for the sake of simple batch processing.

[0034] When calculating the billing for serverless function instances of cloud-based object detection deep learning models, the widely accepted Alibaba Cloud serverless pay-as-you-go billing formula is used. Pay-as-you-go is a billing method that charges after use, with milliseconds as the billing unit. Specifically:

[0035]

[0036] in, The price at which the function is triggered. For the number of calls, For the duration of the function, , , These represent the number of CPU cores, the number of 1024MB units of RAM, and the number of 1024MB units of video memory, respectively. , and Price per unit of CPU, memory, and GPU Price for a single function call.

[0037] Example 1

[0038] like Figure 1 As shown, the present invention provides a cloud-edge collaborative method for target detection, comprising the following steps.

[0039] Step 101: Use an edge device to capture the original image and divide the original image into multiple sub-images.

[0040] Step 101 specifically includes: performing a Gaussian mixture background modeling algorithm on the original image to identify foreground objects in the original image, and drawing a target rectangle for each target; the foreground object is a moving target; dividing the original image into multiple sub-images; initializing a sub-image list for each sub-image; the sub-image list is used to store target information.

[0041] Step 102: Based on the sub-image, the original image is segmented to generate a slice image.

[0042] Step 102 specifically includes: determining the sub-image to which the foreground object belongs based on the bounding box of the target, and storing it in the corresponding sub-image list; based on the sub-image list, calculating the position of the largest bounding rectangle according to all target bounding boxes contained in each sub-image, and cutting out the largest bounding rectangle from the original image to generate a cropped image.

[0043] Step 103: Upload all image slices and their corresponding service level targets to the cloud scheduler, and upload the corresponding service level targets to the cloud server.

[0044] Step 104: Deploy the no-service function of the object detection deep learning model using the cloud server; the no-service function of the object detection deep learning model satisfies the service level objectives of all the sliced ​​images.

[0045] Step 105: Using the cloud scheduler, define one or more canvases based on the number of slices and the GPU memory of the no-service function of the object detection deep learning model, and use a bin packing algorithm to stitch all the slices into the canvas.

[0046] Step 106: Input the spliced ​​canvas with cropped images into the no-service function of the object detection deep learning model for batch processing.

[0047] Before step 106, the method further includes: in the offline stage, inputting a typical spliced ​​canvas with cropped images into the no-server function of the object detection deep learning model for batch processing, and recording the time spent by the no-server function of the object detection deep learning model for each batch size of images.

[0048] In practical applications, typical spliced ​​canvases with cropped images are common, numerous, and statistically significant. This involves randomly generating a large number of spliced ​​canvases, then batching them into inputs of batches n=1, 2, 3, ..., N. The average time taken for these batches to be processed by the no-service function of the object detection deep learning model is then calculated; N is the total number of batches, determined by the canvas size and batch size.

[0049] Each batch size refers to the time spent by the object detection deep learning model without a service function for batches of size (n=1,2,3…..,N).

[0050] This invention can handle image slices of any size and canvases of any size. Therefore, this invention can pre-calculate the average time spent on canvases of different batches of images of the same size during the offline phase.

[0051] During the online phase, whenever the cloud scheduler receives a slice of image sent from the edge device, it saves all the slices to the cache queue as a batch request, recording the resolution information, request start time, and service level target.

[0052] A bin packing algorithm is used to stitch the slices in the cache queue into a canvas. If the current canvas cannot meet the bin packing requirements, a new canvas is created until all the slices in the cache queue can be stitched together, and the number of canvases is recorded.

[0053] The relaxation time is calculated based on the number of canvases and the time taken for each batch of images to be processed by the no-service function of the object detection deep learning model; the relaxation time is the time taken to input all current canvases as a batch into the no-service function of the object detection deep learning model.

[0054] Calculate the deadline for all requests based on the request start time and the service level target.

[0055] The start time of the no-service function of the object detection deep learning model is calculated based on the deadline and the relaxation time.

[0056] At the start of the execution time of the no-service function of the object detection deep learning model, the canvas with stitched images is input into the no-service function of the object detection deep learning model for batch processing.

[0057] In practical applications, the start execution time of the no-service function of the object detection deep learning model. for: .in, The deadline is [date / time]. , To request a start time, For service level objectives; For relaxation time, , For the number of canvases, The time spent by the object detection deep learning model without a service function for processing images of each batch size.

[0058] In practical applications, before step 106, the process further includes: when the cloud scheduler receives a new batch request, it obtains the current time when the new batch request is received, but the current time exceeds the start execution time of the no-service function of the object detection deep learning model, immediately inputs the previous batch requests into the no-service function of the object detection deep learning model for batch processing, and clears the cache queue, retaining only the new batch requests.

[0059] In practical applications, before step 106, the following steps are also included: when the cloud scheduler receives a new batch request, but the number of canvases is greater than or equal to the maximum number of canvases that the no-service function of the object detection deep learning model can process simultaneously, the previous batch requests are immediately input into the no-service function of the object detection deep learning model for batch processing, and the cache queue is cleared, retaining only the new batch requests.

[0060] Example 2

[0061] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a cloud-edge collaborative device for target detection is provided below.

[0062] A cloud-edge collaborative device for target detection includes: an edge device for acquiring an original image and dividing the original image into multiple sub-images; based on the sub-images, cutting the original image to generate slices, uploading all slices to a cloud scheduler, and uploading the service level targets corresponding to all slices to a cloud server.

[0063] The cloud server is used to deploy the no-service function of the object detection deep learning model; the no-service function of the object detection deep learning model satisfies the service level objectives of all the sliced ​​images.

[0064] The cloud scheduler is used to define one or more canvases based on the number of image slices and the GPU memory of the no-server function of the object detection deep learning model, and to use a bin packing algorithm to stitch all the image slices into the canvases; the canvases with stitched image slices are then input into the no-server function of the object detection deep learning model for batch processing.

[0065] In practical applications, such as Figure 2 As shown, on the edge device side:

[0066] Step 1: Capture images taken by a high-definition camera, with a resolution of 4K. Pixel.

[0067] Step 2: Perform Gaussian mixture background modeling algorithm on the image to identify foreground objects in the image, i.e. moving targets, and draw a rectangle around each target.

[0068] Step 3: Divide the graph into equal parts Each sub-image is 3840 / P × 2160 / P pixels in size. Each sub-image is initialized with an empty list to store target information and prepare for adaptive sub-image cutting. P is a positive integer, and the specific value is determined by the user.

[0069] Step 4: For each identified foreground object, determine which sub-image it belongs to. The determination rule is the pixel area of ​​the overlap between the foreground object's rectangle and the sub-image. Save the determined target information, such as the pixel position of the top left corner of the rectangle and the pixel position of the bottom right corner, in the corresponding sub-image list.

[0070] Step 5: For each sub-image, calculate the position of the largest bounding rectangle of all target rectangles, and cut this largest bounding rectangle out of the original image; this is called a slice. If there is no target information in the sub-image list, skip this step. This step completes the adaptive sub-image cutting.

[0071] Step 6: Upload all sliced ​​images to the cloud scheduler, along with the service level target, i.e., time. The requirement is that all subgraphs need to be in time. The subsequent target detection and inference tasks are completed within the system.

[0072] The cloud scheduler includes both offline and online phases.

[0073] Offline phase: Select a typical A layered canvas with different resolutions was used. Each canvas was then batch-processed using a serverless function of an object detection deep learning model. The canvas was copied 1-10 times as batch sizes 1-10, and the processing time for each batch size was recorded. .set up If the batch size is , then for The time taken by the function to process the data at the batch size; this batch size function is a no-service function for the object detection deep learning model.

[0074] Online Phase: Step 1: Whenever the scheduler receives a slice of image from an edge device, it treats it as a request, saves it to the cache queue, and records its resolution and request start time. Service level objectives. The instance immediately stitches together all graphs in the current cache queue. On a black canvas of a certain size, the Guillotine algorithm is used to solve the 2D binning problem, completing the puzzle by stitching together as many image slices as possible onto a single image. If the canvas cannot meet the binning requirements, a new one is created. A new canvas of a certain size is created until it can hold all the slices in the cache queue. At this point, the number of canvases is recorded. .

[0075] Step 2: Calculate the relaxation time Based on the current number of canvases in the scheduler queue puzzle. and offline stage records The relaxation time is calculated using the following formula:

[0076] .

[0077] The relaxation time is represented as the estimated time spent batching all current canvases into a single batch and feeding it into the no-service function of the Yolov8x object detection deep learning model.

[0078] Step 3: Calculate the start time of the no-service function of the object detection deep learning model. The deadlines for all requests can be determined from the scheduler's cache queue. The calculation is as follows:

[0079] .

[0080] Therefore, the deadline for the request is equal to the deadline for the earliest batch of requests.

[0081] The above two formulas can be used to calculate .

[0082] .

[0083] Therefore, it is only necessary to have a time of When the batch of canvases with stitched images is sent to the no-service function of the object detection deep learning model for execution, the SLO of the batch request for all images in the cache queue can be satisfied.

[0084] Cloud server: Used to deploy serverless function instances of deep learning models for object detection.

[0085] If the current scheduler receives a new batch request, but the execution time is calculated according to steps 1-3 in the online phase... If the current time has expired, the request will not be appended to the previous batch; instead, the previous batch of requests will be immediately sent to the no-service function of the object detection deep learning model, and the scheduler will clear the cache queue but retain only the newly received request.

[0086] If the current scheduler receives a new request, but the number of canvases calculated according to step 1 of the online phase... ,in, If the number of canvases that the current object detection deep learning model's no-service function can process simultaneously is the maximum number of canvases that can be processed simultaneously, then the request will not be appended to the previous batch. In this case, the previous batch of requests will be immediately sent to the object detection deep learning model's no-service function, and the scheduler will clear the cache queue but only retain the newly received request.

[0087] Whenever the no-server function instance of the object detection deep learning model receives an image request from an edge device, steps 1-3 in the online phase are performed: repackaging and updating the start time. The cache queue is only cleared when all images are actually sent to the deep learning model for execution, awaiting the next request. Each time the cloud-based object detection deep learning model performs serverless function inference, it needs to be adjusted based on the actual execution time and resource configuration. The formula is used to calculate the cost.

[0088] Example 3

[0089] An electronic device includes a memory and a processor, the memory storing a computer program and the processor running the computer program to cause the electronic device to perform the cloud-edge collaborative method for target detection as described in Embodiment 1.

[0090] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the cloud-edge collaborative method for target detection as described in Embodiment 1.

[0091] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0092] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A cloud-edge collaborative method for target detection, characterized in that, include: The original image is acquired using an edge device, and then divided into multiple sub-images, specifically including: A Gaussian mixture background modeling algorithm is applied to the original image to identify foreground objects in the original image, and a target rectangle is drawn for each target; the foreground object is a moving target. The original image is divided into multiple sub-images; each sub-image is initialized with a sub-image list; the sub-image list is used to store target information. Based on the sub-image, the original image is segmented to generate a slice image, specifically including: The sub-image to which the foreground object belongs is determined based on the bounding box of the target, and then stored in the corresponding sub-image list; Based on the sub-image list, the position of the largest bounding rectangle is obtained according to all target rectangles contained in each sub-image, and the largest bounding rectangle is cut out from the original image to generate a sliced ​​image; Upload all image slices and their corresponding service level targets to the cloud scheduler, and upload the service level targets corresponding to all image slices to the cloud server; A service-free function for a deep learning model of object detection is deployed using the cloud server; the service-free function of the deep learning model of object detection satisfies the service level objectives of all the sliced ​​images. The cloud scheduler uses the number of slices and the GPU memory of the no-service function of the object detection deep learning model to define one or more canvases, and the binning algorithm is used to stitch all the slices into the canvas; The spliced ​​canvas with image slices is input into the no-service function of the object detection deep learning model for batch processing; The spliced ​​canvas with cropped images is input into the service-free function of the object detection deep learning model for batch processing, which also includes the following steps: In the offline phase, a typical spliced ​​canvas with cropped images is input into the no-server function of the object detection deep learning model for batch processing, and the time spent by the no-server function of the object detection deep learning model for each batch size is recorded. During the online phase, whenever the cloud scheduler receives a slice of image sent from the edge device, it saves all the slices to the cache queue as a batch request, recording the resolution information, request start time, and service level target. The binning algorithm is used to stitch the slices in the cache queue into a canvas. If the current canvas cannot meet the binning requirements, a new canvas is created until all the slices in the cache queue can be stitched together. The number of canvases is recorded. The relaxation time is calculated based on the number of canvases and the time taken for each batch of images to be processed by the no-service function of the object detection deep learning model; the relaxation time is the time taken to input all current canvases as a batch into the no-service function of the object detection deep learning model. Calculate the deadline for all requests based on the request start time and the service level target; Calculate the start time of the no-service function of the object detection deep learning model based on the deadline and the relaxation time. At the start of the execution time of the no-service function of the object detection deep learning model, the canvas with stitched images is input into the no-service function of the object detection deep learning model for batch processing.

2. The cloud-edge collaborative method for target detection according to claim 1, characterized in that, The start execution time of the no-service function of the object detection deep learning model for: ； in, The deadline is [date / time]. , To request a start time, For service level objectives; For relaxation time, , For the number of canvases, The time spent by the object detection deep learning model without a service function for processing images of each batch size.

3. The cloud-edge collaborative method for target detection according to claim 1, characterized in that, The spliced ​​canvas with cropped images is input into the service-free function of the object detection deep learning model for batch processing, which also includes the following steps: When the cloud scheduler receives a new batch request, it obtains the current time when the new batch request was received. However, if the current time exceeds the start time of the no-service function of the object detection deep learning model, it immediately inputs the previous batch requests into the no-service function of the object detection deep learning model for batch processing, and clears the cache queue, retaining only the new batch requests.

4. The cloud-edge collaborative method for target detection according to claim 1, characterized in that, The spliced ​​canvas with cropped images is input into the service-free function of the object detection deep learning model for batch processing, which also includes the following steps: When the cloud scheduler receives a new batch request, but the number of canvases is greater than or equal to the maximum number of canvases that the no-server function of the object detection deep learning model can process simultaneously, it immediately inputs the previous batch requests into the no-server function of the object detection deep learning model for batch processing, and clears the cache queue, retaining only the new batch requests.

5. A cloud-edge collaborative device for target detection, characterized in that, The cloud-edge collaborative method for target detection according to any one of claims 1-4 includes: An edge device is used to acquire the original image and divide the original image into multiple sub-images; Based on the sub-image, the original image is segmented to generate slices, all slices are uploaded to the cloud scheduler, and the service level targets corresponding to all slices are uploaded to the cloud server; The cloud server is used to deploy the no-service function of the object detection deep learning model; the no-service function of the object detection deep learning model satisfies the service level objectives of all the sliced ​​images; The cloud scheduler is used to define one or more canvases based on the number of slices and the GPU memory of the no-service function of the object detection deep learning model, and to use a binning algorithm to stitch all the slices into the canvas; The spliced ​​canvas with image slices is input into the no-service function of the object detection deep learning model for batch processing.

6. An electronic device, characterized in that, It includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the cloud-edge collaborative method for target detection as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the cloud-edge collaborative method for target detection as described in any one of claims 1-4.

Images