Image data processing method and system based on cloud edge collaboration

Abstract

The application relates to the technical field of video monitoring and discloses an image data processing method and system based on cloud edge cooperation, which are used for realizing optimized allocation of computing resources and efficient execution of video intelligent analysis tasks. The method comprises the following steps: an edge gateway firstly calls a student model to perform preliminary processing on a target image of a terminal; then a decision function is called to judge whether the preliminary processing result triggers an uploading mechanism facing a cloud server, if yes, the target image is uploaded, and after receiving a secondary processing result returned after the cloud server calls a teacher model, the secondary processing result is output to a user; if no, the preliminary processing result is directly output to the user; wherein the condition for triggering the uploading mechanism is that a predicted value of an importance factor in the preliminary processing result is greater than a set first threshold value, a predicted value of a performance index factor is less than a set second threshold value, and a predicted value of a context correlation factor is greater than a set third threshold value.

Description

Technical Field

[0001] This invention relates to the field of video surveillance technology, and in particular to an image data processing method and system based on cloud-edge collaboration. Background Technology

[0002] The rapid evolution of artificial intelligence technology has brought revolutionary opportunities to the field of video intelligent analysis. Large-scale models can achieve advanced intelligent functions ranging from simple object detection and behavior recognition to complex scene understanding, event prediction, and multimodal information fusion, greatly enhancing the value mining potential of video data. Their powerful generalization ability and ability to learn complex patterns have enabled video analysis to reach new heights in accuracy and depth.

[0003] However, this powerful capability also brings serious challenges. The training and inference process of large models consumes enormous computing resources, memory bandwidth, and power, placing extremely high demands on hardware. This sharply contradicts the stringent requirements of video applications, especially real-time monitoring and smart property quality verification scenarios, which demand low latency and high response speeds.

[0004] On the one hand, if large-scale model inference is entirely reliant on the cloud, all video data needs to be transmitted to a central server. This not only incurs high network bandwidth costs, especially when processing massive amounts of high-definition video streams, but also leads to significant end-to-end latency. For applications requiring real-time decision-making (such as intrusion detection alarms and emergency dangerous behavior warnings), this latency may be unacceptable. Furthermore, data transmission over public networks or insecure environments also poses privacy and security risks.

[0005] On the other hand, if you try to deploy the complete large model directly on resource-constrained edge devices (such as cameras, edge servers, and edge gateways), you will encounter many serious resource bottlenecks.

[0006] Resource limitations: The hardware resources (computing power, memory capacity, storage space) and energy supply (power consumption limitations) of edge devices are typically far lower than those of powerful cloud servers. Faced with hundreds of billions or even trillions of model parameters, edge devices are inadequate, unable to provide sufficient computing power for real-time or near-real-time inference, and unable to accommodate massive model weights and intermediate computational data.

[0007] Performance and stability issues: Models deployed at the edge may overheat due to continuous high load operation, affecting their lifespan and stability; actual inference performance will drop sharply, failing to meet real-time requirements; for battery-powered mobile or portable edge devices, huge power consumption will cause the battery to be depleted quickly, seriously affecting deployment range and usability; in extreme cases, the device may not be able to load or start a large model at all.

[0008] High Edge Computing Costs: Running large models at the edge requires significant hardware upgrades to edge devices. This necessitates procuring professional-grade edge computing equipment equipped with high-performance processors, large-capacity memory, and storage, rather than traditional low-cost cameras or basic gateways. This dramatic increase in hardware costs results in extremely low ROI for small-scale projects or projects in scenarios like property management, where budgets are limited.

[0009] Project Scale and Cost-Effectiveness Imbalance: For small projects or small-scale deployments, the high cost of edge hardware investment is extremely uneconomical. Projects may only need to process a limited number of video streams or meet relatively basic analytics needs, and their value output is far from sufficient to cover the huge hardware investment required at the edge to support large models. This approach not only increases initial construction costs but also raises subsequent maintenance and upgrade costs, making edge deployment solutions economically unattractive.

[0010] Therefore, solutions that rely solely on the cloud or entirely on the edge cannot effectively meet the diverse requirements of modern video intelligence applications regarding performance, latency, cost, and privacy. There is an urgent need for an innovative solution that intelligently integrates the advantages of both the cloud and the edge to optimize the allocation of computing resources and efficiently execute video intelligence analysis tasks. Summary of the Invention

[0011] The purpose of this invention is to disclose an image data processing method and system based on cloud-edge collaboration, so as to achieve optimized allocation of computing resources and efficient execution of video intelligent analysis tasks.

[0012] To achieve the above objectives, the cloud-edge collaborative image data processing method of the present invention includes:

[0013] Step S1: The edge gateway calls the student machine model to perform initial processing on the target image of the terminal;

[0014] Step S2: The edge gateway calls the upload decision function to determine whether the initial processing result triggers the upload mechanism to the cloud server. If yes, the target image is uploaded, and after receiving the secondary processing result obtained and returned by the cloud server after calling the teacher machine model, the secondary processing result is output to the user. If no, the initial processing result is directly output to the user.

[0015] The initial processing results include the predicted values ​​of importance factors, performance index factors, and context relevance factors for each frame of image. The conditions for triggering the upload mechanism are: the predicted value of the importance factor is greater than the set first threshold, the predicted value of the performance index factor is less than the set second threshold, and the predicted value of the context relevance factor is greater than the set third threshold.

[0016] The importance factor represents the degree of criticality of the identified object or event at the business level; the larger the value, the more critical it is.

[0017] The performance index factor is a reliability index of the prediction results of the identified objects or events; the larger the value, the more reliable the prediction.

[0018] The value of the context relevance factor depends on the deviation between the identified object or event and the normal situation defined by the preset business rules; the greater the deviation, the higher the value; the preset business rules are dynamic rules read after the context engine is triggered, and the dynamic rules are differentiated in the knowledge base based on time sequence and scenario.

[0019] Preferably, the present invention further includes:

[0020] Step S3: The cloud server collects and stores the target images uploaded by each edge gateway in batches. When the number of target images stored in the current batch exceeds a set fourth threshold, the student machine model is retrained based on the collected target images to drive the performance of the student machine model to iterate according to a set evolution relationship. The distributed student machine models are then updated consistently according to the iterated version.

[0021] The evolutionary relationship is as follows: ;in, This represents the performance metrics after model iteration. This represents the performance metrics before model iteration. This represents the performance metrics of the teacher-computer interface model. This represents the iterative evolution factor.

[0022] Preferably, the present invention further includes:

[0023] Step S4: The cloud server creates and / or updates the pairing relationship between the target terminal and the edge gateway with the goal of minimizing the sum of the computing time and communication overhead of each execution node.

[0024] Preferably, the present invention further includes:

[0025] The edge gateway obtains a quality check form pre-configured by the user, and establishes a mapping relationship between the corresponding camera terminal and the student machine model based on the quality check form. After the teacher machine model or any of the student machine models identifies the object or event that will generate a work order based on the real-time video stream of the corresponding camera, it automatically tracks, extracts and saves image frames and / or video segments related to the work order execution quality.

[0026] Preferably, in the upload mechanism of the edge gateway to the cloud server, the edge gateway of the present invention determines whether the target image needs assistance from adjacent video segments based on the object or event in the initial processing result. If so, the target image and the preceding and following image frames are edited into video segments of uniform duration and uploaded. If it is determined that the target image does not need assistance from adjacent video segments, only the target image is uploaded.

[0027] Preferably, the present invention further includes:

[0028] Step S5: Any of the edge gateways performs periodic or dynamic management of the student machine models in the cache. When it is determined whether there is cache space to preload at least one potential student machine model in addition to the student machine model required by the current service, if so, the time series of student machine models called in the local history is input into the LSTM model to predict the potential student machine models that can be loaded and the preloading process is performed; the current service is determined by the current monitoring task and the quality inspection form, and the monitoring task includes the inspection task.

[0029] Preferably, the present invention further includes:

[0030] Step S6: Each of the edge gateways periodically calculates the cache hit rate based on the LSTM model in the most recent time period, and determines whether the cache hit rate is lower than the set fifth threshold. If so, it reports the input time series and actual label value of the hit error to the cloud server.

[0031] Step S7: The cloud server collects, stores and counts the number of data entries reported by each edge gateway that have a hit error. When the number of data entries stored in the current batch exceeds the set sixth threshold, the LSTM model is retrained based on the collected batch of data, and the retrained LSTM model is distributed to each edge gateway for consistency update.

[0032] To achieve the above objectives, the present invention also discloses an image data processing system based on cloud-edge collaboration, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0033] The present invention has the following beneficial effects:

[0034] 1. Real-time performance and computing power balance: The uploaded content is filtered by a decision function that makes comprehensive decisions based on importance factors, performance index factors and context relevance factors. While ensuring the real-time performance of video analysis, the capabilities of the cloud-based teacher-computer model are fully utilized, and the one-to-many computing power distribution relationship between the teacher-computer model and the student-computer model is reasonably balanced. This achieves optimized allocation of computing resources and efficient execution of intelligent video analysis tasks.

[0035] 2. Improved resource utilization: Dynamic task allocation and layered model deployment reduce unnecessary cloud transmission and computation, significantly reducing bandwidth and computing power consumption.

[0036] 3. System elastic scalability: Edge gateway nodes can work independently or collaborate with the cloud to form an elastic computing cluster to adapt to application needs of different scales.

[0037] 4. Continuous learning capability: Through the collaborative feedback between the large cloud model and the edge sub-model, the system has the ability to continuously learn and optimize, has strong scalability, can adapt to the ever-changing video analysis needs, and can solve the problem of insufficient computing power in traditional edge computing through intelligent scheduling.

[0038] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0039] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0040] Figure 1 This is a schematic diagram of the image data processing method based on cloud-edge collaboration disclosed in an embodiment of the present invention. Detailed Implementation

[0041] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0042] Example 1

[0043] This embodiment constructs a collaborative architecture of "large cloud model + small edge model" for scenarios such as video surveillance, smart property management, and public security. By using the edge gateway as an intelligent scheduling node, the computing task allocation strategy is dynamically adjusted according to video data characteristics, network conditions, and edge device resources. This ensures the real-time performance of key video analysis while making full use of the powerful capabilities of the large cloud model.

[0044] This embodiment can specifically adopt the following three-layer architecture to achieve the collaborative scheduling of video intelligent large models:

[0045] Cloud layer: Deploys large models with numerous parameters, responsible for model training, global feature extraction, complex inference tasks, and task coordination across edge nodes. It also maintains model and knowledge bases in the cloud, providing model updates and knowledge supplementation services to edge nodes.

[0046] Edge gateway layer: It has data preprocessing capabilities, integrates intelligent scheduling algorithms, and determines task allocation strategies based on real-time conditions. In addition, the edge gateway supports multiple video encoding formats and protocol conversions (such as GB28181, ONVIF, etc.), and has a built-in lightweight AI model library.

[0047] Terminal layer: This layer includes devices such as surveillance cameras and sensors, responsible for raw video capture and execution of edge gateway commands. Terminal devices communicate with the edge gateway through an optimized SDK, supporting efficient data transmission in low-bandwidth environments.

[0048] The performance modeling and cost analysis based on the above architecture are as follows:

[0049] I. Delay Modeling: Assume the overall system response time is... It can be broken down into: ;in, For the time it takes for the terminal camera to capture data; For data transmission time from the terminal to the edge or from the edge to the cloud; For model inference time; This refers to the time it takes for the cloud to return feedback results and trigger an action. Therefore, by introducing a lightweight edge-side model for inference, and It can significantly reduce latency, making it especially suitable for scenarios with high latency requirements.

[0050] II. Bandwidth Compression Modeling: Define the original video stream size as D0, and the amount of data uploaded after edge sub-model filtering as D′, then the compression ratio... for: Experiments show that in object detection tasks, uploading only keyframes or ROI (Region of Interest) can achieve a compression ratio of 1 / 10 to 1 / 30, greatly alleviating cloud pressure. If the compression ratio is 0.1, daily upload traffic is reduced to 10% of its original value, and bandwidth costs decrease by approximately 90%.

[0051] To this end, this embodiment innovatively proposes a collaborative mechanism of "cloud-based professional large model guidance and edge-distilled sub-model execution":

[0052] Distillation Sub-models and Deployment Mechanism: The cloud-based professional large model trains multiple lightweight sub-models (i.e., student machine models) with specific task capabilities based on the teacher-student distillation model, and deploys them to edge gateway devices with different hardware configurations. The sub-models can then perform efficient preliminary processing of video streams locally, such as object detection, action recognition, and preliminary judgment of abnormal behavior.

[0053] Knowledge distillation:

[0054] Let the teacher model output be yT=T(x) and the student model output be yS=S(x), then the loss function is... for: ;in, It is used to control the distillation weights; LCE (CrossEntropy Loss) is a hard-label loss. It minimizes the KL divergence loss.

[0055] By employing model distillation and lightweighting techniques, edge devices can run AI inference tasks without the need for high-performance GPUs, thereby reducing the procurement cost of individual edge nodes.

[0056] In this embodiment, after completing the initial local inference, the edge device equipped with the sub-model determines whether it needs to upload some segments, key frame cropping images, etc. to the cloud for further analysis based on factors such as the importance of the recognition results and the confidence threshold. For abnormal events that are more difficult to process, the upload process is triggered, requesting the cloud professional model to provide higher-precision recognition and decision support.

[0057] Therefore, this embodiment defines a dedicated decision function. Specifically:

[0058] ;

[0059] in, Importance factor, representing the identified objects or events. In terms of criticality at the business level, a higher value indicates greater criticality. The context relevance factor has a value that depends on the deviation between the identified object or event and the normal situation defined by the preset business rules. The greater the deviation, the higher the value. The preset business rules are dynamic rules that are read after the context (such as time context, regional context, etc.) engine is triggered. The dynamic rules are differentiated in the knowledge base based on time sequence and scenario. Indicates the maximum value. This is a performance metric factor used to represent the reliability of prediction results for identified objects or events; that is, a series of prediction results for multiple predicted objects or events. The maximum value in the range indicates that the larger the value, the more reliable it is. These are the set thresholds, referred to as the first, second, and third thresholds for ease of description. Therefore, the conditions for triggering the upload mechanism in this embodiment are: the predicted value of the importance factor is greater than the set first threshold, the predicted value of the performance index factor is less than the set second threshold, and the predicted value of the context relevance factor is greater than the set third threshold.

[0060] The values ​​of the above factors can be, for example: I(fire) = 10.0; I(fight) = 9.0; I(intrusion) = 7.0; I(normal pedestrian) = 0.1; I(cats and dogs) = 0.0.

[0061] For example, an overflowing trash can is a relatively static event. Therefore, detecting whether a trash can is overflowing can be achieved using a single image frame. When the camera captures a single frame of an overflowing trash can, the edge gateway immediately triggers an alarm and dispatches a work order. After the cleaning staff has handled the situation, the execution quality of the work order can be quantified through a video of the handling process. In this process, the overflowing of the trash can is associated with the time (e.g., peak hours of 7-8 pm) and space (trash sorting station) of the business scenario. This embodiment can deploy a set of corresponding student and teacher machine models based on such business rules and establish a mapping relationship between them and the corresponding camera terminals. Similarly, a set of dedicated student and teacher machine models can also be deployed based on another type of business (e.g., elevator maintenance). Among them, different supporting models based on different business rules can be set differently based on the thresholds of the above three factors.

[0062] In summary, the essence of the method in this embodiment can be regarded as Figure 1 The following steps are shown:

[0063] Step S1: The edge gateway calls the student machine model to perform initial processing on the target image of the terminal.

[0064] In this step, the initial processing results include the predicted values ​​of importance factors, performance index factors, and context relevance factors for each frame of the image.

[0065] Step S2: The edge gateway calls the upload decision function to determine whether the initial processing result triggers the upload mechanism to the cloud server. If so, the target image is uploaded, and after receiving the secondary processing result obtained and returned by the cloud server after calling the teacher machine model, the secondary processing result is output to the user. If not, the initial processing result is directly output to the user.

[0066] In this step, the conditions for triggering the upload mechanism are: the predicted value of the importance factor is greater than the set first threshold, the predicted value of the performance index factor is less than the set second threshold, and the predicted value of the context relevance factor is greater than the set third threshold.

[0067] Furthermore, the edge gateway in this embodiment can also obtain a quality inspection form pre-configured by the user, and establish a mapping relationship between the corresponding camera terminal and the student machine model based on the quality inspection form, so that after the teacher machine model or any student machine model identifies the object or event that will generate a work order based on the real-time video stream of the corresponding camera, it can automatically track, extract and save image frames and / or video segments related to the work order execution quality.

[0068] In addition, in the upload mechanism to the cloud server, the edge gateway can determine whether the target image needs to be supported by video segments adjacent to it based on the objects or events in the initial processing results. If so, the target image and the preceding and following image frames are edited into video segments of uniform duration and uploaded. If it is determined that the target image does not need to be supported by adjacent video segments, only the target image is uploaded.

[0069] Based on the above decision function, the upload rate in this embodiment can typically be controlled between 10% and 30%, meaning only a small number of video clips need to be uploaded to the cloud for in-depth analysis. Let the total number of cameras be... The average daily raw data volume of each video stream is Upload rate The total amount of data uploaded is: Cloud resource consumption can be reduced by 70% to 90%, significantly reducing GPU computing costs and storage overhead.

[0070] Example 2

[0071] This embodiment further expands upon Embodiment 1 described above. Specifically, the cloud-based teacher-computer model performs in-depth analysis upon receiving uploaded content and provides decision analysis results, while simultaneously synchronizing these results back to the edge gateway. By continuously collecting uploaded samples and corresponding feedback information from the edge gateway, the cloud supports periodic updates and performance optimization of the student-computer model, thereby continuously improving the recognition accuracy of the edge model and the rationality of the uploaded content selection strategy, forming an adaptive optimization closed-loop system.

[0072] Therefore, the following model update mechanism can be adopted in this embodiment:

[0073] Let the cloud model update frequency be Δt, and the performance improvement of the edge device after each update be η. This process is achieved through the following closed-loop optimization mechanism:

[0074] Model update and consistency guarantee mechanism: All model training and iteration are centrally completed in the cloud. The cloud aggregates differentiated samples uploaded by each edge node according to rules, constructs a global validation set, and trains a general lightweight student model based on this set, fundamentally eliminating the model differentiation problem caused by data differences across multiple nodes. Each edge node deploys the same model with completely identical functions and interchangeability, ensuring standardized system behavior. Model consistency is guaranteed through a strict firmware-based management process: After the new generation model is validated, it is uniformly and batch-deployed to all edge devices through a secure OTA channel.

[0075] Performance optimization quantification relationship: The cloud leverages feedback from its stronger teacher model to drive the edge model performance to progressively approximate the following relationship:

[0076]

[0077] Where P represents the model performance metric (such as mAP, accuracy, etc.). This represents the performance metrics after model iteration. This represents the performance metrics before model iteration. This represents the performance metrics of the teacher-computer interface model. This represents the iterative evolution factor. Therefore, by gradually correcting the edge model bias through feedback from a stronger model in the cloud, a closed-loop optimization system can be formed, thereby reducing the frequency of manual on-site debugging and upgrades, and thus reducing some of the maintenance costs that rely on manpower.

[0078] Example 3

[0079] Based on the two embodiments described above, this embodiment discloses a fusion-compatible dynamic scheduling algorithm based on multi-factor evaluation, which mainly includes the following functional logic:

[0080] Algorithm Polling: By using polling strategies and algorithms, and considering the real-time and efficiency requirements in different scenarios, the polling mechanism checks video streams or image frames, combines machine learning to predict the probability of video events, dynamically adjusts the polling frequency, and applies different AI algorithms for detection, recognition, or analysis.

[0081] Resource status monitoring: Real-time collection of resource usage information for each edge node, including CPU, memory, network bandwidth, and storage, to create a resource heatmap. This data is a key input for evaluating task execution time.

[0082] Task Priority Assessment: Priority weights are assigned to each video analysis task based on factors such as video scene type (e.g., public safety, property quality inspection), task urgency, and data sensitivity.

[0083] Task allocation model:

[0084] Multiple edge nodes are treated as a distributed computing cluster, and dynamic task allocation is performed with the goal of minimizing the overall completion time. The core optimization objective function is:

[0085]

[0086] in, For the task at the execution node The computation time depends on the node's hardware computing power and real-time load (obtained through resource status monitoring). This represents the communication overhead for task data, the value of which depends on network quality (obtained through a network quality awareness module). This model is solved using a shortest path algorithm (such as Dijkstra's algorithm) or reinforcement learning methods to achieve optimal scheduling.

[0087] Furthermore, existing technologies can be used to monitor network latency, packet loss rate, and bandwidth availability between the cloud and the edge, with the data directly serving as communication overhead. The calculation is based on the scheduler's dynamic adjustment of the task allocation strategy. When network conditions are poor, tasks are preferentially allocated to edge nodes or nodes with good network quality to minimize communication overhead.

[0088] Ideally, while meeting performance requirements, energy consumption should be considered as another optimization constraint, prioritizing low-energy computing schemes, which is particularly suitable for mobile edge devices. This requires a trade-off between the total time consumption and energy consumption objectives.

[0089] Based on the above, the cloud server in this embodiment can at least create and / or update the pairing relationship between the target terminal and the edge gateway with the goal of minimizing the sum of the computing time and communication overhead of each execution node.

[0090] Example 4

[0091] Based on the three embodiments described above, the edge gateway can further integrate any one or more of the following technologies:

[0092] (1) Intelligent instruction filtering: Analyze the frequency and mode of control instructions, distinguish between high-frequency and low-frequency instructions, reduce unnecessary equipment operation, and improve response efficiency.

[0093] (2) Video Data Stream Management: The edge gateway achieves intranet penetration through the P2P protocol, and the cloud can remotely configure and manage the edge gateway; local users (within the same local area network) watch videos through the local area network, without consuming Internet bandwidth resources. Plan a video concurrency monitoring mechanism to monitor video concurrency and playback status by packet loss rate and pre-setting the bandwidth of a single project, and set up response mechanisms.

[0094] (3) Data security processing: The video data is encrypted and anonymized at the edge to protect privacy data.

[0095] (4) Distributed computing framework: Dynamically allocate computing tasks among multiple edge gateways to form an edge computing cluster and improve overall processing capabilities.

[0096] (5) Based on historical task patterns, preload potentially needed AI models to edge nodes to reduce real-time loading latency. Based on historical task patterns, use LSTM (Long Short-Term Memory) to predict future (…). Possible models to use The model can typically take the form of: .

[0097] The motivation for the model preloading and intelligent prediction mechanism in this embodiment is as follows: To address the problem of real-time model loading latency, a time series prediction algorithm based on long short-term memory (LSTM) network is adopted. Using the time series of historical model call records as input, it learns the call patterns and periodic modes of different AI task models (such as face recognition, vehicle detection, smoke recognition, etc.) in specific scenarios. With its gating mechanism, the LSTM network effectively captures the long-term dependencies in the call sequence and predicts the most likely models to be called and their probabilities within the future time window T. According to the prediction results, the top K models with the highest probabilities (K is the cache capacity) are intelligently preloaded from the cloud to the edge gateway cache.

[0098] Preloading the prediction model into the edge node cache in advance can reduce the real-time download latency.

[0099] The cache hit rate model can be used to evaluate the effectiveness of the prediction preloading strategy. Let: M be the total number of models in the system's cloud model library (Total number of models), which defines the entire optional range of the cache; K be the maximum capacity of the local cache of the edge gateway (Cache capacity), that is, the number of models that can be stored simultaneously (K < M), which is a hardware limitation on the edge side; count_hit be the number of times the model called by the AI inference request is exactly in the local cache within a certain period of time (Cache hit count); total_request_count be the total number of AI inference requests within the same period of time (Totalrequest count). Then the formula for calculating the cache hit rate H is: .

[0100] Among them, although M and K do not directly appear in the numerator and denominator of the calculation formula, they are the core premises that define the challenge of this scenario and the value of the algorithm. Since K < M, it is impossible to cache all models locally, so intelligent selection must be made. The goal of the LSTM prediction algorithm is to accurately select the most likely K models to be accessed from the model library with a total of M models under the given cache capacity K and put them into the cache, so as to maximize the final hit rate H.

[0101] Based on the inference, under the LSTM prediction model, the hit rate can reach over 90%, significantly improving the edge response speed. The core innovation of this mechanism is to use LSTM prediction to advance the loading latency to idle periods. Based on the strong learning ability of the LSTM model for time series patterns and by analogy with the performance improvement of the intelligent cache prefetching scheme in the industry (under the condition that K is much smaller than M, usually improving the hit rate by 20% - 50% compared with the traditional LRU algorithm), under the cache capacity K, this embodiment can be expected to increase and stabilize the cache hit rate H above 90%, significantly reducing the real-time latency and improving the edge response speed.

[0102] Based on the above, the method of this embodiment may further include the following steps:

[0103] Step S5: Any edge gateway performs periodic or dynamic management of the student machine model in the cache. When it is determined whether there is cache space to preload at least one potential student machine model in addition to the student machine model required by the current business, if so, the time series of student machine models called in the local history is input into the LSTM model to predict the potential student machine models that can be loaded and the preloading process is performed. The current business is determined by the current monitoring task and quality inspection form. The monitoring task includes inspection task.

[0104] Step S6: Any edge gateway periodically calculates the cache hit rate based on the LSTM model within the most recent time period, and determines whether the cache hit rate is lower than the set fifth threshold. If so, it reports the input time series and actual label value of the false hit to the cloud server.

[0105] Step S7: The cloud server collects, stores and counts the number of data entries reported by each edge gateway that have mis-hit the data in batches. When the number of data entries stored in the current batch exceeds the set sixth threshold, the LSTM model is retrained based on the collected batch of data, and the retrained LSTM model is distributed to each edge gateway for consistency updates.

[0106] Example 5

[0107] This embodiment discloses an image data processing system based on cloud-edge collaboration, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method.

[0108] In summary, the methods and systems disclosed in the above embodiments of the present invention have at least the following beneficial effects:

[0109] 1. Real-time performance and computing power balance: The uploaded content is filtered by a decision function that makes comprehensive decisions based on importance factors, performance index factors and context relevance factors. While ensuring the real-time performance of video analysis, the capabilities of the cloud-based teacher-computer model are fully utilized, and the one-to-many computing power distribution relationship between the teacher-computer model and the student-computer model is reasonably balanced. This achieves optimized allocation of computing resources and efficient execution of intelligent video analysis tasks.

[0110] 2. Improved resource utilization: Dynamic task allocation and layered model deployment reduce unnecessary cloud transmission and computation, significantly reducing bandwidth and computing power consumption.

[0111] 3. System elastic scalability: Edge gateway nodes can work independently or collaborate with the cloud to form an elastic computing cluster to adapt to application needs of different scales.

[0112] 4. Continuous learning capability: Through the collaborative feedback between the large cloud model and the edge sub-model, the system has the ability to continuously learn and optimize, has strong scalability, can adapt to the ever-changing video analysis needs, and can solve the problem of insufficient computing power in traditional edge computing through intelligent scheduling.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cloud-edge collaboration based image data processing method, characterized in that, Comprise: Step S1, the edge gateway calls the student machine model to perform preliminary processing on the target image of the terminal; Step S2, the edge gateway calls the upload decision function to judge whether the preliminary processing result triggers the upload mechanism facing the cloud server, if yes, the target image is uploaded, and after receiving the secondary processing result returned by the teacher machine model called by the cloud server, the secondary processing result is output to the user; If no, the preliminary processing result is directly output to the user; Wherein, the preliminary processing result includes the predicted value of the importance factor of each frame image, the predicted value of the performance index factor and the predicted value of the context correlation factor; the condition for triggering the upload mechanism is that the predicted value of the importance factor is greater than the set first threshold value, the predicted value of the performance index factor is less than the set second threshold value and the predicted value of the context correlation factor is greater than the set third threshold value; The importance factor is the key degree of the recognized object or event in the business level, and the greater the value represents the more critical; The performance index factor is the prediction result reliability index of the recognized object or event, and the greater the value represents the more reliable; The value of the context correlation factor depends on the deviation of the recognized object or event from the normal situation defined by the preset business rule; the greater the deviation, the higher the value; the preset business rule is the dynamic rule read by the context engine after being triggered, and the dynamic rule is differentiated classified based on time sequence and scene in the knowledge base. 2.The cloud-edge collaboration based image data processing method of claim 1, wherein, Also include: Step S3, the cloud server collects and stores each target image uploaded by the edge gateway in batches, and when the number of stored target images in the current batch exceeds the set fourth threshold value, re-trains the student machine model based on the collected target images in the batch to drive the student machine model performance to iterate according to the set evolution relationship, and updates the consistency of each student machine model distributedly deployed according to the version after iteration: Wherein, the evolution relationship is: ; wherein, denotes the performance index of the model after iteration, denotes the performance index of the model before iteration, denotes the performance index of the teacher machine model, denotes the iteration evolution factor. 3.The cloud-edge collaboration based image data processing method of claim 1, wherein, Also include: Step S4, the cloud server newly establishes and / or updates the pairing relationship between the target terminal and the edge gateway with the sum of the calculation time and the communication overhead of each execution node as the target. 4.The cloud-edge collaboration based image data processing method according to any one of claims 1 to 3, characterized in that, Also include: The edge gateway acquires the quality check form pre-configured by the user, establishes the mapping relationship between the corresponding camera terminal and the student machine model according to the quality check form, so that after the teacher machine model or any student machine model recognizes the object or event that will generate a work order based on the real-time video stream of the corresponding camera, the image frames and / or video segments related to the execution quality of the work order are automatically tracked, extracted and saved.

5. The cloud-edge collaboration based image data processing method according to claim 4, characterized in that, In the upload mechanism facing the cloud server, the edge gateway judges whether the adjacent video segment of the target image is needed according to the object or event in the preliminary processing result, if yes, the target image and the adjacent image frames are cut into video segments according to the unified time length and uploaded; if it is judged that the adjacent video segment of the target image is not needed, only the target image is uploaded. 6.The cloud-edge collaboration based image data processing method of claim 4, wherein, Also include: Step S5, any of the edge gateways periodically or dynamically manages the student machine model in the cache. When it is determined that, in addition to the student machine model required by the current service, there is still cache space to pre-load at least one potential student machine model, if so, the time sequence of the student machine model called in the local history record is input into the LSTM model to predict the potential student machine model that can be loaded and perform pre-loading processing; the current service is determined by the current monitoring task and the quality check form, and the monitoring task includes an inspection task. 7.The cloud edge collaboration based image data processing method of claim 6, wherein, Also includes: Step S6, any of the edge gateways periodically counts the cache hit rate based on the LSTM model in the recent time period, and determines whether the cache hit rate is lower than the fifth threshold value, if so, reports the hit error input time sequence and the actual label value to the cloud server; Step S7, the cloud server collects, stores and counts the number of data reported by each edge gateway, and when the number of data stored in the current batch exceeds the sixth threshold value, the LSTM model is retrained based on the collected data in the batch, and the retrained LSTM model is distributed to each edge gateway for consistent update. 8.A cloud-edge collaboration based image data processing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, The processor executes the computer program to realize the method of any one of claims 1-7.

Images