A machine vision model training method and system based on edge computing power

Abstract

This invention discloses a machine vision model training method and system based on edge computing power, belonging to the fields of machine learning, edge computing, and computer vision technology. It involves acquiring machine vision images and constructing a pre-labeled model for automatic pre-labeling; then manually correcting some pre-labels to optimize the pre-labeled model and obtain corrected labeled images; constructing a machine vision model and training it based on the labeled images; allocating training task variables according to hardware perception to dynamically select the training task execution location (edge ​​or cloud); deploying the trained machine vision model at the edge for image recognition inference; and then manually sampling the model's inference results to evaluate the accuracy of the machine vision model. If the accuracy is low, a new round of model training is initiated, and the evaluated machine vision model is used as the pre-labeled model.

Description

Technical Field

[0001] This invention belongs to the fields of machine learning, edge computing and computer vision technology, and specifically relates to a machine vision model training method and system based on edge computing power. Background Technology

[0002] With the popularization of the Internet of Things and smart terminals, the demand for machine vision models has surged in fields such as smart IoT, multimodal perception, content acquisition and editing, low-altitude economy, industrial inspection, and autonomous driving. This has also led to the emergence of edge and cloud collaboration for machine vision model training. Terminal devices or servers with independent data acquisition, processing, and computing capabilities located close to the data source are used as edge devices. Local computing resources (such as CPUs, GPUs, and NPUs) deployed on the edge devices (such as AI computing boards and smart terminals) support real-time data processing and constitute edge computing power. Through the collaborative work of cloud servers and edge devices, data sharing and resource optimization are achieved, supporting distributed training and unified management.

[0003] However, current technologies still rely on the cloud for data annotation and model training. Uploading data to the cloud is time-consuming and cannot meet real-time requirements (such as drone detection). Furthermore, high-definition image / video transmission consumes significant network resources, creating bandwidth bottlenecks and incurring high costs. Uploading sensitive data (such as medical images) to the cloud may also violate privacy regulations, posing a risk of privacy breaches. On the other hand, existing edge computing is still limited to model inference. Model training cannot be performed based on the computing power of edge devices and still requires cloud computing servers. This prevents full utilization of edge computing power and hinders autonomous learning and dynamic model updates based on edge computing power, lacking a closed-loop capability for the entire edge training and inference process. In addition, training data annotation still heavily relies on manual annotation, reducing efficiency. The rigid edge-cloud resource scheduling also prevents dynamic optimization of latency and bandwidth consumption. Summary of the Invention

[0004] To address the shortcomings of existing technologies and achieve a fully automated closed-loop process from data acquisition to model iteration at the edge, thereby reducing reliance on the cloud, this invention adopts the following technical solution:

[0005] A machine vision model training method based on edge computing power includes the following steps:

[0006] Acquire machine vision images;

[0007] A pre-annotation model is constructed to automatically pre-annotate the image. Then, some pre-annotations are manually corrected to optimize the pre-annotation model and obtain the corrected annotated image.

[0008] The machine vision model is constructed and trained on it based on labeled images. Training task variables can be allocated according to hardware perception, and the execution location of the training task (edge ​​or cloud) can be dynamically selected. The trained machine vision model is then deployed at the edge to perform the inference process of image recognition.

[0009] The inference results are manually sampled to evaluate the accuracy of the machine vision model. When the accuracy is low, a new round of model training is started, and the evaluated machine vision model is used as the pre-labeled model.

[0010] Furthermore, for manually corrected images, the pre-labeling model finds other similar images and assigns correction weights to these other images based on their similarity, so that the recognition results of these other images shift towards the manually corrected results. Thus, a small amount of manual correction can have a wider impact on the pre-labeling results.

[0011] Furthermore, if a manually selected image is found to be incorrectly identified, and this image is one of the other images after the correction weight adjustment, it indicates that the similarity-based association adjustment has changed the category of the other image to the incorrect category, or that a correction error occurred during the manual correction process. In this case, the manually corrected image is reviewed. If the manual correction is correct, the correction weight is adjusted in reverse so that the recognition result of the other image returns to the result before the manual correction.

[0012] Furthermore, the manually corrected and / or manually sampled images are those labeled with low confidence.

[0013] Furthermore, by constructing an edge training efficiency model that integrates edge training time with edge computing power, data volume, and machine vision model complexity, we quantify the reduced dependence of edge training on the cloud and demonstrate the efficiency advantages of localized edge processing. In the edge training efficiency model, the product of the number of tasks and the computational cost per iteration at the edge is divided by the edge computing power, and then the product of the data transmission coefficient and the data volume is added. When the edge trains alone, the data transmission coefficient is zero; when the edge and the cloud collaborate on training, the data transmission coefficient is greater than zero.

[0014] Based on gradient descent, we perform edge-end closed-loop iterative convergence analysis on machine vision model updates. By measuring the descent rate of the loss function of the machine vision model, we prove the effectiveness of edge-end closed-loop training and ensure that the model converges quickly during iteration. The descent rate of the loss function is the product of the square of the norm of the loss function gradient divided by the loss value of the iteration and the learning rate.

[0015] Furthermore, by minimizing the total latency using a mixed-integer programming model, tasks are allocated between the edge and the cloud, thereby optimizing the scheduling of edge-cloud resources. Minimizing the total latency is achieved by adding the computation time of the tasks allocated to the edge and the cloud to the data transmission time between the edge and the cloud.

[0016] Using a mixed-integer programming model, we define the task assignment variable x. i ∈{0,1}, dynamically select the task execution location (edge ​​or cloud); the system can dynamically adjust the data upload strategy of the task according to bandwidth usage and transmission time to avoid bandwidth bottlenecks.

[0017] Furthermore, the current load of the edge terminal is obtained by allocating the current computing tasks at the edge terminal. Based on the ratio of the current load to the total computing capacity of the edge terminal, it is determined whether the edge terminal exceeds its computing capacity. If it does, some computing tasks are migrated to the cloud.

[0018] During the task execution process, the load on the edge may change, resulting in excessive execution latency. In this case, a decision can be made on whether to migrate the task based on the real-time feedback of the task to ensure the real-time performance of the task and the load balance of the system. If the computation time of the task on the edge is greater than the latency threshold of the task on the edge, and / or the current load on the edge is greater than the load threshold of the edge, then some computation tasks will be migrated to the cloud.

[0019] Furthermore, the edge device performs feature analysis on the scene of the acquired machine vision images, establishes and updates a scene feature parameter library to perform data augmentation on the edge device training data, and realizes scene perception and adaptive evolutionary updates. The complexity of data augmentation is dynamically adjusted according to the capacity of the machine vision model mounted on the edge device and the edge device information. When the device has sufficient available time, the performance indicators of the computer vision model, the information of the edge device, and the historical data of data augmentation operations are used as the state, and the selection and adjustment of data augmentation operations are used as actions. Based on the lightweight reinforcement learning update strategy of model feedback, the device learns to improve the performance of the edge device, thereby balancing the data augmentation effect and the consumption of edge computing resources.

[0020] Furthermore, a hardware-aware development and training model is established, closely integrating the training process of the machine vision model with the characteristics of the target hardware. This allows the trained model to perform better on specific hardware, incorporating hardware constraints into the training objectives of the machine vision model. This not only optimizes the model's accuracy but also its inference performance on specific hardware. Feature quantification is performed on model performance metrics and information from edge devices. Network models with different structures and parameter levels are developed and designed for different edge devices. A model latency prediction dataset is established, and a latency predictor is constructed and trained to predict inference latency based on the latency prediction dataset. A loss function is constructed, fusing the machine vision model's own object detection loss and inference latency loss. Iterative training and optimization evaluation are performed on specific edge target hardware platforms. The latency of the model on the hardware platform is coupled for overall training iteration, optimization evaluation, and policy feedback parameters are updated to the model. Finally, the optimal subset of models is retained, thereby controlling the trade-off between accuracy and latency. This allows the model's performance on the target hardware to be considered during the training phase, and the model's speed to be optimized during the model development and training phases.

[0021] Traditional development models mostly involve designing structures and conducting experimental analysis on high-performance computing devices. Even the same model will produce different latency when migrated to different hardware devices. IoT devices with limited computing power need to make the most of their limited computing power. Models with too many parameters and overly complex structures may not run properly, while models with too few parameters may not run properly. Conversely, existing computing resources are not being fully utilized. Hardware latency is a key factor in model design and development, mainly to avoid traditional development models. Constructing inference latency loss through hardware latency and using it as part of the loss function to guide the design of a reasonable and effective network structure is currently the optimal solution.

[0022] A machine vision model training system based on edge computing power includes an edge device and a cloud device. According to the aforementioned machine vision model training method based on edge computing power, data is collected at the edge device, and a machine vision model is trained based on the collected images for image recognition.

[0023] The advantages and beneficial effects of this invention are as follows:

[0024] This invention utilizes key technologies such as hardware-aware training, cloud-edge dynamic collaborative architecture, hybrid annotation strategies, and heterogeneous hardware adaptation to achieve a closed-loop process at the edge, encompassing data acquisition, annotation, training, deployment, and model iteration. This reduces reliance on the cloud, supports integrated edge training and inference, and significantly improves personalized training speed, real-time performance, and privacy security. It also supports elastic edge-cloud collaboration, with the cloud providing elastic computing power and complex task processing capabilities, while the edge ensures real-time performance and privacy. The two are seamlessly integrated through dynamic scheduling and federated learning, addressing the shortcomings of traditional solutions in terms of efficiency, privacy, and flexibility. This provides an efficient, secure, and scalable machine vision solution for high-real-time scenarios such as the Internet of Things, multimodal perception, content acquisition and editing, low-altitude economy, industrial inspection, and smart cities. Attached Figure Description

[0025] Figure 1 This is an architecture diagram of the system in an embodiment of the present invention.

[0026] Figure 2 This is a flowchart of the method in an embodiment of the present invention.

[0027] Figure 3 This is a closed-loop flowchart of edge data in one embodiment of the present invention. Detailed Implementation

[0028] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0029] like Figure 1 As shown, a machine vision model training system based on edge computing power includes cloud and edge terminals. It achieves efficient interaction between data and tasks through standardized interfaces. The overall architecture adopts a layered design. The edge terminal mainly includes a data acquisition layer, a computing layer, and an application layer. The data acquisition layer integrates devices such as cameras and sensors, supports real-time data capture and local caching, and ensures that high-value data (such as scene content editing segments and abnormal event video segments) are processed first. The computing layer deploys AI computing boards, such as RK3588, NVIDIA Jetson AGX Xavier, or Huawei Ascend Atlas 300, to provide local computing power support. The application layer covers modules such as data annotation, model training, model deployment, monitoring and evaluation, and cloud synchronization, forming a closed-loop iterative process from data to model.

[0030] The data acquisition layer captures raw data in real time through devices such as cameras and sensors. Cameras support H.265 encoding; sensor data is transmitted to the edge computing layer via the MQTT protocol, ensuring low latency. The acquisition module has a built-in priority queue, prioritizing abnormal event data to ensure real-time response. The data acquired by the acquisition layer flows in two directions: one for annotation, which can be used to train the model; and the other for model inference, where the results are used in specific business applications, such as contextual content acquisition, pest and disease identification, and pollution detection.

[0031] The computing layer provides local computing power based on heterogeneous hardware platforms and provides computing power through AI computing boards. Its functions include two parts: one is as training data for the model, and the other is for model inference after the model is deployed.

[0032] AI computing boards include Rockchip boards, Ascend boards, and NVIDIA boards. Among them, the Rockchip board (RK3588) is suitable for low-power scenarios (such as smart cameras), supports INT8 quantization inference, and has a computing power of 6 TOPS; the Ascend board (Atlas300) is designed for Huawei ecosystem devices, optimizes models through AscendCL, and has an FP16 computing power of 16 TFLOPS; the NVIDIA board (JetsonAGX Xavier) supports TensorRT acceleration, has an INT8 computing power of 32 TOPS, and is suitable for high-throughput tasks (such as real-time video analysis).

[0033] The data annotation module combines AI pre-annotation with manual correction, including a pre-annotation engine and a manual correction interface. The pre-annotation engine calls on lightweight edge models and historical version models to generate preliminary annotations with high accuracy. The manual correction interface provides web-based tools that support annotation box correction, category filtering, and confidence filtering. Corrected data is fed back to the pre-annotation model to achieve self-optimization and iteration. The data for the data annotation module comes from the data acquisition module.

[0034] By coordinating the model training module, model deployment module, and monitoring and evaluation module, model training and deployment are carried out to achieve a closed loop of edge training and inference.

[0035] The model training module trains visual models on computing boards, including training various types of computer vision models such as image classification, object detection, instance segmentation, and semantic segmentation. Examples include identifying crop diseases and pests in agricultural scenarios, identifying restaurant fumes and construction dust pollution in environmental scenarios, identifying production and maintenance activities in content generation scenarios, and identifying vehicle black smoke in traffic scenarios. The model training phase requires computing power, provided by different types of AI boards. Since different AI boards rely on different toolkits, adaptation is necessary for different types of computing boards. The model training phase also requires labeled data from the data annotation module. The quality, quantity, and representativeness of the labeled data determine the quality of the trained model. To adapt to AI hardware from different manufacturers, the trained model needs to be converted to a version suitable for different manufacturers. This is achieved by automatically detecting the type of hardware on the board and converting the model accordingly. Furthermore, because different hardware has varying performance, multiple models need to undergo quantization to achieve a balance between model recognition performance and detection rate.

[0036] The model deployment module and monitoring and evaluation module deploy the converted model to edge devices. The deployed model's inference performance (such as FPS and memory usage) is monitored in real time. If the accuracy drops below a certain threshold, the incremental training process is automatically triggered.

[0037] The cloud synchronization module is used for bidirectional interaction between data and models. Through data upload, it uploads encrypted sensitive data (such as facial images) from the edge to the cloud data warehouse via the TLS protocol for federated learning or global model optimization. Through model distribution, it periodically pushes optimized models from the cloud model warehouse to the edge, such as silently updating at night during low bandwidth periods to reduce the impact on business.

[0038] Error handling and disaster recovery mechanisms are employed, including data loss disaster recovery, training interruption recovery, and network fluctuation response. Data loss disaster recovery utilizes a RAID 1 storage architecture at the edge, with critical data backed up in real time to both local and cloud storage. Training interruption recovery uses a checkpoint mechanism to save training progress, allowing recovery from the most recent state after an interruption, with an accuracy loss of less than 1% (for model development and training, two weights, best and last, are retained. Training continues from the breakpoint starting from the last weight. If the model training process is relatively stable, the accuracy loss can be less than 1%. If best weight is used as the starting point, there is no accuracy loss). Network fluctuation response uses the MQTT protocol to support breakpoint resumption, and unfinished tasks are automatically synchronized after network recovery.

[0039] The cloud architecture includes a resource pool and a scheduling center. The resource pool provides large model annotation services (such as complex scene description generation based on GPT-4V) and distributed training clusters, supporting petabyte-level data storage and multi-device collaborative training. The scheduling center dynamically allocates tasks to the edge or the cloud based on a mixed integer programming algorithm. For example, large-scale annotation tasks are split to the cloud GPU cluster, while training tasks with high real-time requirements are kept at the edge. The communication layer uses the MQTT protocol to achieve efficient communication between the edge and the cloud. Data is encrypted and transmitted (AES-256) to ensure security, while compression technology reduces bandwidth usage by 40%.

[0040] The scheduling center is the central module in the cloud, responsible for global task allocation and resource coordination. Based on a mixed-integer programming algorithm, it dynamically determines whether tasks (such as large-scale training or complex annotation) should be allocated to the cloud resource pool or edge devices. For example, when insufficient computing power is detected at the edge (such as Rockchip board load exceeding 80%), the scheduling center automatically migrates some training tasks to the model training cluster in the cloud (such as AWS p4d instances), ensuring that the task response time is less than 200ms.

[0041] The resource pool integrates high-performance computing and storage resources in the cloud, including sub-modules such as labeled large models, model training clusters, data warehouses, and model warehouses;

[0042] It can annotate large models, calling models with hundreds of billions of parameters such as GPT-4V and SAM (Segment Anything Model) to handle complex annotation tasks, with an annotation accuracy of over 90%.

[0043] Model training clusters are deployed using distributed GPU clusters to support parallel training of large-scale models.

[0044] Data warehouse: Data set storage and management;

[0045] The model repository stores optimized model versions and provides format conversion services; for example, converting PyTorch models to ONNX format to adapt to different edge hardware (such as Ascend boards which need to be converted to OM format); the repository ensures model integrity through hash verification and records version iteration logs for easy rollback and auditing.

[0046] like Figure 2 As shown, a machine vision model training method based on edge computing power includes the following steps:

[0047] Step 1: The edge device collects data through cameras, sensors and other acquisition devices to obtain multimodal data such as images and sounds of various types of agricultural crop growth and environmental pollution.

[0048] Step 2: The collected data needs to be pre-labeled. AI pre-labeling can be done in the following ways:

[0049] 1) If an older version of the AI ​​pre-labeled model has been deployed at the edge, then the older version of the model will be used for pre-labeling;

[0050] 2) If the edge only has a pre-trained model, then pre-labeling is performed using the pre-trained model, for example: using a YOLO pre-trained model for pre-labeling;

[0051] 3) In cloud-edge collaboration scenarios, data can be uploaded to the cloud and pre-labeled by models deployed in the cloud, including historical versions of models, pre-trained models, and even large labeled models.

[0052] Step 3: Since pre-labeled data cannot be completely trusted, manual revision is necessary. Humans review and revise the pre-labeled data using a data annotation platform, storing the revised data in the labeled dataset. This dataset can then be used for subsequent model training. In the cloud-edge collaborative model, both the manually revised and pre-labeled data are stored in a data warehouse. There are two methods for manual revision:

[0053] 1) Manual revisions are performed at the edge, which provides a web-based data annotation tool. Data annotators can use this tool to revise and annotate the data. The revised data is stored directly at the edge. If a cloud is available, the revised data can be synchronized to the cloud for storage.

[0054] 2) In the cloud-based collaborative mode, manual revisions are performed in the cloud. The edge device will synchronize the pre-annotated data to the cloud. The cloud also provides web data annotation tools, which are used by annotators to revise and annotate. The cloud can also submit data that needs manual revision and annotation to a third-party cloud annotation platform for annotation, provided that the user's permission is obtained.

[0055] By combining AI pre-annotation with human collaborative correction, annotation costs can be significantly reduced and data quality improved. The quality of pre-annotation is evaluated by the final accuracy rate, A. final as follows:

[0056]

[0057] Among them, A preβ represents the AI ​​pre-annotation accuracy, and β represents the human correction efficiency coefficient, used to reflect the proportion of errors corrected by humans. By establishing an AI pre-annotation strategy, pre-trained models (such as YOLOv5), zero-shot detection models (such as CLIP), or historical versions of models are called to generate preliminary annotation results. Combined with a human correction mechanism and a visual annotation tool, it supports the highlighting of errors, one-click correction, and batch review functions. Ultimately, it aims to reduce annotation costs (compared to all-manual annotation), reduce the annotation error rate, and shorten the annotation cycle.

[0058] Pre-labeling uses AI models (such as YOLOv5, CLIP, and historical versions of models) to perform initial annotations on the raw data. Based on pre-trained knowledge, the AI ​​model automatically generates bounding boxes, labels, confidence scores, and other information. This process significantly improves annotation speed, especially for data with known categories.

[0059] Human correction involves reviewing the AI ​​pre-annotation results by human annotators and correcting erroneous or low-confidence annotations. Human correction primarily focuses on areas with high model pre-annotation error rates (such as images with unclear object recognition or ambiguous data categories). The corrected data is then returned to the pre-annotated model for further optimization.

[0060] By updating the dataset and storing the corrected data in the annotation database for subsequent training, the system achieves dynamic collaboration between automatic annotation and manual correction, continuously optimizing annotation quality.

[0061] AI pre-labeling models can optimize themselves based on manually corrected data, gradually reducing labeling errors. The quality of manual correction directly affects the improvement of the pre-labeling model, enabling the AI ​​model to continuously improve its accuracy.

[0062] In one embodiment, the AI ​​pre-labeling model acquires manually corrected categories and their corresponding data features. It then compares these data features with other data features based on their similarity, establishing associations between them. Weights are assigned to other data features related to the manually corrected features to appropriately adjust their categories. For example, manually corrected data feature 1 has category A with probability p, and similar data feature 2 has category B with probability q. Weight w1 is calculated based on the similarity s between data feature 1 and data feature 2. If weight w1*q > p, the category of data feature 2 is automatically adjusted from B to A. This allows for adaptive category adjustment by influencing the categories of related data features through a single manual correction.

[0063] Furthermore, in step 6, the model will undergo manual sampling and verification. If the category of data feature 2 is found to be A, meaning the association adjustment based on manual correction is effective, no further processing is required. Alternatively, a positive verification adjustment may be performed, such as adjusting the similarity calculation to obtain a higher weight w2, and / or increasing the bias b so that w2*q+b is greater than p. If the category of data feature 2 is found to be B, meaning the association adjustment has instead changed the AI-pre-labeled correct category to an incorrect category, a reverse verification adjustment may be performed, such as adjusting the similarity calculation to obtain a lower weight w3, and / or decreasing the positive bias b so that w2*q+b < p. Additionally, the categories manually verified based on the sampled data features can also be used for association adjustment.

[0064] The above-mentioned verification adjustment is effective for all associated data features involved in the association algorithm, and the true category obtained through verification can also be directly used for the model parameters of the AI ​​pre-labeled model. Starting from classification and recognition, what is being adjusted is the category probability of data feature recognition itself.

[0065] For data feature 2 that was found to have an incorrect category adjustment due to association adjustment during the review process, data feature 1 was first found through reverse association, and its category was manually reviewed to rule out errors in the manual correction itself. If the manual correction was found to be correct, the above review and adjustment were then carried out.

[0066] Through the feedback mechanism, the AI ​​model can continuously optimize based on manually corrected data, improving annotation efficiency and accuracy. On the other hand, manual correction reduces the amount of manual labor by only correcting samples that are difficult for the AI ​​pre-annotated model to label or have low confidence.

[0067] Step 4: Train the model based on the labeled dataset. Model training can take several forms:

[0068] 1) Dataset and model training at the edge: The model training task is directly scheduled, reads the local dataset, and then starts training. Finally, a standard model parameter file is generated, resulting in the trained model. In this mode, all data management and training are completed at the edge. It is mainly suitable for scenarios where the model only needs to be optimized for data from a specific device and scenario, and there is no need to aggregate data from other nodes for training.

[0069] The edge-side closed-loop training system requires the construction of an edge-side training efficiency model and the analysis of its closed-loop iterative convergence. The edge-side training efficiency model quantifies the reduced dependence on the cloud by constructing an edge-side training time as a function of edge device computing power, data volume, and model complexity, reflecting the efficiency advantages of localized processing. The function of edge-side training time is as follows:

[0070]

[0071] Among them, T train Indicates the edge training time, N represents the number of model iterations, and C represents the training time. model F represents the computational cost per iteration of the model (unit: FLOPs). edge D represents the computing power of edge devices (unit: FLOPs / s). data The dataset size is represented in GB, and α represents the data transfer coefficient (α=0 for local processing and α>0 for cloud collaboration).

[0072] The closed-loop iterative convergence analysis is based on gradient descent-based model updates. The loss function descent rate is defined to demonstrate the effectiveness of closed-loop training on the edge side and ensure rapid model iteration convergence. The loss function descent rate is defined as follows:

[0073]

[0074] Where η represents the rate of decrease of the loss function, L t This represents the loss value in the t-th iteration. Indicates the learning rate. This represents the gradient of the loss function.

[0075] Localized data management is implemented at the edge, with pre-defined rules (such as time windows and event triggers) for collecting raw data. Local caching and priority scheduling are supported to ensure that high-value data is processed first. Data preprocessing (such as denoising and normalization) is completed at the edge, avoiding the uploading of raw data to the cloud. A closed-loop training system is implemented on the edge device. After model training and optimization are completed, the performance monitoring module provides real-time feedback on inference results (such as accuracy and latency), driving dynamic adjustments to model parameters. The closed-loop iteration cycle is shortened from the traditional "weekly" level to the "dayly" level, adapting to rapidly changing business needs. Ultimately, this reduces training latency, protects data privacy, and improves model iteration efficiency.

[0076] Adaptive data augmentation for edge-side scene awareness. Edge devices are often deployed in specific scenarios and may face the problem of insufficient training data. Therefore, it is necessary to build lightweight data augmentation models at the edge. The core of lightweight data augmentation models lies in balancing the effect of data augmentation with the consumption of computing resources at the edge.

[0077] The edge continuously collects, analyzes, and stores data. Through the feature analysis module, sampling and calculations are performed according to a specified strategy to update the scene feature parameter library. For example, histogram distribution is obtained through color features, gray-level co-occurrence matrix through texture features, and discrete location distribution through spatial features. The feature parameter library is then constructed and updated using histogram distribution, gray-level co-occurrence matrix, and discrete location distribution. Over time, the scene may change, such as lighting conditions and background environment. Traditional, singular data augmentation methods struggle to cope with such changes in scenarios with small datasets. Therefore, this invention aims to enable the data augmentation model to automatically adapt to these changes, rather than remaining static. This invention uses a genetic evolutionary algorithm to find the most suitable parameter combination for the current scene. The adaptive update process of the scene feature parameter library is as follows:

[0078] 1.1) Initial Population: Randomly select a set of parameters from the parameter library as the initial population. These parameters include the type and intensity of data augmentation operations.

[0079] 1.2) Fitness assessment: The fitness of each parameter is assessed based on the model’s performance under the current parameters (such as accuracy, recall, etc.). Parameters that perform well are considered to be individuals with high fitness.

[0080] 1.3) Selection operation: Based on fitness, select the best-performing parameter combinations to enter the next generation. For example, a roulette wheel selection method can be used, where parameters with higher fitness have a higher probability of being selected.

[0081] 1.4) Crossover and Mutation: Perform crossover and mutation operations on the selected parameters to generate new parameter combinations. The crossover operation swaps some features of two sets of parameters, and the mutation operation randomly changes the values ​​of some parameters to introduce new mutations.

[0082] 1.5) Iterative update: Repeat the above process to continuously optimize the parameters until a certain number of iterations are reached or the fitness no longer improves significantly.

[0083] Besides considering scene awareness and adaptive evolution updates during data augmentation on edge devices, the computational complexity varies for models with different parameter levels. Models with lower parameter levels generally require weaker data augmentation computations, while models with higher parameter levels will have correspondingly higher data augmentation computation complexity. Edge devices have limited computing power, and if the data augmentation operation is too complex, it may cause the device to run slowly or even crash. If the data augmentation complexity is very low, the device's computing power may be idle, which is not conducive to improving the model's performance. Therefore, it is necessary to dynamically adjust the complexity of data augmentation according to the model's capacity (such as the number of parameters and layers).

[0084] Specifically, model width, model depth, operator complexity, and number of parameters are obtained based on model capacity assessment, and CPU, NPU, storage, and memory information are obtained based on device load perception. The model capacity assessment results and dual-device load information are combined with data augmentation computing to perform dynamic strategy updates.

[0085] When choosing a suitable data augmentation process, it is necessary to consider not only the model's own characteristics such as width and depth, but also data such as the device's status and load to comprehensively select a data augmentation strategy.

[0086] When device availability is sufficient, a lightweight reinforcement learning update strategy based on model feedback can be initiated. Reinforcement learning is a method that learns optimal strategies through interaction with the environment. For example, a robot learns how to reach a target location by continuously trying different actions. In the scenario of this invention, the model learns how to improve performance by trying different data augmentation strategies.

[0087] State definition: The state can include the model’s current performance metrics (such as accuracy and recall), device load, and historical data augmentation operations.

[0088] Action definition: An action is the selection and adjustment of data augmentation operations, such as which data augmentation operation to choose and the intensity of the operation.

[0089] Reward Definition: Rewards can be defined based on model performance improvements. For example, a positive reward is given if data augmentation improves the model's accuracy, and a negative reward is given if performance degrades. Considering the limited computing power of edge devices, a lightweight neural network is used to represent the policy network model. Only a small batch of data is sampled each time to interactively update the network parameters without utilizing all the data. Each policy update incorporates historical cached data for comprehensive optimization, avoiding getting stuck in local optimization regions.

[0090] Model development and training that integrates hardware awareness.

[0091] Traditional model development paradigms are mostly decoupled from hardware design patterns. For example, a model developed and trained on a cloud GPU server might have varying efficiency when deployed on other devices due to differences in the devices used. Hardware-aware development and training paradigms, on the other hand, tightly integrate the model training process with the characteristics of the target hardware, aiming to ensure that the trained model performs better on specific hardware.

[0092] The core idea of ​​hardware-aware training is to incorporate hardware constraints (such as latency, power consumption, and memory limitations) into the optimization objective of model training. Specifically, during training, it involves optimizing not only the model's accuracy but also its inference performance on specific hardware. For example, additional regularization terms can be introduced to constrain model complexity, thus taking hardware limitations into account during the training phase.

[0093] To effectively estimate the latency of models with different structures and parameter levels on different hardware devices, this invention develops a latency predictor based on LightGBM. Specifically, it quantizes features such as the number of layers, convolutional kernels, number of channels, operator complexity, and structure of the model. On the other hand, it also quantizes features such as the clock speed, computing power, CPU, and memory of the edge hardware devices. Based on the quantized features, latency prediction is performed using the LightGBM latency predictor. On different hardware platforms, such as mobile Qualcomm chips, MediaTek chips, Rockchip RK series chips, and Huawei Ascend 310 series chips, network models with different structures and parameter levels are developed and designed for extensive experimental testing and analysis. This results in the construction of a rich model latency prediction dataset, which is then used to develop and train the LightGBM latency predictor.

[0094] Using a single edge device allows for manual calculation of latency for a specific model on that device. However, this approach often faces significant challenges. Firstly, device load and status are constantly changing, making it difficult to effectively simulate device operation in an ideal experimental environment. This results in varying latency for the same device under different operating conditions. Secondly, existing manual experience bases often become ineffective under different hardware conditions, requiring the reconstruction of latency tables through re-experiments. Even with devices of similar performance, the results can differ considerably. By constructing rich experimental data from models with different structures and parameter magnitudes across various hardware platforms, latency predictors can be developed and trained to achieve accurate predictive analysis for unknown conditions.

[0095] Construct a loss function by adding the estimated delay as a regularization term:

[0096] Loss = Lossm + λ*Latency

[0097] Here, Lossm represents the model's own loss. For object detection tasks, Lossm consists of classification branch loss and regression branch loss:

[0098] Lossm = Losscls + Lossreg

[0099] During training, the accuracy and inference latency of the model are optimized simultaneously. By adjusting the value of λ, the trade-off between accuracy and latency can be controlled. This method takes into account the performance of the model on the target hardware during the training phase, thereby optimizing the model speed during the model development and training phases.

[0100] For different hardware platforms, a series of models with varying parameter levels are designed as candidate network model sets. Iterative training and optimization evaluation are performed on specific edge target hardware platforms. The latency of the model on the hardware platform is coupled for overall training iteration, optimization evaluation, and policy updates. The parameters are fed back to the model, and finally, the optimal subset of models is retained.

[0101] 2) Dataset in the cloud (or partially in the cloud), model training at the edge: Before starting the model training task, the necessary training dataset needs to be downloaded from the cloud to the edge. Once all the datasets have been successfully downloaded, the model training task begins. This mode mainly utilizes cloud resources for centralized data management (labeling), and then uses the idle computing resources at the edge for training, maximizing resource utilization.

[0102] 3) Dataset at the edge, model training in the cloud: In this case, the dataset needs to be uploaded to the cloud, and then the cloud will start the model training task through the model training cluster. In this mode, dataset management is completed at the edge, while training resources are concentrated in the cloud.

[0103] 4) Dataset and model training are both in the cloud: In this case, the model training task can be started directly in the cloud. This is the traditional model, where both dataset management and model training rely on cloud resources.

[0104] 5) Federated learning mode: The edge device trains locally based on local data, and then uploads the trained model parameters (instead of the training data) to the cloud. The cloud aggregates the parameters of all devices to update the global model, and finally trains a brand new model. This mode is suitable for scenarios that need to aggregate data from different edge devices while protecting data privacy.

[0105] The scheduling center allocates tasks based on minimizing the total latency using a mixed-integer programming model, thereby optimizing the scheduling of edge-cloud resources. The formula for minimizing the total latency is as follows:

[0106]

[0107] Among them, T total x represents minimizing the total delay. i ∈{0,1} indicates that task i is assigned to the edge (1) or the cloud (0), C i F represents the computational requirements of task i.edge F represents edge computing power. cloud D represents cloud computing power. i B represents the amount of data transmitted in task i, and B represents the network bandwidth.

[0108] Using a mixed-integer programming model, we define the task assignment variable x. i ∈{0,1}, dynamically select the task execution location (edge ​​or cloud); the system can dynamically adjust the task's data upload strategy based on bandwidth usage and transmission time to avoid bandwidth bottlenecks. Through localized processing and encryption technology, raw data is stored and processed only at the edge, prohibiting the transmission of unencrypted data to the cloud, ensuring zero leakage of sensitive data; in cloud collaboration mode, homomorphic encryption technology protects the transmission of model parameters, ensuring data privacy, and a federated learning framework is used to aggregate model parameters from multiple edge devices, uploading only the model gradients, avoiding the uploading of raw data; ultimately, cross-device model collaborative training is achieved, task execution efficiency is improved, network resource consumption is reduced, and the risk of data leakage is reduced.

[0109] The scheduling center also needs to consider real-time changes in system load to adjust task allocation and achieve load balancing. If the load at the edge exceeds its computing capacity limit (i.e., the load is higher than the load capacity), the center will adjust the task allocation accordingly. edge If (t) > 1, then some tasks will be migrated to the cloud for processing. The load ratio formula is as follows:

[0110]

[0111]

[0112] Among them, Load edge (t) represents the load ratio at the edge at time t. edge (j) represents the current load at edge j, C j C represents the total computing power of edge j. ij x represents the computational requirement of task i at edge j. ij Indicates whether task i is assigned to edge j (x) ij =1 indicates that it is assigned to the edge j, otherwise it is 0).

[0113] During task execution, the load on edge devices may change, leading to excessively high execution latency. In this case, a decision can be made to migrate the task based on real-time feedback to ensure task real-time performance and system load balancing. The task migration decision model is as follows:

[0114]

[0115] Where Migrate(i,j,k) indicates whether task i needs to be migrated from the edge j to the cloud k, 1 indicates migration, 0 indicates no migration, and t ij t represents the computation time of task i at edge j. threshold This indicates the latency threshold for the task at the edge. (Load) threshold This indicates the load threshold at the edge.

[0116] Step 5: After the model training is completed, it needs to be deployed to the edge. Since there are various types of hardware devices and different AI computing chips at the edge, the model needs to be converted into a specific version first. This can maximize the use of the chip's computing power, thereby accelerating the process and improving the inference efficiency at the edge. After the converted model is deployed to the edge, the model inference work can begin. The model inference will still use the data collected in Step 1. The collected data will be pushed to the deployed model, and the model will complete the inference work.

[0117] There are two ways to perform model conversion:

[0118] 1) Model conversion is performed at the edge, which depends on the capabilities of the edge device. Not all edge devices can perform arbitrary conversions.

[0119] 2) Model conversion is performed in the cloud, which generally supports various types of conversions. Both the trained and converted models are stored in a model repository for centralized management.

[0120] Model inference has two application directions:

[0121] 1) The data after model inference will be pushed to the business system. The business system will drive business based on the results of model inference. For example, it can drive the subsequent pest and disease treatment business process by detecting whether there are pests in the picture, or drive the fog cannon to be turned on by using whether there is dust in the picture.

[0122] 2) Model inference can also be used for pre-labeling work.

[0123] Using tools such as TensorRT and OpenVINO, the trained model is converted into an inference format adapted to the target hardware (e.g., FP16 quantization, layer fusion) to reduce model size and improve inference speed (compared to the unoptimized model); a dynamic hardware adaptation strategy is adopted, based on the hardware computing power F. hardware Automatically select the optimal model version and precision (e.g., enable INT8 quantization for low-computing-power devices); ultimately, achieve support for multiple mainstream edge computing platforms, reduce model inference latency (meeting real-time requirements), and improve hardware resource utilization.

[0124] Step 6: After model inference, the inference result set will be obtained. The inference result set needs to be manually sampled and checked according to the sampling strategy. The accuracy of the inference is marked. The results of manual sampling over a period of time are used to evaluate the accuracy of the model. When the accuracy of the model is lower than the preset threshold, a new round of model training will be started. The new model training will start from step 1.

[0125] Furthermore, after model inference, a set of inference results containing N results is obtained. A portion of these results needs to be sampled for manual review to evaluate the model's accuracy. The choice of sampling algorithm is particularly important because it directly affects the efficiency of manual review and the reliability of the model's accuracy evaluation. Monte Carlo sampling algorithm is a random sampling method widely used in statistics, physics, computer science, and other fields to estimate complex probability distributions, integrals, optimization problems, etc. This invention uses Monte Carlo sampling algorithm to approximate the characteristics of the entire dataset or probability distribution through random sampling.

[0126] Specifically, in one embodiment, the edge data closed-loop process is as follows: Figure 3 As shown, the process includes data collection, AI pre-labeling, manual correction, model training, model deployment, and monitoring and evaluation, in sequence.

[0127] Data acquisition involves capturing raw data in real time using devices such as cameras and sensors.

[0128] Triggering mechanism: Combining time-based and event-based triggering ensures that high-value data is processed first.

[0129] Data preprocessing: image denoising, video compression, and sensor data normalization.

[0130] AI pre-annotation: Calls a lightweight model at the edge to generate preliminary annotation results (boundary, category, and confidence score).

[0131] Model selection strategy: Different models are selected based on the scenario, with the following priority: First, historical versions already existing for the same scenario are preferred; second, if pre-trained models such as YOLO contain the corresponding labels, they are selected; finally, zero-shot models such as CLIP are chosen. The model selection strategy needs to be tailored to the specific scenario, with different pre-labeled models chosen for different scenarios. The main reason is that different models have different strengths. For example, historical versions of models and YOLO models are suitable for existing labels and have higher accuracy on existing labels. Theoretically, historically trained model versions may have higher accuracy than YOLO. CLIP can be applied to new labels, i.e., labels not present in previous models, but with relatively lower accuracy. Therefore, this selection is not based on complexity.

[0132] Output format: The annotation results are stored in COCO format, including image ID, target coordinates and confidence score.

[0133] Manual correction: Annotations are corrected using web-based visualization tools.

[0134] Intelligent assistance: Automatically highlights low-confidence annotations and recommends similar amendment examples (such as "scratches" being incorrectly labeled as "cracks" in historical annotations), significantly improving the efficiency of manual correction.

[0135] Model training: Perform local training on an edge computing board (such as Jetson AGX Xavier).

[0136] Training strategy: Using transfer learning (freezing the backbone network) and incremental learning (updating only the fully connected layers), the training time is reduced to 3 hours.

[0137] Data augmentation: random pruning, lighting perturbation, etc., to improve the model's generalization ability.

[0138] The edge training time model quantifies the advantages of localized processing:

[0139]

[0140] When the edge computing power F edge With 32 FLOPS, 100 iterations, and pure edge mode (α=0), the training time is only 4 hours, which is 50% shorter than the cloud solution.

[0141] Model deployment: Convert the trained model into an inference format adapted to the hardware and deploy it to the local device.

[0142] Dynamic adaptation: Automatically selects the optimal model version based on hardware computing power.

[0143] Monitoring and evaluation: Real-time collection of inference performance metrics (FPS, memory usage, mAP@0.5) and optimization of training strategies through feedback loops.

[0144] Automated iteration: If mAP is detected to drop to a certain threshold, data resampling and incremental model training are triggered.

[0145] Specifically, in another embodiment, through cloud-based collaborative process design, the cloud is used as a global scheduling and optimization center. Through dynamic task allocation, resource coordination, and closed-loop monitoring, efficient collaboration with edge devices is achieved. The specific process is as follows:

[0146] 1. Cloud-based scheduling of data annotation tasks

[0147] Task allocation strategy:

[0148] 1) Simple annotation task: The pre-annotation is completed by a lightweight model at the edge, and after manual correction, it can be directly used for local training.

[0149] 2) Complex annotation tasks: The cloud-based scheduling center calls cloud-based large models or crowdsourcing platforms to complete the annotation based on the complexity of the data.

[0150] Labeling quality control:

[0151] 1) Confidence level classification: The cloud-based annotation results are classified into three levels according to confidence level: high (>0.8), medium (0.5-0.8), and low (<0.5). Low confidence data are automatically assigned to the manual review queue.

[0152] 2) Multi-model cross-validation: When the results of dual annotation of the same data using CLIP (zero samples) and Faster R-CNN (high precision) are inconsistent, an alarm is triggered.

[0153] 2. Dynamic scheduling of model training tasks

[0154] Scheduling decision basis:

[0155] 1) Data scale:

[0156] Small-scale data: Prioritize allocation to the edge and utilize local computing power to complete training.

[0157] Large-scale data: After being split, it is scheduled to the cloud GPU cluster, which improves the efficiency of parallel training.

[0158] 2) Real-time requirements:

[0159] High real-time tasks: Forced assignment to the edge to ensure low training latency.

[0160] Non-real-time tasks: Assign them to the cloud to reduce costs by utilizing off-peak computing power.

[0161] The dynamic task scheduling algorithm optimizes resource allocation through a mixed-integer programming model, reducing task response time from 200ms to 150ms and bandwidth consumption by 35% using the Gurobi solver. The annotation quality evaluation formula verifies the effectiveness of the mixed strategy.

[0162]

[0163] When the pre-labeling accuracy is A pre =75%, manual correction efficiency β=0.8, the final accuracy rate is improved to 95%, and the annotation cycle is shortened by 60%.

[0164] Federated learning model:

[0165] 1) Edge participation: Each edge node trains a sub-model based on local data and only uploads the gradient to the cloud.

[0166] 2) Cloud aggregation: Aggregate gradients to generate a global model and distribute it to the edge, while keeping privacy data locally throughout the process.

[0167] Federated learning can improve the generalization ability of models, with a data leakage risk of 0%.

[0168] 3. Elastic allocation of model conversion services

[0169] Conversion strategy:

[0170] 1) Edge conversion: Suitable for lightweight models and common hardware, it achieves FP16 quantization through TensorRT with low time consumption.

[0171] 2) Cloud Conversion: For complex models or special hardware, the cloud-based OpenVINO / AscendCL toolchain is invoked to support batch conversion and format optimization.

[0172] Adaptive optimization:

[0173] 1) Hardware detection: Automatically identify the computing power of edge devices and select the optimal quantization strategy.

[0174] 2) Latency-accuracy tradeoff: If the device's computing power is insufficient, the model accuracy is dynamically reduced to ensure that the inference latency is less than 100ms.

[0175] 4. Closed-loop feedback of model deployment and monitoring

[0176] Dynamic deployment strategy:

[0177] 1) Intelligent distribution: The cloud pushes the appropriate model version based on the type of edge device (such as camera, robotic arm) and task requirements.

[0178] Example: A smart camera deploys a lightweight YOLOv8 model (10MB), while a robotic arm deploys a high-precision Mask R-CNN model (200MB).

[0179] 2) A / B testing: Deploy both new and old model versions simultaneously, compare performance, and automatically eliminate inefficient models.

[0180] Real-time monitoring and iterative triggering:

[0181] 1) Monitoring indicators:

[0182] Performance metrics: FPS, memory usage, mAP@0.5;

[0183] Business metrics: false positive rate, false negative rate, and response delay.

[0184] 2) Feedback mechanism:

[0185] If mAP@0.5 decreases by 5% or the false positive rate is >10%, the following process will be automatically triggered:

[0186] Data re-acquisition: Capture new scene data (such as product images under changes in lighting) triggered by events.

[0187] Incremental training: Training is completed at the edge or in the cloud based on historical model parameters;

[0188] Hot update: Replace fault models through silent deployment, with service interruption time of less than 2 minutes.

[0189] 5. Error handling and disaster recovery mechanism

[0190] Labeling process:

[0191] 1) Automatic correction: When a labeling conflict is detected in the cloud, the rule engine is invoked to automatically adjust it.

[0192] 2) Manual backup: If the data fails to pass the correction after 3 consecutive attempts, it will be upgraded to the expert team for processing.

[0193] Training phase:

[0194] 1) Checkpoint redundancy: Training progress is saved synchronously at the edge and in the cloud, and can be seamlessly switched in case of single node failure.

[0195] 2) Resource Exceedance Warning: If the edge card temperature is >85°C or the memory usage is >90%, the task will be automatically paused and migrated to the cloud.

[0196] Deployment phase:

[0197] 1) Rollback strategy: If the number of crashes is ≥3 within 24 hours after the new model is launched, it will be automatically rolled back to the previous stable version.

[0198] 2) Gray release: The first batch is deployed to 5% of edge devices, and the full rollout is pushed after verification.

[0199] Security and privacy protection:

[0200] The system employs a multi-layered mechanism to ensure data security. Edge data storage utilizes AES-256 encryption, and the transmission channel uses the TLS 1.3 protocol to prevent eavesdropping. In federated learning, only model gradients are uploaded; the global model is then distributed after cloud aggregation, preventing the leakage of raw data. The access control system is based on the RBAC model, defining the roles of administrators, annotators, and device operators to ensure traceability of operations. For example, in medical scenarios, patient image data is processed entirely locally, with only anonymized model parameters participating in cloud collaboration, fully complying with HIPAA compliance requirements.

[0201] Example 1: Intelligent Traffic Flow Monitoring System

[0202] 1. Data Acquisition: Intersection cameras capture real-time video of traffic flow (1080p resolution, 30 frames / second), and edge devices extract congestion segments according to event-triggered modes;

[0203] 2. Pre-labeling: The CLIP model generates labels such as "high vehicle density" and "pedestrians running red lights," and the accuracy rate reaches 96% after manual correction;

[0204] 3. Local training: Training a lightweight YOLOv8 model with Jetson AGX took 3 hours, a 60% speedup compared to cloud-based solutions;

[0205] 4. Model Deployment: Converted to TensorRT format, inference speed increased to 120 FPS, supporting real-time alarms and indicator light control;

[0206] 5. Cloud Collaboration: By calling cloud annotations for 10% of complex scenes (such as low-light nighttime), the training cycle is shortened by 25% and the model's generalization ability is improved by 15%.

[0207] Example 2: Intelligent Warehousing and Logistics System

[0208] 1. Scenario requirements: Warehouse robots need to dynamically optimize their path planning models to adapt to changes in shelf layout.

[0209] 2. Cloud-based collaboration process:

[0210] 1) Data annotation: The images of the new shelves collected by the robot are annotated in 3D space by the cloud-based SAM model, and the accuracy rate is 98% after manual correction.

[0211] 2) Training scheduling: Due to the large amount of data (50GB), the training is scheduled to the A100 cluster in the cloud, which takes 6 hours (24 hours at the edge).

[0212] 3) Model conversion: The PyTorch model is converted to TensorRT format in the cloud to adapt to the robot's built-in Jetson board.

[0213] 4) Deployment monitoring: After the model is deployed, the path planning latency is detected to increase from 50ms to 120ms, triggering incremental training and hot update, and the latency is restored to 60ms.

[0214] 3. Results: Warehouse efficiency improved by 25%, and the frequency of manual intervention decreased by 70%.

[0215] Comparison and advantages of this invention with existing technologies:

[0216] Dimensions Existing technology This invention Task scheduling Static allocation, choose either edge or cloud Dynamic and elastic scheduling, optimal allocation between edge and cloud annotation efficiency Fully manual or single model pre-annotation AI-plus-human hybrid annotation significantly improves efficiency. Privacy protection Data uploaded to the cloud carries a high risk of leakage. Federated learning + localized processing, zero data leakage Hardware Adaptation Manual adaptation is required for different hardware. Automatically detect computing power and dynamically select the optimal model version. Iteration speed It relies on manual intervention and takes several weeks. Automated closed-loop iteration reduces cycle time to 2 hours

[0217] This invention, through innovative designs such as a cloud-edge collaborative architecture, a hybrid annotation strategy, and automated closed-loop iteration, significantly outperforms existing technologies in terms of efficiency, cost, privacy protection, and flexibility. The following is a detailed comparative analysis across six dimensions:

[0218] 1) Training efficiency and real-time performance

[0219] Existing technologies: Traditional solutions rely on the cloud to complete model training, which requires uploading massive amounts of data to the cloud, resulting in high training latency due to the time-consuming transmission; edge devices only support inference, and model updates need to be downloaded from the cloud periodically, making it impossible to respond to scene changes in real time;

[0220] This invention achieves the entire process of data collection, labeling, training, and deployment at the edge through closed-loop training on the edge, thus shortening training time. It also employs dynamic incremental updates, triggering model iterations in real time through performance monitoring, and enabling hot updates in a short time under abnormal scenarios (such as sudden changes in illumination), ensuring real-time performance.

[0221] In smart security scenarios, the intrusion detection model update cycle has been shortened from one week to one day, reducing the false alarm rate.

[0222] 2) Labeling Costs and Quality

[0223] Existing technologies: annotation relies on manual work or a single pre-trained model, which is costly and has a high error rate; complex tasks (such as medical image segmentation) require professional annotation teams, which are time-consuming and inefficient.

[0224] This invention combines AI-human hybrid annotation, with pre-annotated models (YOLO / CLIP) covering over 70% of annotation tasks. Humans only need to correct low-confidence parts, reducing costs. It employs multi-model cross-validation and joint annotation conflict detection between CLIP and YOLO, reducing the annotation error rate to below 5%. Through cloud-based collaboration on complex tasks, it calls the SAM large model to complete pixel-level segmentation, improving annotation accuracy.

[0225] 3) Privacy protection and data security

[0226] Existing technologies: Data needs to be uploaded to the cloud for processing, which poses a risk of leakage; federated learning solutions are not widespread, and the problem of data silos at the edge is serious.

[0227] This invention: By localizing data processing, the original data is stored at the edge throughout the process, avoiding the leakage of sensitive information; by using federated learning fusion, only the encrypted model gradients are uploaded to the cloud, and the aggregated data is then distributed to the global model, resulting in zero exposure of data privacy.

[0228] 4) Hardware compatibility and resource utilization

[0229] Existing technologies: Models need to be manually adapted to different hardware (such as CPU / GPU / NPU), which is time-consuming and error-prone; edge devices have insufficient computing power utilization and serious resource idleness.

[0230] This invention employs a heterogeneous hardware adaptive mechanism, automatically converting model formats (such as FP16 quantization) through the TensorRT / OpenVINO toolchain, adapting to 10+ types of edge hardware (Rockchip, Jetson, Ascend, etc.), resulting in fast inference speed; through dynamic resource scheduling, the cloud allocates tasks according to the device's computing power (such as low-power boards executing lightweight models), resulting in high resource utilization.

[0231] 5) Network bandwidth and cost

[0232] Existing technology: Continuous uploading of high-definition video streams to the cloud consumes a large amount of bandwidth and has high transmission costs; cloud storage of petabyte-level data makes long-term costs difficult to control.

[0233] This invention compresses edge data using H.265 encoding and local caching, resulting in low bandwidth consumption; it also reduces power consumption through cloud-based collaborative efforts, uploading only anonymized data or model gradients, leading to low daily transmission volume.

[0234] 6) System flexibility and scalability

[0235] Existing technology: rigid architecture, disconnect between edge and cloud functions, making it difficult to adapt to the needs of multiple scenarios; expanding to new devices requires redeveloping interfaces, with integration cycles lasting up to several months.

[0236] This invention supports pure edge, pure cloud, and hybrid modes through a flexible collaboration model, allowing for on-demand switching (such as forced edge processing for emergency tasks); it features an open API and modular design, providing RESTful interfaces and compatibility with 20+ industrial protocols (such as OPCUA and Modbus), shortening the new device access cycle to 2 weeks;

[0237] In projects involving the Internet of Things, content generation, and smart cities, the deployment time for a management system for a thousand-level camera device has been reduced from six months to two months.

[0238] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A machine vision model training method based on edge computing power, characterized in that... The steps include: Acquire machine vision images; A pre-annotation model is constructed to automatically pre-annotate the image. Then, some pre-annotations are manually corrected to optimize the pre-annotation model and obtain the corrected annotated image. Build a machine vision model and train it based on labeled images. Allocate training task variables according to hardware perception. Construct an edge training efficiency model that considers edge training time, edge computing power, data volume, and machine vision model complexity. Dynamically select training tasks to be executed at the edge and / or in the cloud. Deploy the trained machine vision model at the edge for image recognition inference. Establish a hardware-aware development and training model that combines the training process of machine vision models with the characteristics of target hardware, and incorporates hardware constraints into the training objectives of machine vision models. Feature quantization is performed on model performance metrics and information from edge devices. Network models with different structures and parameter levels are developed and designed for different edge devices. A model latency prediction dataset is established, a latency predictor is constructed, and inference latency prediction training is performed based on the latency prediction dataset. A loss function is constructed, and the estimated latency is added as a regularization term to the loss function. The inference results are manually sampled to evaluate the accuracy of the machine vision model. When the accuracy is low, a new round of model training is started, and the evaluated machine vision model is used as the pre-labeled model.

2. The machine vision model training method based on edge computing power according to claim 1, characterized in that: For manually corrected images, the pre-labeled model finds other similar images and assigns correction weights to these other images based on similarity, so that the recognition results of these other images shift towards the manually corrected results.

3. The machine vision model training method based on edge computing power according to claim 2, characterized in that: If a manually selected image is found to be incorrectly identified, and this image is one of the other images after the correction weights have been adjusted, then the manually corrected image is reviewed. If the manual correction is correct, the correction weights are adjusted in reverse so that the recognition results of the other images revert to the results before the manual correction.

4. The machine vision model training method based on edge computing power according to claim 3, characterized in that: The images that are manually corrected and / or manually sampled are those labeled with low confidence.

5. The machine vision model training method based on edge computing power according to claim 1, characterized in that: In the edge-side training efficiency model, the product of the number of tasks and the computational cost per iteration at the edge is divided by the edge's computing power, and then the product of the data transmission coefficient and the data volume is added. When the edge trains alone, the data transmission coefficient is zero; when the edge and cloud collaborate on training, the data transmission coefficient is greater than zero. The function of edge-side training time is as follows: Where N represents the number of iterations, T train C represents the end-to-end training time. model F represents the computational cost of a single iteration of the model. edge D represents the computing power of edge devices. data This represents the dataset size, and α represents the data transmission coefficient. When the edge device processes data locally, α=0, and when the edge device collaborates with the cloud, α>0. For machine vision model updates based on gradient descent, the descent rate of the loss function is defined as the product of the square of the norm of the loss function's gradient, divided by the loss value of each iteration, and the learning rate. The descent rate of the loss function is as follows: Where η represents the rate of decrease of the loss function, L t This represents the loss value in the t-th iteration. Indicates the learning rate. The gradient of the loss function is represented; the convergence of the edge-end closed-loop iterative analysis is performed by using the descent rate of the loss function of the machine vision model, proving the effectiveness of edge-end closed-loop training.

6. The machine vision model training method based on edge computing power according to claim 1, characterized in that: The total latency is minimized by using a mixed-integer programming model to allocate tasks between the edge and the cloud. Minimizing the total latency is achieved by adding the computation time of the tasks allocated to the edge and the cloud to the data transmission time between the edge and the cloud.

7. The machine vision model training method based on edge computing power according to claim 1, characterized in that: The current load of the edge terminal is obtained by allocating the current computing tasks at the edge terminal. Based on the ratio of the current load to the total computing capacity of the edge terminal, it is determined whether the edge terminal exceeds its computing capacity. If it does, some computing tasks are migrated to the cloud. During the task execution process, a decision is made on whether to migrate the task based on the real-time feedback of the task. If the computation time of the task at the edge is greater than the latency threshold of the task at the edge, and / or the current load at the edge is greater than the load threshold of the edge, then some computation tasks will be migrated to the cloud.

8. The machine vision model training method based on edge computing power according to claim 1, characterized in that: The edge device performs feature analysis on the scene of the acquired machine vision images, establishes and updates the scene feature parameter library to perform data augmentation on the edge device training data; and dynamically adjusts the complexity of data augmentation according to the capacity of the machine vision model mounted on the edge device and the edge device information. The system uses the performance metrics of the computer vision model, information from edge devices, and historical data from data augmentation operations as states, and the selection and adjustment of data augmentation operations as actions. Based on the model feedback, a reinforcement learning update strategy is used to learn and improve the performance of edge devices.

9. A machine vision model training system based on edge computing power, comprising an edge device and a cloud device, characterized in that: According to any one of claims 1 to 8, a machine vision model training method based on edge computing power is used to collect data at the edge and train a machine vision model based on the collected images for image recognition.

Images