Machine vision model training method and system based on end side computing power

Abstract

The invention discloses a machine vision model training method and system based on end-side computing power, and belongs to the technical field of machine learning, edge computing and computer vision, and the method comprises the steps: obtaining a machine vision image and constructing a pre-annotation model, so as to carry out the automatic pre-annotation of the image; manually correcting the partial pre-annotation to optimize the pre-annotation model, and obtaining a corrected annotation image; a machine vision model is constructed and is subjected to identification training based on an annotated image, training task variables can be distributed according to hardware perception, then a training task execution position (an edge end or a cloud end) is dynamically selected, and the trained machine vision model is deployed at the edge end so as to carry out an image identification reasoning process; and performing manual spot check on the reasoning result of the model to evaluate the accuracy rate of the machine vision model, starting a new round of model training when the accuracy rate is low, and taking the evaluated machine vision model as the pre-labeling model.

Description

Technical Field

[0001] The present invention belongs to the fields of machine learning, edge computing and computer vision technology, and specifically relates to a machine vision model training method and system based on end-side computing power. Background Art

[0002] With the popularization of the Internet of Things and smart terminals, the demand for machine vision models has surged in the fields of smart Internet of Things, multimodal perception, content editing, low-altitude economy, industrial inspection, autonomous driving, etc., which has also led to the emergence of a way for the edge and cloud to collaborate in machine vision model training. Terminal devices or servers with independent data collection, processing and computing capabilities close to the data source are used as edge devices. Local computing resources (such as CPU, GPU, NPU) deployed at the edge (such as AI computing boards and smart terminals) support real-time data processing, which constitute the end-side computing power. Through the collaborative work of cloud servers and edge devices, data sharing and resource optimization are realized, and distributed training and unified management are supported.

[0003] However, model training in existing technologies still relies on the cloud to complete data labeling and model training. Since data needs to be uploaded to the cloud, it takes a significant amount of time and cannot meet real-time requirements (such as drone detection). High-definition image / video transmission takes up a lot of network resources, there is a bandwidth bottleneck, and it is costly. Uploading sensitive data (such as medical images) to the cloud may also violate privacy protection regulations and pose a risk of privacy leakage. On the other hand, existing edge computing is still limited to model inference. Model training cannot be based on the computing power of edge devices and still requires the use of cloud computing servers. Therefore, it is impossible to fully utilize the edge-side computing power, nor can it autonomously learn and dynamically update the model based on the edge-side computing power. It lacks the closed-loop capability of the entire process of end-side training and inference. In addition, the labeling of training data still relies heavily on manual labeling, which reduces efficiency, and the edge-cloud resource scheduling is rigid, and it is impossible to dynamically optimize latency and bandwidth consumption. Summary of the Invention

[0004] To address the shortcomings of existing technologies, achieve an automated closed-loop process from data collection to model iteration on the edge, and reduce reliance on the cloud, the present invention adopts the following technical solutions:

[0005] A machine vision model training method based on end-side computing power includes the following steps:

[0006] Acquire machine vision images;

[0007] Construct a pre-annotation model to automatically pre-annotate images, then manually correct some of the pre-annotations to optimize the pre-annotation model and obtain the corrected annotated images;

[0008] Build a machine vision model and train it for recognition based on annotated images. Allocate training task variables based on hardware perception, dynamically select the training task execution location (edge ​​or cloud), and deploy the trained machine vision model on the edge for image recognition reasoning.

[0009] The inference results are manually spot-checked to evaluate the accuracy of the machine vision model. When the accuracy is low, a new round of model training is started, and the evaluated machine vision model is used as the pre-labeled model.

[0010] Furthermore, for manually corrected images, the pre-labeling model finds other images similar to them and assigns correction weights to other images based on similarity, so that the recognition results of other images are shifted towards the manually corrected results, thereby enabling a small amount of manual correction to have a wider impact on the pre-labeling results.

[0011] Furthermore, if an image that is identified incorrectly during manual spot checking is the other image after adjustment with the correction weight, it means that the category of the other image is changed to an incorrect category based on the association adjustment based on similarity, or a correction error occurs during the manual correction process, the manually corrected image is reviewed. If the manual correction is correct, the correction weight is adjusted in reverse so that the recognition result of the other image returns to the result before the manual correction.

[0012] Furthermore, the manual correction and / or manual spot check is performed on images with low confidence annotations.

[0013] Furthermore, by constructing an edge-side training efficiency model that compares edge training time with edge computing power, data volume, and machine vision model complexity, we quantify the reduced reliance of edge-side training on the cloud and demonstrate the efficiency advantage of localized edge processing. In the edge-side training efficiency model, the product of the number of tasks and the amount of computation required for a single iteration on the edge is divided by the edge computing power, and then the product of the data transmission coefficient and the data volume is added. When the edge is training alone, the data transmission coefficient is zero. When the edge and the cloud collaborate on training, the data transmission coefficient is greater than zero.

[0014] Based on the update of the machine vision model based on gradient descent, the convergence of the edge closed-loop iteration is analyzed. The effectiveness of the edge closed-loop training is proved by the descent rate of the loss function of the machine vision model, ensuring the rapid convergence of the model iteration. The descent rate of the loss function is the product of the square of the norm of the loss function gradient divided by the iterative loss value and the learning rate.

[0015] Furthermore, the total delay is minimized through a mixed integer programming model, and tasks are allocated between the edge and the cloud to achieve edge-cloud resource scheduling optimization. The minimized total delay is obtained by adding the computing time of the tasks allocated to the edge and the cloud to the data transmission time between the edge and the cloud.

[0016] Define the task allocation variable x through the mixed integer programming model i ∈{0,1}, dynamically select the task execution location (edge ​​or cloud); the system can dynamically adjust the task data upload strategy according to bandwidth usage and transmission time to avoid bandwidth bottlenecks.

[0017] Furthermore, the current edge load is obtained by the assigned current edge computing tasks. Based on the ratio of the current load to the total computing capacity of the edge, it is determined whether the edge exceeds its computing capacity. If so, part of the computing tasks are migrated to the cloud.

[0018] During the execution of the identification task, the load on the edge may change, resulting in excessive execution delay. At this time, it can be decided whether to migrate the task based on the real-time feedback of the task to ensure the real-time performance of the task and the load balance of the system. If the calculation time of the task on the edge is greater than the delay threshold of the task on the edge, and / or the current load on the edge is greater than the edge load threshold, part of the computing task will be migrated to the cloud.

[0019] Furthermore, the edge performs feature analysis on the scenes of the collected machine vision images, establishes and updates a scene feature parameter library, and performs data enhancement on the edge training data to achieve scene perception and adaptive evolutionary updates; dynamically adjusts the complexity of data enhancement according to the capacity of the machine vision model carried on the edge and the edge device information; when the device has ample available time, the performance indicators of the computer vision model, the information of the edge device, and the historical data of the data enhancement operation are used as states, and the selection and adjustment of the data enhancement operation are used as actions. A lightweight reinforcement learning update strategy based on model feedback is used to learn and improve the performance of the edge device, thereby balancing the data enhancement effect and the consumption of edge computing resources.

[0020] Furthermore, a hardware-aware development and training model is established to closely integrate the training process of the machine vision model with the characteristics of the target hardware, so that the trained model has better performance on specific hardware. The hardware constraints are incorporated into the training objectives of the machine vision model, which not only optimizes the accuracy of the model, but also optimizes the inference performance of the model on specific hardware. The model performance indicators and edge device information are quantified, and network models with different structures and parameter levels are developed and designed on different edge devices. A model delay prediction dataset is established, a delay predictor is constructed, and inference delay prediction training is performed based on the delay prediction dataset. A loss function is constructed to integrate the target detection loss and inference delay loss of the machine vision model itself, and iterative training and optimization evaluation are performed on a specific edge target hardware platform. The overall training iteration and optimization evaluation are performed by coupling the model delay on the hardware platform. The strategy parameters are updated and fed back to the model. Finally, the optimal model subset is retained, thereby controlling the trade-off between accuracy and delay, taking into account the performance of the model on the target hardware during the training phase, and optimizing the model speed during the model development and training phase.

[0021] Traditional development models mostly design structures and conduct experimental analysis on high-computing devices. Even if the same model is later migrated to different hardware devices, the delay generated will be different. The limited computing power of IoT weak-computing devices needs to be fully utilized as much as possible. Models with too many parameters and too complex structures may not run properly. Conversely, the existing computing power resources are not fully utilized. The delay on the hardware device side is a factor in model design and development, mainly to avoid traditional development models. Constructing inference delay loss through hardware delay and using it as part of the loss function to guide the design of reasonable and effective network structures is the current optimal solution.

[0022] A machine vision model training system based on end-side computing power includes an edge end and a cloud end. According to the machine vision model training method based on end-side computing power, data collection is performed on the edge end, and a machine vision model is trained based on the collected images for image recognition.

[0023] The advantages and beneficial effects of the present invention are:

[0024] The present invention uses key technologies such as hardware perception training, cloud-edge dynamic collaborative architecture, hybrid labeling strategy, and heterogeneous hardware adaptation to achieve a closed loop of the entire process from data collection, labeling, training, deployment, and model iteration at the edge, thereby reducing dependence on the cloud, supporting the integration of edge training and reasoning, and significantly improving the speed, real-time performance, and privacy security of personalized training. At the same time, it supports elastic edge-cloud collaboration, with the cloud providing elastic computing power and complex task processing capabilities, and the edge ensuring real-time performance and privacy security. The two are seamlessly connected through dynamic scheduling and federated learning, solving the shortcomings of traditional solutions in efficiency, privacy, and flexibility, and providing efficient, secure, and scalable machine vision solutions for high-real-time scenarios such as smart Internet of Things, multimodal perception, content acquisition and editing, low-altitude economy, industrial detection, and smart cities. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is an architecture diagram of the system in an embodiment of the present invention.

[0026] Figure 2 is a flow chart of a method in an embodiment of the present invention.

[0027] Figure 3 It is a closed-loop flow chart of edge data in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0029] like Figure 1 As shown in the figure, a machine vision model training system based on edge computing power includes the cloud and edge, and realizes efficient interaction between data and tasks through standardized interfaces. The overall architecture adopts a layered design. The edge mainly includes the data acquisition layer, computing layer and application layer. The data acquisition layer integrates cameras, sensors and other devices to support real-time data capture and local caching, ensuring that high-value data (such as scene content editing clips and abnormal event video clips) is processed first. The computing layer deploys AI computing boards such as RK3588, NVIDIA Jetson AGX Xavier or Huawei Ascend Atlas 300 to provide local computing power support. The application layer covers modules such as data labeling, model training, model deployment, monitoring and evaluation, and cloud synchronization, forming a closed-loop iterative process from data to model.

[0030] The data collection layer captures raw data in real time through cameras, sensors, and other devices. Cameras support H.265 encoding. Sensor data is transmitted to the edge computing layer via the MQTT protocol, ensuring low latency. The collection module has a built-in priority queue, prioritizing abnormal event data to ensure real-time response. Data collected by the data collection layer flows in two directions: one is used for labeling, which can be used for model training; the other is provided for model inference. The results of model inference are used in specific business applications, such as scenario content editing, pest and disease identification, and pollution detection.

[0031] The computing layer provides local computing power based on heterogeneous hardware platforms and provides computing power through AI computing boards. Its functions include two parts: one is to serve as training data for the model, and the other is used for model inference after the model is deployed.

[0032] AI computing boards include Rockchip, Ascend, and NVIDIA. The Rockchip RK3588 is suitable for low-power scenarios (such as smart cameras), supports INT8 quantized inference, and has a computing power of 6 TOPS. The Ascend Atlas 300 is designed for Huawei ecosystem devices, optimizes models through AscendCL, and has an FP16 computing power of 16 TFLOPS. The NVIDIA Jetson AGX Xavier supports TensorRT acceleration, has an INT8 computing power of 32 TOPS, and is suitable for high-throughput tasks (such as real-time video analysis).

[0033] The data labeling module combines AI pre-labeling and manual correction, including a pre-labeling engine and a manual correction interface. The pre-labeling engine calls the edge lightweight model and historical version model to generate preliminary labels with high accuracy; the manual correction interface provides a web-based tool that supports label box correction, category screening, and confidence filtering; the corrected data is fed back to the pre-labeling model to achieve self-optimization and iteration; the data of the data labeling module comes from the data acquisition module.

[0034] Through the coordinated settings of the model training module, model deployment module, and monitoring and evaluation module, model training and deployment are carried out to achieve a closed-loop training and inference on the terminal side.

[0035] The model training module performs visual model training on the computing board, including training of various types of computer vision models such as image classification, object detection, instance segmentation, and semantic segmentation. For example, it can identify crop pests and diseases in agricultural scenarios, identify restaurant fumes and construction site dust pollution in environmental protection scenarios, identify production management and maintenance activities in content generation scenarios, and identify vehicle black smoke in traffic scenarios. The model training phase requires relevant computing power, which is provided by different types of AI boards. Because different AI boards rely on different toolkits, they need to be adapted for different types of computing boards. The data annotation module is used to annotate data during the model training phase. The quality and quantity of the annotated data, as well as its representativeness, determine the quality of the trained model. To adapt to AI hardware from different manufacturers, the trained model needs to be converted into a model version suitable for different manufacturers. The model is automatically converted by automatically detecting the type of board hardware to convert it into a model suitable for different boards. At the same time, because the performance of different hardware varies, in order to achieve performance balance, multiple models also need to be quantized to achieve a balance between model recognition performance and detection rate.

[0036] The model deployment module and monitoring and evaluation module deploy the converted model to the edge device. The deployed model monitors the inference performance (such as FPS and memory usage) in real time. If the accuracy drops by more than a certain threshold, the incremental training process is automatically triggered.

[0037] The cloud synchronization module is used for two-way interaction between data and models. Through data upload, sensitive data (such as facial images) encrypted at the edge is uploaded to the cloud data warehouse via the TLS protocol for federated learning or global model optimization. Through model distribution, optimized models are regularly pushed from the cloud model warehouse to the edge, such as silent updates during low-bandwidth periods at night to reduce the impact on the business.

[0038] Error handling and disaster recovery mechanisms are adopted, including data loss disaster recovery, training interruption recovery, and network fluctuation response. Data loss disaster recovery uses a RAID 1 storage architecture at the edge, and key data is backed up locally and in the cloud in real time. Training interruption recovery uses a checkpoint mechanism to save training progress, and can be restored from the most recent state after an interruption, with accuracy loss less than 1% (for model development training, two weights, best and last, are retained. The last weight is used as the starting point, and training continues from the breakpoint. If the model training process is relatively smooth, the accuracy loss can be less than 1%. If the best weight is used as the starting point, it is assumed that there is no accuracy loss). The MQTT protocol is used to support breakpoint resumption in response to network fluctuations, and unfinished tasks are automatically synchronized after the network is restored.

[0039] The cloud architecture includes a resource pool and a dispatching center. The resource pool provides large-model labeling services (such as complex scene description generation based on GPT-4V) and distributed training clusters, supporting PB-level data storage and multi-device collaborative training. The dispatching center dynamically allocates tasks to the edge or cloud based on a mixed integer programming algorithm. For example, large-scale labeling tasks can be split to a cloud GPU cluster, while training tasks with high real-time requirements are retained at the edge. The communication layer uses the MQTT protocol to achieve efficient communication between the edge and the cloud. Data encryption transmission (AES-256) ensures security, and compression technology can reduce bandwidth usage by 40%.

[0040] The dispatch center is the cloud's central module, responsible for global task allocation and resource coordination. Using a mixed-integer programming algorithm, it dynamically determines whether tasks (such as large-scale training or complex labeling) should be allocated to the cloud resource pool or edge devices. For example, if insufficient computing power is detected on the edge (e.g., Rockchip board load exceeds 80%), the dispatch center automatically migrates some training tasks to a cloud-based model training cluster (such as an AWS p4d instance), ensuring task response times below 200ms.

[0041] The resource pool integrates high-performance computing and storage resources in the cloud, including sub-modules such as large model annotation, model training cluster, data warehouse, and model warehouse;

[0042] Label large models, call GPT-4V, SAM (Segment Anything Model) and other models with hundreds of billions of parameters to handle complex labeling tasks, with a labeling accuracy of over 90%;

[0043] Model training cluster, deploying distributed GPU clusters to support parallel training of large-scale models;

[0044] Data warehouse, data set storage and management;

[0045] The model warehouse stores optimized model versions and provides format conversion services. For example, it converts PyTorch models into ONNX format to adapt to different edge hardware (for example, Ascend boards need to be converted to OM format). The warehouse ensures model integrity through hash verification and records version iteration logs for easy rollback and auditing.

[0046] like Figure 2 As shown in the figure, a machine vision model training method based on edge computing power includes the following steps:

[0047] Step 1: The edge collects data through cameras, sensors and other collection devices to obtain various types of multimodal data such as images and sounds on agricultural crop growth and environmental pollution;

[0048] Step 2: The collected data needs to be pre-labeled. AI pre-labeling can be performed in the following ways:

[0049] 1) If an older version of the AI ​​pre-labeling model has been deployed on the edge, the older version of the model will be used for pre-labeling.

[0050] 2) If the edge only has a pre-trained model, pre-labeling is performed using the pre-trained model, for example, using the YOLO pre-trained model for pre-labeling;

[0051] 3) In cloud-edge collaboration scenarios, data can be uploaded to the cloud and pre-labeled by models deployed on the cloud, including historical version models, pre-trained models, and even large labeled models.

[0052] Step 3: Since pre-labeled data cannot be fully trusted, manual revision is required. The pre-labeled data is manually reviewed and revised through the data labeling platform and stored in the labeled data set. The data in the data set can be used in the subsequent model training phase. In the cloud-edge collaborative mode, the manually revised and pre-labeled data will be stored in the data warehouse. There are two ways to manually revise:

[0053] 1) Manual revisions are performed on the edge. The edge provides a web data annotation tool, which allows data annotators to perform data revisions. The revised data is directly stored on the edge. If a cloud service is available, the revised data can be synchronized to the cloud for storage.

[0054] 2) In the cloud-based collaborative mode, manual revisions are performed in the cloud. The edge will synchronize pre-annotated data to the cloud. The cloud also provides a web data annotation tool, and the annotators will perform revisions and annotations based on the cloud-based annotation tools. The cloud can also submit data that requires manual revision and annotation to a third-party cloud annotation platform for annotation, of course, with the user's permission.

[0055] Through AI pre-labeling and manual collaborative correction, the labeling cost can be significantly reduced and the data quality can be improved. The pre-labeling quality is evaluated by the final accuracy rate. The final accuracy rate is A final as follows:

[0056]

[0057] Among them, A preβ represents the AI ​​pre-labeling accuracy, while β represents the manual correction efficiency coefficient, reflecting the proportion of manual corrections to incorrect labels. By establishing an AI pre-labeling strategy, invoking pre-trained models (such as YOLOv5), zero-shot detection models (such as CLIP), or historical version models to generate preliminary labeling results, combined with a manual correction mechanism and visual labeling tools that support error highlighting, one-click correction, and batch review, we can ultimately achieve the goal of reducing labeling costs (compared to manual labeling), lowering the labeling error rate, and shortening the labeling cycle.

[0058] Pre-labeling uses AI models (such as YOLOv5, CLIP, and historical version models) to perform preliminary labeling on raw data. The AI ​​model automatically generates bounding boxes, labels, confidence levels, and other information based on pre-trained knowledge. This process significantly improves labeling speed, especially for data with known categories.

[0059] Manual correction involves human annotators reviewing the AI-pre-annotated results and correcting any errors or low-confidence annotations. This correction primarily focuses on areas where the model's pre-annotation error rate is high (such as images with unclear object recognition or unclear data categories). The corrected data is then fed back to the pre-annotated model for further optimization.

[0060] By updating the data set, the corrected data is stored in the annotation database for subsequent training. In this way, the system realizes the dynamic coordination of automatic annotation and manual correction, and continuously optimizes the annotation quality.

[0061] The AI ​​pre-labeling model will self-optimize based on manually corrected data to gradually reduce labeling errors. The quality of manual corrections directly affects the improvement of the pre-labeling model, enabling the AI ​​model to continuously improve its accuracy.

[0062] In one embodiment, the AI ​​pre-labeling model obtains the manually corrected category and the data features corresponding to the category, compares the similarity of the data features with other data features, builds an association relationship between the data features through the similarity, and assigns weights to other data features related to the manually corrected data features to appropriately adjust the categories of the other associated data features. For example: the category of the manually corrected data feature 1 is A, and the corresponding category probability is p. The category of the data feature 2 similar to the data feature 1 is B, and the corresponding category probability is q. The weight w1 is calculated based on the similarity s of the data feature 1 and the data feature 2. If the weight w1*q>p, the category of the data feature 2 is automatically adjusted from B to A, thereby affecting the categories of other associated data through a single manual correction, thereby achieving adaptive category adjustment.

[0063] In addition, in step 6, the model will be manually spot-checked and reviewed. If the category of data feature 2 is A, that is, the association adjustment based on the manual correction is valid, then no processing is performed, or a positive review adjustment is performed, such as adjusting the similarity calculation to obtain a higher weight w2, and / or increasing the bias b so that w2*q+b is larger than the value of p; if the category of data feature 2 is B, that is, the association adjustment changes the AI ​​pre-labeled correct category to an incorrect category, then a reverse review adjustment is performed, such as adjusting the similarity calculation to obtain a lower weight w3, and / or lowering the bias b so that w2*q+b<p. In addition, based on the spot-checked data features, the manually reviewed categories can also be used for association adjustment.

[0064] The above-mentioned review-based adjustment is effective for all associated data features involved in the association algorithm, and the true category obtained through review can also be directly used for the model parameters of the AI ​​pre-labeling model. Starting from classification recognition, what is adjusted is the category probability of data feature recognition itself.

[0065] For data feature 2 whose category adjustment error is caused by association adjustment found during the review process, first find data feature 1 through reverse association, and manually review its category to eliminate the correction error of the manual correction itself. After confirming that the manual correction is correct, the above-mentioned review and adjustment are carried out.

[0066] Through the feedback mechanism, the AI ​​model can be continuously optimized based on manually corrected data to improve labeling efficiency and accuracy. On the other hand, manual correction reduces manual labor by only correcting samples that are difficult for the AI ​​pre-labeling model to label or have low confidence.

[0067] Step 4: Perform model training based on the labeled dataset. Model training can be performed in the following ways:

[0068] 1) Datasets and model training at the edge: Model training tasks are directly scheduled. These tasks read the local dataset, begin model training, and ultimately generate a standard model parameter file, resulting in a trained model. This model manages and trains all data at the edge. It's particularly suitable for scenarios where the model only needs to be optimized for data from a specific device scenario, and there's no need to aggregate data from other nodes for training.

[0069] The on-device closed-loop training system requires the construction of an on-device training efficiency model and closed-loop iterative convergence analysis. This model quantifies the reduced reliance on the cloud for on-device training by building a function of edge device computing power, data volume, and model complexity, demonstrating the efficiency advantages of localized processing. The function of on-device training time is as follows:

[0070]

[0071] Among them, T train represents the end-side training time, N represents the number of model iterations, and C model Indicates the computational cost of a single iteration of the model (unit: FLOPs), F edge Indicates the computing power of edge devices (unit: FLOPs / s), D data represents the dataset size (unit: GB), α represents the data transmission coefficient (α=0 for local processing and α>0 for cloud collaboration).

[0072] Closed-loop iterative convergence analysis is based on gradient descent model updates. The loss function descent rate is defined to demonstrate the effectiveness of on-device closed-loop training and ensure rapid convergence of model iterations. The loss function descent rate is defined as follows:

[0073]

[0074] Among them, η represents the rate of decrease of the loss function, L t represents the loss value of the tth iteration, represents the learning rate, represents the gradient of the loss function.

[0075] Localized data management is performed at the edge. Raw data is collected according to preset rules (such as time windows and event triggers). Local caching and priority scheduling are supported to ensure that high-value data is processed first. Data preprocessing (such as denoising and normalization) is completed at the edge, eliminating the need to upload raw data to the cloud. This achieves a closed-loop training loop on the edge. After model training and optimization are completed on the edge device, the performance monitoring module provides real-time feedback on inference results (such as accuracy and latency), driving dynamic adjustment of model parameters. The closed-loop iteration cycle is shortened from the "weekly" level of traditional solutions to "days" to adapt to rapidly changing business needs. Ultimately, it reduces training latency, protects data privacy, and improves model iteration efficiency.

[0076] Adaptive data enhancement based on scenario awareness on the edge. Edge devices are often deployed in specific scenarios and may face the problem of insufficient training data. Therefore, it is necessary to build a lightweight data enhancement model on the edge. The key to lightweight data enhancement models is to balance the effect of data enhancement and the consumption of edge computing resources.

[0077] The edge end will continuously collect data for analysis and storage, and perform sampling calculations according to the specified strategy through the feature analysis module to calculate and update the scene feature parameter library. For example: the histogram distribution is obtained through color features, the grayscale co-occurrence matrix is ​​obtained through texture features, and the discrete position distribution is obtained through spatial features. The feature parameter library is constructed and updated through the histogram distribution, grayscale co-occurrence matrix, and discrete position distribution. As time goes by, the scene may change, such as lighting conditions, background environment, etc. The traditional single data enhancement method is difficult to cope with such changes in small data volume scenarios. Therefore, the present invention needs to enable the data enhancement model to automatically adapt to these changes, rather than remaining unchanged. The present invention uses a genetic evolutionary algorithm to find the parameter combination that best suits the current scene. The process of adaptive updating of the scene feature parameter library is as follows:

[0078] 1.1) Initial population: A set of parameters is randomly selected from the parameter library as the initial population. These parameters include the type and strength of the data augmentation operation.

[0079] 1.2) Fitness Evaluation: The fitness of each parameter set is evaluated based on the model's performance (e.g., accuracy, recall, etc.) under the current parameters. Parameters with good performance are considered to have high fitness.

[0080] 1.3) Selection: Based on fitness, select the best performing parameter combinations to enter the next generation. For example, a roulette wheel selection method can be used. Parameters with high fitness have a higher probability of being selected.

[0081] 1.4) Crossover and mutation: Perform crossover and mutation operations on selected parameters to generate new parameter combinations. The crossover operation exchanges some features of two sets of parameters, while the mutation operation randomly changes the values ​​of certain parameters to introduce new variations.

[0082] 1.5) Iterative update: Repeat the above process and continuously optimize the parameters until a certain number of iterations is reached or the fitness no longer improves significantly.

[0083] In addition to considering scene perception and adaptive evolutionary updates during the data enhancement process of the end-side device, the computational complexity presented is also different for models with different parameter magnitudes. Models with lower parameter magnitudes generally require weaker data enhancement calculations, and models with higher parameter magnitudes have correspondingly increased complexity in data enhancement calculations. The computing power of edge devices is limited. If the data enhancement operation is too complex, it may cause the device to run slowly or even crash. If the data enhancement complexity is very low, the device computing power may be idle, which is not conducive to improving the model's effect. Therefore, it is necessary to dynamically adjust the complexity of data enhancement according to the capacity of the model (such as the number of parameters, number of layers, etc.).

[0084] Specifically, the model width, model depth, operator complexity, parameter quantity, etc. are obtained based on the model capacity assessment, and the CPU, NPU, storage, and memory information are obtained based on the device load perception. The model capacity assessment results and dual-device load information are combined with data enhancement calculations to perform dynamic strategy updates.

[0085] When choosing a suitable data enhancement process, we must not only consider the width, depth and other characteristics of the model itself, but also consider data such as the status and load of the device to comprehensively select the data enhancement strategy.

[0086] When the device has ample time available, a lightweight reinforcement learning update strategy based on model feedback can be initiated. Reinforcement learning is a method of learning optimal strategies through interaction with the environment. For example, a robot learns how to reach a target location by repeatedly trying different actions. In the scenario of the present invention, the model learns how to improve performance by trying different data augmentation strategies.

[0087] State definition: The state can include the current performance indicators of the model (such as accuracy and recall), the load of the device, and the history of data augmentation operations.

[0088] Action definition: Action is the selection and adjustment of data augmentation operations, such as which data augmentation operation to choose and the intensity of the operation.

[0089] Reward Definition: Rewards can be defined based on model performance improvements. For example, if data augmentation improves model accuracy, a positive reward is given; if performance degrades, a negative reward is given. Considering the limited computing power of edge devices, a lightweight neural network is used to represent the policy network model. Only a small batch of data is sampled each time to interactively update network parameters without utilizing all data. Each policy update is combined with historical cached data for comprehensive optimization to avoid being trapped in a localized optimization zone.

[0090] Model development and training that integrates hardware awareness.

[0091] Traditional model development models are largely hardware-decoupled. For example, a model of a specific size may be developed and trained on a cloud GPU server. When deployed on other devices, the efficiency of the model may vary depending on the device. Hardware-aware development and training approaches closely integrate the model training process with the characteristics of the target hardware, aiming to achieve better performance for the trained model on specific hardware.

[0092] The core concept of hardware-aware training is to incorporate hardware constraints (such as latency, power consumption, and memory limitations) into the optimization objectives of model training. Specifically, during training, the model's accuracy is optimized, as well as its inference performance on specific hardware. For example, additional regularization terms can be introduced to constrain model complexity, thus accounting for hardware limitations during training.

[0093] In order to effectively estimate the delay of models with different structures and parameter magnitudes on different hardware devices, the present invention develops and implements a delay predictor based on LightGBM. Specifically, the number of layers, convolution kernels, number of channels, operator complexity, structure, etc. of the model are feature quantized. On the other hand, the main frequency, computing power, CPU, memory, etc. of the edge hardware devices are also feature quantized. Based on the quantized features, delay prediction is performed through the LightGBM delay predictor. On different hardware platforms, such as: mobile Qualcomm, MediaTek chips, Rockchip RK series chips, Huawei Ascend 310 series chips and other hardware devices, network models with different structures and parameter magnitudes are developed and designed to carry out extensive experimental testing and analysis, so as to construct a rich model delay prediction data set and then develop and train the LightGBM delay predictor.

[0094] If a single edge device is used, the delay calculation for the corresponding model on a specific device can be manually implemented. However, this approach often faces significant challenges. On the one hand, the load and status of the device are constantly changing, and an ideal experimental environment cannot effectively simulate the device's operating information, resulting in different delays for the same device under different operating conditions. On the other hand, when faced with different hardware conditions, the existing manual experience library often becomes invalid, and new experiments need to be conducted to construct the delay table. Even for devices with similar performance, the results may vary significantly. By constructing rich experimental data with models of different structures and parameter magnitudes on different hardware platforms, it is possible to develop and train a delay predictor for accurate prediction and analysis of unknown conditions.

[0095] Construct the loss function and add the estimated delay as a regularization term to the loss function:

[0096] Loss = Lossm + λ*Latency

[0097] Among them, Lossm represents the loss of the model itself. For the target detection task, Lossm consists of classification branch loss and regression branch loss:

[0098] Lossm = Losscls + Lossreg

[0099] During the training process, the model's accuracy and inference latency are optimized simultaneously. By adjusting the value of λ, the trade-off between accuracy and latency can be controlled. This approach can take into account the performance of the model on the target hardware during the training phase, thereby optimizing the model's speed during the model development and training phase.

[0100] For different hardware platforms, a series of models with different parameter levels are designed as candidate network model sets. They are iteratively trained and optimized on specific edge target hardware platforms. The coupled model delay on the hardware platform is used for overall training iteration, optimization evaluation, and the update strategy is used to feed back parameters to the model, ultimately retaining the optimal model subset.

[0101] 2) Datasets in the cloud (or partially in the cloud), model training on the edge: Before starting model training, the required datasets must be downloaded from the cloud to the edge. Once all datasets have been successfully downloaded, model training begins. This model primarily leverages cloud resources for centralized data management (labeling), while training utilizes idle computing resources on the edge, maximizing resource utilization.

[0102] 3) Dataset at the edge, model training in the cloud: In this scenario, the dataset needs to be uploaded to the cloud, which then initiates the model training task using the model training cluster. In this model, dataset management is performed at the edge, while training resources are centralized in the cloud.

[0103] 4) Datasets and model training in the cloud: In this scenario, model training can be initiated directly on the cloud. This is a traditional model where both dataset management and model training rely on cloud resources.

[0104] 5) Federated Learning Mode: The edge performs local training based on local data, and then uploads the trained model parameters (instead of the training data) to the cloud. The cloud aggregates the parameters of all devices to update the global model, and finally trains a new model. This mode is suitable for scenarios that require both aggregating data from different edge devices and protecting data privacy.

[0105] The dispatch center allocates tasks based on minimizing the total delay through a mixed integer programming model to optimize the scheduling of edge-cloud resources. The formula for minimizing the total delay is as follows:

[0106]

[0107] Among them, T total Minimize the total delay, x i ∈{0,1} indicates that task i is assigned to the edge (1) or the cloud (0), C i represents the computational requirements of task i, Fedge Indicates edge computing power, F cloud represents cloud computing power, D i represents the data transmission volume of task i, and B represents the network bandwidth.

[0108] Define the task allocation variable x through the mixed integer programming model i ∈{0,1}, dynamically selecting the task execution location (edge ​​or cloud); the system can dynamically adjust the task data upload strategy based on bandwidth usage and transmission time to avoid bandwidth bottlenecks. Through local processing and encryption technology, data is stored and processed only at the edge, prohibiting unencrypted data transmission to the cloud, ensuring zero leakage of sensitive data. In cloud collaboration mode, homomorphic encryption technology protects model parameter transmission and ensures data privacy. A federated learning framework is used to aggregate model parameters from multiple edge devices, uploading only model gradients and avoiding uploading raw data. Ultimately, cross-device model collaborative training is achieved, improving task execution efficiency, reducing network resource utilization, and mitigating the risk of data leakage.

[0109] The dispatch center also needs to consider the real-time changes in system load to adjust the distribution of tasks and achieve load balancing. If the load on the edge exceeds its computing capacity limit (i.e., the load ratio edge (t)>1), some tasks are migrated to the cloud for processing. The load ratio formula is as follows:

[0110]

[0111]

[0112] Among them, Load edge (t) represents the load ratio of the edge at time t, Load edge (j) represents the current load of edge j, C j represents the total computing power of edge j, C ij represents the computing requirements of task i at edge j, x ij Indicates whether task i is assigned to edge j (x ij =1 indicates that it is assigned to edge end j, otherwise 0).

[0113] During task execution, the load on edge devices may change, resulting in excessive execution delays. In this case, we can decide whether to migrate the task based on real-time feedback to ensure task real-time performance and system load balancing. The task migration decision model is as follows:

[0114]

[0115] Migrate(i,j,k) indicates whether task i needs to be migrated from edge j to cloud k. 1 indicates migration and 0 indicates no migration. ij represents the computation time of task i at edge j, t threshold Indicates the delay threshold of the task on the edge, Load threshold Indicates the edge load threshold.

[0116] Step 5: After model training is completed, the model needs to be deployed to the edge. Since there are various types of hardware devices and AI computing chips on the edge, the model needs to be converted into a specific version first. This can maximize the computing power of the chip, thereby achieving acceleration and improving the reasoning efficiency of the edge. After the converted model is deployed on the edge, model reasoning can begin. Model reasoning will still use the data collected in step 1. The collected data will be pushed to the deployed model, and the model will complete the reasoning work.

[0117] There are two ways to convert models:

[0118] 1) Model conversion at the edge depends on the capabilities of the edge. Not all edge devices can perform any type of conversion.

[0119] 2) Perform model conversion in the cloud. Generally, the cloud can perform various types of conversions. Both trained and converted models are stored in the model repository for centralized management.

[0120] Model reasoning has two application directions:

[0121] 1) The data after model inference is pushed to the business system, which drives the business based on the results of model inference. For example, the system can detect the presence of pests in an image to drive the subsequent pest control business process, or use the presence of dust in an image to activate a fog cannon.

[0122] 2) Model inference can also be used as pre-annotation work.

[0123] Based on tools such as TensorRT and OpenVINO, the trained model is converted into an inference format that is compatible with the target hardware (such as FP16 quantization and layer fusion), which reduces the model size and improves the inference speed (compared to the unoptimized model); a dynamic hardware adaptation strategy is used to adjust the hardware computing power according to the F hardware Automatically select the optimal model version and accuracy (such as enabling INT8 quantization for low-computing power devices); ultimately, support multiple mainstream edge computing platforms, reduce model inference latency (to meet real-time requirements), and improve hardware resource utilization.

[0124] Step 6: After model inference, a set of inference results will be obtained. Subsequently, manual spot checks will be conducted on the inference result set. The results of the model inference will be reviewed according to the spot check strategy and the accuracy of the inference will be marked. The results of manual spot checks over a period of time will be counted to evaluate the accuracy of the model. When the model accuracy falls below the preset threshold, a new round of model training will be initiated, and the new model training will start from step 1.

[0125] Furthermore, after model inference, an inference result set containing N results is obtained. A part of the samples needs to be extracted for manual review to evaluate the accuracy of the model. The choice of sampling algorithm becomes particularly important because it directly affects the efficiency of manual review and the reliability of model accuracy evaluation. The Monte Carlo sampling algorithm is a method based on random sampling, which is widely used in statistics, physics, computer science and other fields to estimate complex probability distributions, integrals, optimization problems, etc. The present invention uses the Monte Carlo sampling algorithm to approximate the characteristics of the entire data set or probability distribution through random sampling.

[0126] Specifically, in one embodiment, the edge data closed loop process is as follows: Figure 3 As shown, it includes data collection, AI pre-labeling, manual correction, model training, model deployment, and monitoring and evaluation.

[0127] Data collection captures raw data in real time through cameras, sensors and other devices.

[0128] Trigger mechanism: Combine time triggering with event triggering to ensure that high-value data is processed first.

[0129] Data preprocessing: image denoising, video compression, and sensor data normalization.

[0130] AI pre-labeling: Call the lightweight model on the edge to generate preliminary labeling results (target box, category, and confidence).

[0131] Model selection strategy: Choose different models based on the scenario. The priority is: first, give priority to the existing historical version of the same scenario. Secondly, if the pre-trained model such as YOLO contains the corresponding label, choose the pre-trained model. Finally, choose a zero-shot model such as CLIP. The model selection strategy needs to be based on the specific scenario. Different pre-labeled models are selected for different scenarios. The main reason is that different models have different strengths. For example, the historical version model and the YOLO model are suitable for the labels they already have, and have higher accuracy for the existing labels. In theory, the historically trained model version may have a higher accuracy than YOLO. CLIP can be applied to new labels, that is, labels that the previous model did not have, but the accuracy is relatively low. Therefore, this is not selected based on complexity.

[0132] Output format: The annotation results are stored in COCO format, including image ID, object coordinates and confidence score.

[0133] Manual correction: perform annotation correction through web-based visualization tools.

[0134] Intelligent assistance: Automatically highlight low-confidence annotations and recommend similar correction examples (such as "scratches" in historical annotations mistakenly marked as "cracks"), significantly improving the efficiency of manual corrections.

[0135] Model training: Perform local training on edge computing boards such as Jetson AGX Xavier.

[0136] Training strategy: Using transfer learning (freezing the backbone network) and incremental learning (updating only the fully connected layers), training time was shortened to 3 hours.

[0137] Data enhancement: random cropping, lighting perturbation, etc., to improve the generalization ability of the model.

[0138] The on-device training time model quantifies the advantages of localized processing:

[0139]

[0140] When the edge computing power F edge =32FLOPS, number of iterations N=100, and pure edge mode (α=0), the training time is only 4 hours, which is 50% shorter than the cloud solution.

[0141] Model deployment: Convert the trained model into an inference format suitable for the hardware and deploy it to the local device.

[0142] Dynamic adaptation: Automatically select the optimal model version based on hardware computing power.

[0143] Monitoring and evaluation: Collect inference performance metrics (FPS, memory usage, mAP@0.5) in real time and optimize training strategies through feedback loops.

[0144] Automatic iteration: If the mAP is detected to drop to a certain threshold, data re-collection and model incremental training are triggered.

[0145] Specifically, in another embodiment, through cloud-based collaborative process design, the cloud is used as a global scheduling and optimization center, and efficient collaboration with the edge is achieved through dynamic task allocation, resource coordination, and closed-loop monitoring. The specific process is as follows:

[0146] 1. Cloud-based scheduling of data annotation tasks

[0147] Task allocation strategy:

[0148] 1) Simple labeling tasks: Pre-labeling is completed by a lightweight model on the edge, and after manual correction, it is directly used for local training.

[0149] 2) Complex labeling tasks: The cloud dispatch center calls on a large cloud model or crowdsourcing platform to complete the labeling based on the complexity of the data.

[0150] Annotation quality control:

[0151] 1) Confidence grading: Cloud-based annotation results are classified into three levels based on confidence: high (>0.8), medium (0.5-0.8), and low (<0.5). Low-confidence data is automatically assigned to the manual review queue.

[0152] 2) Multi-model cross-validation: The same data is double-labeled using CLIP (zero-sample) and Faster R-CNN (high-precision). An alarm is triggered when the results are inconsistent.

[0153] 2. Dynamic Scheduling of Model Training Tasks

[0154] Scheduling decision-making basis:

[0155] 1) Data scale:

[0156] Small-scale data: Prioritize distribution to the edge and use local computing power to complete training.

[0157] Large-scale data: After splitting, it is dispatched to the cloud GPU cluster to improve parallel training efficiency.

[0158] 2) Real-time requirements:

[0159] Highly real-time tasks: These tasks are forcibly assigned to the edge to ensure low training latency.

[0160] Non-real-time tasks: Assign them to the cloud to reduce costs by utilizing idle computing power.

[0161] The dynamic task scheduling algorithm optimizes resource allocation through a mixed integer programming model and uses the Gurobi solver to reduce task response time from 200ms to 150ms, reducing bandwidth consumption by 35%. The annotation quality assessment formula verifies the effectiveness of the hybrid strategy:

[0162]

[0163] When the pre-labeling accuracy A pre =75%, manual correction efficiency β=0.8, the final accuracy rate is improved to 95%, and the labeling cycle is shortened by 60%.

[0164] Federated learning model:

[0165] 1) Edge participation: Each edge node trains a sub-model based on local data and only uploads the gradient to the cloud.

[0166] 2) Cloud aggregation: Aggregate gradients to generate a global model, which is then sent to the edge. Private data is retained locally throughout the process.

[0167] Federated learning can improve the generalization ability of the model, and the risk of data leakage is 0%.

[0168] 3. Elastic Allocation of Model Conversion Services

[0169] Conversion strategy:

[0170] 1) Edge conversion: Suitable for lightweight models and common hardware, it implements FP16 quantization through TensorRT and is time-efficient.

[0171] 2) Cloud conversion: For complex models or special hardware, call the cloud-based OpenVINO / AscendCL tool chain to support batch conversion and format optimization.

[0172] Adaptive Optimization:

[0173] 1) Hardware detection: Automatically identify the computing power of edge devices and select the optimal quantization strategy.

[0174] 2) Latency-Accuracy Tradeoff: If the device computing power is insufficient, the model accuracy is dynamically reduced to ensure that the inference latency is less than 100ms.

[0175] 4. Closed-loop feedback for model deployment and monitoring

[0176] Dynamic deployment strategy:

[0177] 1) Intelligent distribution: The cloud pushes the appropriate model version based on the edge device type (such as camera, robotic arm) and task requirements.

[0178] Example: A lightweight YOLOv8 model (10 MB) is deployed on a smart camera, and a high-precision Mask R-CNN model (200 MB) is deployed on a robotic arm.

[0179] 2) A / B testing: Deploy new and old model versions simultaneously, compare their performance, and automatically eliminate inefficient models.

[0180] Real-time monitoring and iteration triggering:

[0181] 1) Monitoring indicators:

[0182] Performance indicators: FPS, memory usage, mAP@0.5;

[0183] Business indicators: false detection rate, missed detection rate, and response delay.

[0184] 2) Feedback mechanism:

[0185] If mAP@0.5 drops by 5% or the false positive rate is greater than 10%, the following process is automatically triggered:

[0186] Data recapture: Capture new scene data (e.g., product images under changing lighting conditions) triggered by events.

[0187] Incremental training: Training is completed on the edge or in the cloud based on historical model parameters;

[0188] Hot update: Replace the faulty model through silent deployment, with service interruption time less than 2 minutes.

[0189] 5. Error handling and disaster recovery mechanism

[0190] Marking stage:

[0191] 1) Automatic correction: The cloud detects annotation conflicts and calls the rule engine to automatically adjust.

[0192] 2) Manual backup: Data that fails three consecutive revisions will be escalated to an expert team for processing.

[0193] Training session:

[0194] 1) Checkpoint redundancy: Training progress is saved synchronously on the edge and in the cloud, enabling seamless switching in the event of a single node failure.

[0195] 2) Resource overlimit warning: If the edge card temperature is greater than 85°C or the memory is greater than 90%, the task will be automatically paused and migrated to the cloud.

[0196] Deployment phase:

[0197] 1) Rollback strategy: If a new model crashes ≥ 3 times within 24 hours after it goes online, it will automatically roll back to the previous stable version.

[0198] 2) Grayscale release: The first batch is deployed to 5% of edge devices, and then pushed to the entire network after verification.

[0199] Security and Privacy Protection:

[0200] The system ensures data security through a multi-layered mechanism. Edge data storage utilizes AES-256 encryption, and transmission channels utilize the TLS 1.3 protocol to prevent eavesdropping. In federated learning, only model gradients are uploaded, and the global model is distributed after aggregation in the cloud, preventing the leakage of raw data. The permission control system, based on the RBAC model, divides the roles of administrators, annotators, and device operators to ensure traceability. For example, in medical scenarios, patient imaging data is processed entirely locally, and only desensitized model parameters are used for cloud collaboration, fully complying with HIPAA compliance requirements.

[0201] Example 1: Intelligent Traffic Flow Monitoring System

[0202] 1. Data collection: Intersection cameras capture real-time traffic video (resolution 1080p, 30 frames per second), and the edge device captures congestion segments based on event triggering.

[0203] 2. Pre-labeling: The CLIP model generates labels such as "high vehicle density" and "pedestrian running a red light," with an accuracy rate of 96% after manual correction.

[0204] 3. Local training: Jetson AGX trains a lightweight YOLOv8 model in 3 hours, 60% faster than cloud-based solutions.

[0205] 4. Model deployment: Convert to TensorRT format, increase inference speed to 120 FPS, and support real-time alarms and traffic light control;

[0206] 5. Cloud collaboration: Calling the cloud to annotate 10% of complex scenes (such as low-light conditions at night), shortening the training cycle by 25%, and improving model generalization capabilities by 15%.

[0207] Example 2: Smart Warehousing and Logistics System

[0208] 1. Scenario requirements: Warehouse robots need to dynamically optimize path planning models to adapt to changes in shelf layout.

[0209] 2. Cloud collaboration process:

[0210] 1) Data Annotation: New shelf images collected by the robot are annotated in 3D space using the cloud-based SAM model, with an accuracy of 98% after manual correction.

[0211] 2) Training Scheduling: Due to the large amount of data (50GB), training was scheduled to the cloud-based A100 cluster, which took 6 hours (24 hours on the edge).

[0212] 3) Model conversion: The cloud converts the PyTorch model into TensorRT format to adapt it to the robot's built-in Jetson board.

[0213] 4) Deployment Monitoring: After the model was deployed, it was detected that the path planning delay increased from 50ms to 120ms, triggering incremental training and hot updates, and the delay was restored to 60ms.

[0214] 3. Effect: Warehousing efficiency increased by 25% and the frequency of manual intervention decreased by 70%.

[0215] Comparison and advantages of the present invention over the prior art:

[0216] Dimensions Existing technology The present invention Task Scheduling Static allocation, choose between edge or cloud Dynamic elastic scheduling, edge-cloud collaborative optimal allocation Annotation efficiency Fully manual or single model pre-labeling AI+manual hybrid annotation significantly improves efficiency Privacy Protection Data is uploaded to the cloud, which poses a high risk of leakage. Federated learning + localized processing, zero data leakage Hardware Adaptation Manual adaptation to different hardware is required Automatically detect computing power and dynamically select the optimal model version Iteration speed Relying on manual intervention, the cycle can take several weeks Automated closed-loop iteration, shortening the cycle to 2 hours

[0217] This invention significantly outperforms existing technologies in terms of efficiency, cost, privacy protection, and flexibility through innovative designs such as a cloud-edge collaborative architecture, a hybrid annotation strategy, and automated closed-loop iteration. The following is a detailed comparative analysis from six dimensions:

[0218] 1) Training efficiency and real-time performance

[0219] Existing technologies: Traditional solutions rely on the cloud to complete model training, requiring massive amounts of data to be uploaded to the cloud. This time-consuming transmission leads to high training latency. The edge only supports inference, and model updates must be periodically downloaded from the cloud, making it unable to respond to scenario changes in real time.

[0220] This invention uses closed-loop training on the edge to complete the entire process of data collection, labeling, training, and deployment at the edge, shortening training time. It also uses dynamic incremental updates to trigger model iterations in real time through performance monitoring, completing hot updates in a short time under abnormal scenarios (such as sudden changes in lighting) to ensure real-time performance.

[0221] In smart security scenarios, the intrusion detection model update cycle is shortened from 1 week to 1 day, and the false alarm rate is reduced.

[0222] 2) Labeling cost and quality

[0223] Existing technologies: Labeling relies on manual labor or a single pre-trained model, which is costly and has a high error rate. Complex tasks (such as medical image segmentation) require professional labeling teams, which are time-consuming and inefficient.

[0224] This invention combines AI-human hybrid annotation, with the pre-annotation model (YOLO / CLIP) covering more than 70% of the annotation tasks. Manual correction is only required for low-confidence parts, reducing costs. Multi-model cross-validation is adopted, and CLIP and YOLO are jointly used for annotation conflict detection, reducing the annotation error rate to below 5%. Through cloud-based collaboration on complex tasks, the SAM large model is called to complete pixel-level segmentation, improving annotation accuracy.

[0225] 3) Privacy protection and data security

[0226] Existing technology: Data needs to be uploaded to the cloud for processing, which poses a risk of leakage; federated learning solutions are not popular, and the problem of data silos at the edge is serious.

[0227] This invention: Through data localization processing, the original data is retained at the edge throughout the process to avoid the leakage of sensitive information; through federated learning fusion, only the encrypted model gradient is uploaded to the cloud, and the global model is issued after aggregation, with zero exposure to data privacy.

[0228] 4) Hardware adaptation and resource utilization

[0229] Existing technologies: Models need to be manually adapted to different hardware (such as CPU / GPU / NPU), and conversion is time-consuming and error-prone. Edge device computing power is underutilized, resulting in serious resource idleness.

[0230] This invention adopts a heterogeneous hardware adaptation mechanism, automatically converts model formats (such as FP16 quantization) through the TensorRT / OpenVINO tool chain, adapts to 10+ types of edge hardware (Rockchip, Jetson, Ascend, etc.), and has a fast inference speed. Through dynamic resource scheduling, the cloud allocates tasks according to the device computing power (such as low-power boards executing lightweight models), and has high resource utilization.

[0231] 5) Network bandwidth and cost

[0232] Existing technology: High-definition video streams are continuously uploaded to the cloud, occupying a large amount of bandwidth and resulting in high transmission costs; the cloud also stores PB-level data, making long-term costs difficult to control.

[0233] The present invention compresses edge data using H.265 encoding and local caching, which reduces bandwidth usage. It also reduces power consumption through cloud collaboration, only uploading desensitized data or model gradients, resulting in a low average daily transmission volume.

[0234] 6) System flexibility and scalability

[0235] Existing technologies: Rigid architecture, fragmented edge and cloud functionality, making it difficult to adapt to multiple scenarios; expansion of new devices requires redevelopment of interfaces, with integration cycles lasting several months.

[0236] This invention: Through flexible collaboration modes, it supports pure edge, pure cloud, and hybrid modes, switching on demand (such as mandatory edge processing for urgent tasks). It also features an open API and modular design, provides a RESTful interface, and is compatible with more than 20 industrial protocols (such as OPCUA and Modbus). The new device integration cycle is shortened to 2 weeks.

[0237] In smart IoT, content generation, and smart city projects, the deployment time of a management system for thousands of cameras has been shortened from six months to two months.

[0238] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A machine vision model training method based on end-side computing power, characterized by The steps include: Acquire machine vision images; Construct a pre-annotation model to automatically pre-annotate images, then manually correct some of the pre-annotations to optimize the pre-annotation model and obtain the corrected annotated images; Build a machine vision model and train it for recognition based on annotated images. Assign training task variables based on hardware perception, dynamically select training tasks to execute on the edge and / or in the cloud, and deploy the trained machine vision model on the edge for image recognition reasoning. The inference results are manually spot-checked to evaluate the accuracy of the machine vision model. When the accuracy is low, a new round of model training is started, and the evaluated machine vision model is used as the pre-labeled model.

2. The machine vision model training method based on end-side computing power according to claim 1, characterized in that: For manually corrected images, the pre-labeling model finds other images similar to the images and assigns correction weights to the other images based on similarity, so that the recognition results of the other images are shifted towards the manually corrected results.

3. The machine vision model training method based on end-side computing power according to claim 2, characterized in that: If an image that is identified incorrectly during manual spot checking is the other image after adjustment with the correction weight, the manually corrected image will be reviewed. If the manual correction is correct, the correction weight will be adjusted in reverse to make the recognition result of the other image return to the result before the manual correction.

4. The machine vision model training method based on end-side computing power according to claim 3 is characterized by: The manual correction and / or manual spot check is performed on images with low confidence annotations.

5. The machine vision model training method based on device-side computing power according to claim 1, characterized in that: By building an edge-side training efficiency model that combines edge training time with edge computing power, data volume, and machine vision model complexity, the model multiplies the number of tasks by the edge's single iteration computational effort, divided by the edge's computing power, and then adds the data transmission coefficient to the product of the data volume. When the edge trains alone, the data transmission coefficient is zero. When the edge and the cloud collaborate on training, the data transmission coefficient is greater than zero. Based on the update of the machine vision model based on gradient descent, the convergence of the edge closed-loop iteration is analyzed. The effectiveness of the edge closed-loop training is proved by the descent rate of the loss function of the machine vision model. The descent rate of the loss function is the product of the square of the norm of the loss function gradient divided by the iterative loss value and the learning rate.

6. The machine vision model training method based on terminal computing power according to claim 1, characterized in that: The total delay is minimized by a mixed integer programming model to allocate tasks to the edge and cloud. The minimized total delay is obtained by adding the computing time of the tasks allocated to the edge and cloud to the data transmission time between the edge and cloud.

7. The machine vision model training method based on terminal computing power according to claim 1, characterized in that: The current edge load is obtained by the assigned edge computing tasks. Based on the ratio of the current load to the total computing capacity of the edge, it is determined whether the edge exceeds its computing capacity. If so, some computing tasks are migrated to the cloud. During the execution of the identification task, whether to migrate the task is decided based on the real-time feedback of the task. If the computing time of the task at the edge is greater than the delay threshold of the task at the edge, and / or the current load at the edge is greater than the edge load threshold, part of the computing task will be migrated to the cloud.

8. The machine vision model training method based on terminal computing power according to claim 1, characterized in that: The edge performs feature analysis on the scene of the collected machine vision image, establishes and updates the scene feature parameter library to perform data enhancement on the edge training data; dynamically adjusts the complexity of the data enhancement based on the capacity of the machine vision model installed on the edge and the edge device information; The performance indicators of the computer vision model, the information of the edge device, and the historical data of the data augmentation operation are used as the state, and the selection and adjustment of the data augmentation operation are used as the action. The reinforcement learning update strategy based on model feedback is used to learn to improve the performance of the edge device.

9. The machine vision model training method based on terminal computing power according to claim 1, characterized in that: Establish a hardware-aware development and training model that combines the machine vision model training process with the target hardware characteristics, incorporating hardware constraints into the machine vision model training objectives; Quantify the features of model performance indicators and edge device information, develop and design network models with different structures and parameter magnitudes on different edge devices, establish a model delay prediction dataset, build a delay predictor and perform prediction training for inference delay based on the delay prediction dataset; construct a loss function that integrates the target detection loss and inference delay loss of the machine vision model itself.

10. A machine vision model training system based on edge computing power, including edge devices and cloud devices, characterized by: According to the machine vision model training method based on edge computing power according to any one of claims 1 to 9, data collection is performed on the edge, and a machine vision model is trained based on the collected images for image recognition.

Images