An edge cloud-based visual recognition algorithm polymorphic model system and an application system thereof

Abstract

The application discloses a kind of based on edge cloud visual recognition algorithm polymorphism model system and its application system, belong to computer vision technical field, including cloud and edge, cloud includes polymorphism model management module, model optimization module, edge includes preprocessing module, the polymorphism model management module includes n relevant visual recognition model, based on the calculation precision, complexity and load of different visual recognition model, through intelligent decision mechanism, the most suitable model is automatically matched for current task, the model optimization module, collects the visual recognition model sample in use.The application discloses a kind of based on edge cloud visual recognition algorithm polymorphism model system and its application system, can realize model management, model screening, model optimization, resource optimization, significantly improve the response speed of visual recognition system, resource utilization efficiency, applicable to various real-time, accuracy and security have high requirement application scene.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to an edge cloud-based visual recognition algorithm polymorphic model system and its application system. Background Art

[0002] A visual recognition model is a model based on computer vision technology that aims to enable computers to recognize and understand the content in images or videos. It simulates the workings of the human visual system and uses image processing and pattern recognition technology to recognize objects or scenes in images.

[0003] Traditional visual recognition systems often rely on centralized cloud computing, resulting in high latency and inflexible resource allocation. With the development of edge computing, migrating visual recognition tasks to edge nodes has become a trend. However, these systems still face problems such as poor model adaptability and insufficient resource optimization. They are unable to flexibly select visual recognition models based on the needs of the actual usage environment, lack certain functionality, and suffer from low practicality.

[0004] In response to the above, this case proposes a visual recognition algorithm polymorphic model system based on edge cloud and its application system to solve the above technical problems. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes an edge cloud-based visual recognition algorithm polymorphic model system and its application system, which solves the above-mentioned technical problems by improving the detection and processing methods.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A polymorphic model system for visual recognition algorithms based on edge cloud includes a cloud and an edge. The cloud includes a polymorphic model management module and a model optimization module, and the edge includes a preprocessing module, wherein:

[0008] The polymorphic model management module includes n related visual recognition models, and automatically matches the most suitable model for the current task through an intelligent decision-making mechanism based on the calculation accuracy, complexity and load of different visual recognition models;

[0009] The model optimization module collects samples of the visual recognition model in use, and based on the correct and incorrect samples obtained, incrementally updates the unused models, thereby continuously improving the visual recognition model;

[0010] The preprocessing module performs data preprocessing and feature extraction at the edge through data preprocessing algorithms and feature extraction, thereby reducing the amount of data transmitted to the cloud and improving the processing speed.

[0011] Furthermore, the polymorphic model management module includes n related visual recognition models, and automatically matches the most suitable model for the current task through an intelligent decision-making mechanism based on the calculation accuracy, complexity and load of different visual recognition models, including the following steps:

[0012] Based on the computational accuracy, complexity, and load of different visual recognition models, the visual recognition models within the module are classified, and the specificity classification is performed based on the computational accuracy, complexity, and load of the model;

[0013] Establish an intelligent decision-making matching model to automatically match the most suitable model for the current task based on the priority of the current visual recognition task.

[0014] Furthermore, the visual recognition models within the module are classified based on the computational accuracy, complexity, and load of different visual recognition models, and specific classification is performed based on the computational accuracy, complexity, and load of the model. The specific steps are as follows:

[0015] For the comprehensive visual recognition model, the computational accuracy of the model is specifically classified according to the model's F1 score. The F1 score is an indicator that comprehensively measures the model's precision and recall rate. The specific steps are as follows:

[0016] When the F1 score of the relevant model is less than 0.7, the relevant model is marked as a low-precision model;

[0017] When the F1 score of the relevant model is 0.7≤≤0.9, the relevant model is marked as a medium-precision model;

[0018] When the F1 score of the relevant model is greater than 0.9, the relevant model is marked as a high-precision model;

[0019] Based on the number of floating-point operations (FLOPs) of the model, the complexity of the model is specifically classified. The specific steps are as follows:

[0020] When the FLOPs of the relevant model is less than 10 8 , then the relevant model is marked as a low-complexity model;

[0021] When the 10 8 ≤FLOPs≤10 9 , then the relevant model is marked as a medium complexity model;

[0022] When the FLOPs of the relevant model is greater than 10 9 , then the relevant model is marked as a high complexity model;

[0023] Based on the model's CPU usage, the model load is specifically classified. The specific steps are as follows:

[0024] When the CPU usage of the relevant model is less than 30%, the relevant model is marked as a low-load model;

[0025] When 30% of the relevant model is ≤CPU usage ≤70%, the relevant model is marked as a medium load model;

[0026] When the CPU usage of a related model is greater than 70%, the related model is marked as a high-load model.

[0027] Furthermore, the intelligent decision-making matching model is established to automatically match the most suitable model for the current task based on the priority of the current visual recognition task. The specific steps are as follows:

[0028] Establish an intelligent decision-making matching priority mode, including accuracy priority, complexity priority, load priority, and mixed priority, set different levels of model scores, and divide the judgment conditions into three equal parts based on the judgment conditions for different levels of models:

[0029] Among them, the scores of low-precision models are 2 points, 3 points, and 4 points respectively, the scores of medium-precision models are 5 points, 6 points, and 7 points respectively, the scores of high-precision models are 8 points, 9 points, and 10 points respectively, the scores of low-complexity models are 8 points, 9 points, and 10 points respectively, the scores of medium-complexity models are 5 points, 6 points, and 7 points respectively, the scores of high-complexity models are 2 points, 3 points, and 4 points respectively, the scores of low-load models are 8 points, 9 points, and 10 points respectively, the scores of medium-load models are 5 points, 6 points, and 7 points respectively, and the scores of high-load models are 2 points, 3 points, and 4 points respectively;

[0030] In the accuracy priority mode, based on the accuracy requirements of the current task, the models that meet the accuracy requirements of the current task are retained, and the first residual comprehensive score Z1 of each model in the retained models is calculated separately, where:

[0031] Z1=B+C;

[0032] In the above formula, B represents the model complexity score, C represents the model load score, the first remaining comprehensive score Z1 is sorted in descending order, and the first model is selected as the most suitable model. When the first remaining comprehensive score Z1 is the same, the model load score C of the models with the same score is sorted in descending order, and the first model is selected as the most suitable model. When the model load score C is the same, the model complexity score B of the models with the same score is sorted in descending order, and the first model is selected as the most suitable model.

[0033] In the complexity priority mode, based on the complexity requirements of the current task, the models that meet the complexity requirements of the current task are retained, and the second residual comprehensive score Z2 of each model in the retained models is calculated separately, where:

[0034] Z2=A+C;

[0035] In the above formula, A represents the model accuracy score, C represents the model load score, the second residual comprehensive score Z2 is sorted in descending order, and the first model is selected as the most suitable model. When the second residual comprehensive score Z2 is the same, the model accuracy scores A of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model load scores C of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model.

[0036] In the load priority mode, based on the load requirements of the current task, the models that meet the load requirements of the current task are retained, and the third residual comprehensive score Z3 of each model in the retained models is calculated respectively, where:

[0037] Z3=A+B;

[0038] In the above formula, A represents the model accuracy score, B represents the model complexity score, the third remaining comprehensive score Z3 is sorted in descending order, and the first model is selected as the most suitable model. When the third remaining comprehensive score Z3 is the same, the model accuracy scores A of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model complexity scores B of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model.

[0039] In the hybrid priority mode, when the current task has accuracy requirements, complexity requirements, and load requirements, the model that meets the current task requirements is retained, and the comprehensive score Z4 is calculated for the retained model, where:

[0040] Z4=A+B+C;

[0041] Arrange the comprehensive score Z4 in descending order and select the first model as the most suitable model;

[0042] When the current task meets any two of the accuracy requirements, complexity requirements, and load requirements, among the models that meet the requirements, the remaining requirement scores are sorted in descending order, and the first model is selected as the most suitable model.

[0043] Furthermore, the model optimization module collects samples of the visual recognition models in use and performs incremental updates on unused models based on the correct and incorrect samples obtained to ensure continuous improvement of the algorithm. The specific steps are as follows:

[0044] Based on the selected optimal visual recognition model, relevant parameters during use are recorded, including correct samples and error samples. The unused visual recognition models are updated and optimized in the cloud using the correct and error samples. The specific steps are as follows:

[0045] Collect samples that are correctly identified by the visual recognition model in use, along with their correct labels, as well as misclassified samples and their correct labels. Classify the types of misclassified samples into categories, including misclassified samples and missed samples.

[0046] Based on the correct and incorrect samples obtained, an incremental learning dataset is constructed. The dataset includes all incorrectly recognized samples and one-third of correctly recognized samples. The unused visual recognition model is fine-tuned using the online gradient descent method to adapt to the new data. The algorithm formula is:

[0047] θ t+1 =θ t -η·▽J(θ t ,x t ,y t );

[0048] Among them, θ t represents the model parameters at the current time step t, θ t+1 The updated model parameters are obtained by adjusting the current model parameters using new data. η represents the learning rate, ▽J(θ t ,x t ,y t ) represents the loss function J on the model parameters θ t gradient.

[0049] Furthermore, the model optimization module is also provided with a self-updating optimization system, which establishes a specific database based on samples during each use of the visual recognition model. The specific steps are as follows:

[0050] Collect the error samples obtained in each optimal recognition model recognition process, establish a specific database, and based on the type of samples, establish a misclassified sample database and a missed sample database in the specific database;

[0051] Regularly conduct secondary specificity training for the visual recognition model added to the polymorphic model management module, including misclassification specificity training and missed detection specificity training. After each optimal visual model recognition, collect new error samples and label the samples. Add the newly collected error samples to the specificity database according to classification. Set up a cleaning mechanism, regularly check the data status in the specificity database, and remove samples before N time points.

[0052] Furthermore, the preprocessing module implements a fast data preprocessing algorithm and feature extraction method at the edge, including the compression algorithm JPEG compression to reduce the volume of image data, reduce the amount of data transmission and improve the processing speed, and removes noise in the data to be transmitted through median filtering to improve data quality.

[0053] Furthermore, the pre-processing module also includes a resource optimization scheduling algorithm to dynamically allocate computing and storage resources of edge nodes. The specific steps are as follows:

[0054] Collect the resource usage of relevant preprocessing tasks during the preprocessing process, including CPU and GPU core usage and memory capacity, classify the urgency of the preprocessing tasks, including deadlines and task priorities, and calculate the priority scores λ of different tasks:

[0055] For the deadline t, based on the loose time threshold t1 and the urgent time threshold t2, the deadline score value is set based on the threshold:

[0056] When t>t1, it means the related task is not urgent, and the score is set to 3 points;

[0057] When t2<t<t1, it means the related task is generally urgent, and the score is set to 6 points;

[0058] When t≤t2, it means the related task is very urgent and the score is set to 10 points;

[0059] For task priority, it is directly set by the user, with a score of 1 to 10;

[0060] λ=ω1·T+ω2·Y;

[0061] Where λ represents the priority score of the task, T and Y represent the deadline score and task priority score of the relevant task respectively, ω1 and ω2 represent the weight ratio of the two, and the sum of ω1 and ω2 is 1;

[0062] According to the priority scores λ of different preprocessing tasks, the tasks are processed in descending order.

[0063] Furthermore, heterogeneous computing acceleration modules, including GPU and TPU accelerators, are also provided in the cloud and edge to optimize the model inference process. The specific steps are as follows:

[0064] Set up a hardware abstraction layer to uniformly manage different types of accelerators, hide the complexity of specific hardware, and provide a unified interface. Install drivers for related hardware and ensure system and application compatibility. Convert the model into a format suitable for the target accelerator, adjust the batch size or data parallelism to achieve optimal performance of the accelerator, integrate the acceleration module with the existing computing framework, and use different accelerators for calculations.

[0065] An application system includes an edge server, a cloud server, and a cloud computing platform.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] 1. In the present invention, by setting up a polymorphic model management module, based on the computing power of the cloud, all visual recognition models under its jurisdiction are divided according to the accuracy, complexity, and load requirements of the current task. Through an intelligent decision-making mechanism, the most suitable model is selected according to the actual situation to perform visual recognition tasks on the edge. The most suitable model is selected according to the accuracy requirements of the current task, ensuring that the visual recognition task can be completed with the highest accuracy, reducing misidentification and missed identification. According to the complexity and load requirements of the task, the appropriate model is selected, which can efficiently use computing resources and avoid overuse of highly complex models or oversimplified models.

[0068] 2. In this invention, by comprehensively analyzing the accuracy, complexity, and load requirements of the current task, and based on the decision-making effect of the intelligent decision-making module, the complexity and load of the selected model are reduced while meeting the task requirements, which can improve the processing efficiency of the system and reduce unnecessary resource waste. Through different priority modes, the most suitable model can be flexibly selected according to the actual situation, thereby enhancing functionality;

[0069] 3. In the present invention, by setting up a model optimization module in conjunction with a polymorphic model management module, samples of the selected optimal model are collected during the visual recognition process. For unselected models, the powerful computing power of the cloud server is used to incrementally update the existing visual recognition model. This fully utilizes the computing resources of the cloud, avoids overloading edge devices, and enables all models to be updated based on real-time sample data, thereby enhancing practicality.

[0070] 4. In this invention, a specificity database is established by combining the error sample database in the recognition process. The cloud-based visual recognition model is regularly updated with specificity, including misclassified samples and missed samples, so that the model is more adaptable to special situations in actual applications and optimizes performance and system robustness.

[0071] The entire edge cloud-based visual recognition algorithm polymorphic model system and its application system can realize model management, model screening, and model optimization, significantly improving the response speed and resource utilization efficiency of the visual recognition system. It is suitable for various application scenarios with high requirements for real-time, accuracy, and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is a block diagram of a visual recognition algorithm polymorphic model system based on edge cloud and its application system of the present invention;

[0073] Figure 2 This is a schematic diagram of an edge cloud-based visual recognition algorithm polymorphic model system and its application polymorphic model management module of the present invention. DETAILED DESCRIPTION

[0074] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0075] Example 1:

[0076] like Figure 1-Figure 2 As shown, a visual recognition algorithm polymorphic model system based on edge cloud and its application system include a cloud end and an edge end. The cloud end includes a polymorphic model management module and a model optimization module, and the edge end includes a preprocessing module, wherein:

[0077] The polymorphic model management module includes n related visual recognition models. Based on the computational accuracy, complexity, and load of different visual recognition models, it automatically matches the most suitable model for the current task through an intelligent decision-making mechanism. The following steps are included:

[0078] Based on the computational accuracy, complexity, and load of different visual recognition models, the visual recognition models within the module are classified, and specific classification is performed based on the computational accuracy, complexity, and load of the model. The specific steps are as follows:

[0079] For the comprehensive visual recognition model, the computational accuracy of the model is specifically classified according to the model's F1 score. The F1 score is an indicator that comprehensively measures the model's precision and recall rate. The specific steps are as follows:

[0080] When the F1 score of the relevant model is less than 0.7, the relevant model is marked as a low-precision model;

[0081] When the F1 score of the relevant model is 0.7≤≤0.9, the relevant model is marked as a medium-precision model;

[0082] When the F1 score of the relevant model is greater than 0.9, the relevant model is marked as a high-precision model;

[0083] Based on the number of floating-point operations (FLOPs) of the model, the complexity of the model is specifically classified. The specific steps are as follows:

[0084] When the FLOPs of the relevant model is less than 10 8 , then the relevant model is marked as a low-complexity model;

[0085] When the 10 8 ≤FLOPs≤10 9 , then the relevant model is marked as a medium complexity model;

[0086] When the FLOPs of the relevant model is greater than 10 9 , then the relevant model is marked as a high complexity model;

[0087] Based on the model's CPU usage, the model load is specifically classified. The specific steps are as follows:

[0088] When the CPU usage of the relevant model is less than 30%, the relevant model is marked as a low-load model;

[0089] When 30% of the relevant model is ≤CPU usage ≤70%, the relevant model is marked as a medium load model;

[0090] When the CPU usage of a related model is greater than 70%, the related model is marked as a high-load model.

[0091] It should be noted that the F1 score is an indicator that comprehensively measures the precision and recall of the model. Its calculation formula is:

[0092]

[0093] Among them, P represents the precision of the model, R represents the recall rate of the model, and the F1 score ranges from 0 to 1, where 1 represents perfect precision and recall rate, and 0 represents the worst precision and recall rate. Through the F1 score, the precision and recall rate of related models can be comprehensively considered to avoid the one-sidedness that occurs when relying solely on precision or recall rate to evaluate the model.

[0094] Establish an intelligent decision-making matching model, and automatically match the most suitable model for the current task based on the priority of the current visual recognition task. The specific steps are as follows:

[0095] Establish an intelligent decision-making matching priority mode, including accuracy priority, complexity priority, load priority, and mixed priority, set different levels of model scores, and divide the judgment conditions into three equal parts based on the judgment conditions for different levels of models:

[0096] Among them, the scores of low-precision models are 2 points, 3 points, and 4 points respectively, the scores of medium-precision models are 5 points, 6 points, and 7 points respectively, the scores of high-precision models are 8 points, 9 points, and 10 points respectively, the scores of low-complexity models are 8 points, 9 points, and 10 points respectively, the scores of medium-complexity models are 5 points, 6 points, and 7 points respectively, the scores of high-complexity models are 2 points, 3 points, and 4 points respectively, the scores of low-load models are 8 points, 9 points, and 10 points respectively, the scores of medium-load models are 5 points, 6 points, and 7 points respectively, and the scores of high-load models are 2 points, 3 points, and 4 points respectively;

[0097] In the accuracy priority mode, based on the accuracy requirements of the current task, the models that meet the accuracy requirements of the current task are retained, and the first residual comprehensive score Z1 of each model in the retained models is calculated separately, where:

[0098] Z1=B+C;

[0099] In the above formula, B represents the model complexity score, C represents the model load score, the first remaining comprehensive score Z1 is sorted in descending order, and the first model is selected as the most suitable model. When the first remaining comprehensive score Z1 is the same, the model load score C of the models with the same score is sorted in descending order, and the first model is selected as the most suitable model. When the model load score C is the same, the model complexity score B of the models with the same score is sorted in descending order, and the first model is selected as the most suitable model.

[0100] In the complexity priority mode, based on the complexity requirements of the current task, the models that meet the complexity requirements of the current task are retained, and the second residual comprehensive score Z2 of each model in the retained models is calculated separately, where:

[0101] Z2=A+C;

[0102] In the above formula, A represents the model accuracy score, C represents the model load score, the second residual comprehensive score Z2 is sorted in descending order, and the first model is selected as the most suitable model. When the second residual comprehensive score Z2 is the same, the model accuracy scores A of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model load scores C of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model.

[0103] In the load priority mode, based on the load requirements of the current task, the models that meet the load requirements of the current task are retained, and the third residual comprehensive score Z3 of each model in the retained models is calculated respectively, where:

[0104] Z3=A+B;

[0105] In the above formula, A represents the model accuracy score, B represents the model complexity score, the third remaining comprehensive score Z3 is sorted in descending order, and the first model is selected as the most suitable model. When the third remaining comprehensive score Z3 is the same, the model accuracy scores A of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model complexity scores B of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model.

[0106] In the hybrid priority mode, when the current task has accuracy requirements, complexity requirements, and load requirements, the model that meets the current task requirements is retained, and the comprehensive score Z4 is calculated for the retained model, where:

[0107] Z4=A+B+C;

[0108] Arrange the comprehensive score Z4 in descending order and select the first model as the most suitable model;

[0109] When the current task meets any two of the accuracy requirements, complexity requirements, and load requirements, among the models that meet the requirements, the remaining requirement scores are sorted in descending order, and the first model is selected as the most suitable model.

[0110] It should be noted that, taking the low-load model as an example, when the CPU usage of the relevant model is less than 30%, the relevant model is marked as a low-load model, that is, the relevant model with a CPU usage of ≤10% is scored as 10 points, the relevant model with a CPU usage of 10% < ≤20% is scored as 9 points, and the relevant model with a CPU usage of 20% < <30% is scored as 9 points. Based on the accuracy requirements, complexity requirements and load requirements of the current task, the visual recognition model required for the current task is matched through the intelligent decision-making matching model, and the most suitable model is matched first to improve the response speed of the visual recognition system.

[0111] Example 2:

[0112] The model optimization module collects samples of the visual recognition models in use, and based on the correct and incorrect samples obtained, incrementally updates the unused models, continuously improving the visual recognition models. The specific steps are as follows:

[0113] Based on the selected optimal visual recognition model, relevant parameters during use are recorded, including correct samples and error samples. The unused visual recognition models are updated and optimized in the cloud using the correct and error samples. The specific steps are as follows:

[0114] Collect samples that are correctly identified by the visual recognition model in use, along with their correct labels, as well as misclassified samples and their correct labels. Classify the types of misclassified samples into categories, including misclassified samples and missed samples.

[0115] Based on the correct and incorrect samples obtained, an incremental learning dataset is constructed. The dataset includes all incorrectly recognized samples and one-third of correctly recognized samples. The unused visual recognition model is fine-tuned using the online gradient descent method to adapt to the new data. The algorithm formula is:

[0116] θ t+1 =θ t -η·▽J(θ t ,x t ,y t );

[0117] Among them, θ t represents the model parameters at the current time step t, θ t+1 The updated model parameters are obtained by adjusting the current model parameters using new data. η represents the learning rate, ▽J(θ t ,x t ,y t ) represents the loss function J on the model parameters θ t gradient.

[0118] It should be noted that misclassified samples are samples that the visual recognition model classifies into the wrong category, and missed samples are samples that the visual recognition model fails to identify that should be identified. η represents the learning rate, which controls the step size of each parameter update. A larger learning rate can easily lead to unstable training, while a smaller learning rate may lead to slow convergence. You can consult experts in related fields for flexible settings. Through the gradient descent algorithm, the model parameters are adjusted according to the calculated gradient to minimize the loss function, thereby improving the performance of the model on new data.

[0119] The model optimization module also has a self-updating optimization system that builds a specific database based on samples from each use of the visual recognition model. The specific steps are as follows:

[0120] Collect the error samples obtained in each optimal recognition model recognition process, establish a specific database, and based on the type of samples, establish a misclassified sample database and a missed sample database in the specific database;

[0121] Regularly conduct secondary specificity training for the visual recognition model added to the polymorphic model management module, including misclassification specificity training and missed detection specificity training. After each optimal visual model recognition, collect new error samples and label the samples. Add the newly collected error samples to the specificity database according to classification. Set up a cleaning mechanism, regularly check the data status in the specificity database, and remove samples before N time points.

[0122] It should be noted that the visual recognition model added to the polymorphic model management module is regularly trained for secondary specificity. The normal time period is 15 days, which can also be flexibly set according to the specific use environment. At the same time, samples before time point N are removed. Time point N is usually set to 6 months, and can also be adjusted according to the specific use environment requirements.

[0123] Example 3:

[0124] The preprocessing module performs data preprocessing and feature extraction at the edge through data preprocessing algorithms and feature extraction, reducing the amount of data transmitted to the cloud and improving processing speed;

[0125] The preprocessing module implements fast data preprocessing algorithms and feature extraction methods at the edge, including JPEG compression to reduce the size of image data, reduce data transmission volume and increase processing speed, and median filtering to remove noise from the data to be transmitted to improve data quality;

[0126] The preprocessing module also includes a resource optimization scheduling algorithm to dynamically allocate computing and storage resources of edge nodes. The specific steps are as follows:

[0127] Collect the resource usage of relevant preprocessing tasks during the preprocessing process, including CPU and GPU core usage and memory capacity, classify the urgency of the preprocessing tasks, including deadlines and task priorities, and calculate the priority scores λ of different tasks:

[0128] For the deadline t, based on the loose time threshold t1 and the urgent time threshold t2, the deadline score value is set based on the threshold:

[0129] When t>t1, it means the related task is not urgent, and the score is set to 3 points;

[0130] When t2<t<t1, it means the related task is generally urgent, and the score is set to 6 points;

[0131] When t≤t2, it means the related task is very urgent and the score is set to 10 points;

[0132] For task priority, it is directly set by the user, with a score of 1 to 10;

[0133] λ=ω1·T+ω2·Y;

[0134] Where λ represents the priority score of the task, T and Y represent the deadline score and task priority score of the relevant task respectively, ω1 and ω2 represent the weight ratio of the two, and the sum of ω1 and ω2 is 1;

[0135] According to the priority scores λ of different preprocessing tasks, the tasks are processed in descending order.

[0136] It should be noted that ω1 and ω2 represent the weight ratio of the two, which are generally set to ω1 = 0.65 and ω2 = 0.35. The values ​​of ω1 and ω2 can also be adjusted according to actual usage. By adjusting different ω1 and ω2 values, the task priority in different situations can be flexibly adjusted.

[0137] Heterogeneous computing acceleration modules, including GPU and TPU accelerators, are also set up in the cloud and edge to optimize the model inference process. The specific steps are as follows:

[0138] Set up a hardware abstraction layer to uniformly manage different types of accelerators, hide the complexity of specific hardware, and provide a unified interface. Install drivers for related hardware and ensure system and application compatibility. Convert the model into a format suitable for the target accelerator, adjust the batch size or data parallelism to achieve optimal performance of the accelerator, integrate the acceleration module with the existing computing framework, and use different accelerators for calculations.

[0139] It should be noted that the GPU accelerator is an external GPU that exists in the form of an independent graphics card. It is connected to the PCIe slot on the edge and cloud computer motherboards to perform auxiliary accelerated calculations. TPU acceleration refers to the use of Tensor Processing Unit for computing acceleration.

[0140] The present invention provides an edge cloud-based visual recognition algorithm polymorphic model system and its application system. When in use, the cloud-based task requirements based on the edge end are used to screen suitable visual recognition models to participate in the visual recognition task, and at the same time, samples in the recognition process are collected. By pre-processing the data and quickly transmitting it back to the cloud, all visual recognition models are incrementally updated, and the system's response, recognition, and training are accelerated based on GPU and TPU to improve the response speed and resource utilization efficiency of the visual recognition system.

[0141] In the embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and other division methods may be used in actual implementation. The modules described as separate components may or may not be physically separate, and the components displayed as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the method of this embodiment.

[0142] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A multi-state model system for visual recognition algorithms based on edge cloud, characterized by: It includes the cloud and edge. The cloud includes the polymorphic model management module and the model optimization module, and the edge includes the preprocessing module, including: The polymorphic model management module includes n related visual recognition models. Based on the computational accuracy, complexity, and load of different visual recognition models, it automatically matches the most suitable model for the current task through an intelligent decision-making mechanism: Based on the computational accuracy, complexity, and load of different visual recognition models, the visual recognition models within the module are classified, and specific classification is performed based on the computational accuracy, complexity, and load of the model. The specific steps are as follows: For the comprehensive visual recognition model, the computational accuracy of the model is specifically classified according to the model's F1 score. The F1 score is an indicator that comprehensively measures the model's precision and recall rate. The specific steps are as follows: When the F1 score of the relevant model is less than 0.7, the relevant model is marked as a low-precision model; When the F1 score of the relevant model is 0.7≤≤0.9, the relevant model is marked as a medium-precision model; When the F1 score of the relevant model is greater than 0.9, the relevant model is marked as a high-precision model; Based on the number of floating-point operations (FLOPs) of the model, the complexity of the model is specifically classified. The specific steps are as follows: When the FLOPs of the relevant model is less than 10 8 , then the relevant model is marked as a low-complexity model; When the 10 8 ≤FLOPs≤10 9 , then the relevant model is marked as a medium complexity model; When the FLOPs of the relevant model is greater than 10 9 , then the relevant model is marked as a high complexity model; Based on the model's CPU usage, the model load is specifically classified. The specific steps are as follows: When the CPU usage of the relevant model is less than 30%, the relevant model is marked as a low-load model; When 30% of the relevant model is ≤CPU usage ≤70%, the relevant model is marked as a medium load model; When the CPU usage of the relevant model is greater than 70%, the relevant model is marked as a high-load model; Among them, the F1 score is an indicator that comprehensively measures the precision and recall rate of the model. Its calculation formula is: Among them, P represents the precision of the model, R represents the recall of the model, and the F1 score ranges from 0 to 1, where 1 represents perfect precision and recall, and 0 represents the worst precision and recall. The F1 score can be used to comprehensively consider the precision and recall of related models, avoiding the one-sidedness that occurs when relying solely on precision or recall to evaluate the model; Establish an intelligent decision-making matching model, including accuracy priority, complexity priority, load priority, and mixed priority. Based on the priority of the current visual recognition task, automatically match the most suitable model for the current task. The specific steps are as follows: Establish an intelligent decision-making matching priority mode, including accuracy priority, complexity priority, load priority, and mixed priority, set different levels of model scores, and divide the judgment conditions into three equal parts based on the judgment conditions for different levels of models: Among them, the scores of low-precision models are 2 points, 3 points, and 4 points respectively, the scores of medium-precision models are 5 points, 6 points, and 7 points respectively, the scores of high-precision models are 8 points, 9 points, and 10 points respectively, the scores of low-complexity models are 8 points, 9 points, and 10 points respectively, the scores of medium-complexity models are 5 points, 6 points, and 7 points respectively, the scores of high-complexity models are 2 points, 3 points, and 4 points respectively, the scores of low-load models are 8 points, 9 points, and 10 points respectively, the scores of medium-load models are 5 points, 6 points, and 7 points respectively, and the scores of high-load models are 2 points, 3 points, and 4 points respectively; In the accuracy priority mode, based on the accuracy requirements of the current task, the models that meet the accuracy requirements of the current task are retained, and the first residual comprehensive score Z1 of each model in the retained models is calculated separately, where: Z1=B+C; In the above formula, B represents the model complexity score, C represents the model load score, the first remaining comprehensive score Z1 is sorted in descending order, and the first model is selected as the most suitable model. When the first remaining comprehensive score Z1 is the same, the model load score C of the models with the same score is sorted in descending order, and the first model is selected as the most suitable model. When the model load score C is the same, the model complexity score B of the models with the same score is sorted in descending order, and the first model is selected as the most suitable model. In the complexity priority mode, based on the complexity requirements of the current task, the models that meet the complexity requirements of the current task are retained, and the second residual comprehensive score Z2 of each model in the retained models is calculated separately, where: Z2=A+C; In the above formula, A represents the model accuracy score, C represents the model load score, the second residual comprehensive score Z2 is sorted in descending order, and the first model is selected as the most suitable model. When the second residual comprehensive score Z2 is the same, the model accuracy scores A of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model load scores C of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model. In the load priority mode, based on the load requirements of the current task, the models that meet the load requirements of the current task are retained, and the third residual comprehensive score Z3 of each model in the retained models is calculated respectively, where: Z3=A+B; In the above formula, A represents the model accuracy score, B represents the model complexity score, the third remaining comprehensive score Z3 is sorted in descending order, and the first model is selected as the most suitable model. When the third remaining comprehensive score Z3 is the same, the model accuracy scores A of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model complexity scores B of the models with the same score are sorted in descending order, and the first model is selected as the most suitable model. In the hybrid priority mode, when the current task has accuracy requirements, complexity requirements, and load requirements, the model that meets the current task requirements is retained, and the comprehensive score Z4 is calculated for the retained model, where: Z4=A+B+C; Arrange the comprehensive score Z4 in descending order and select the first model as the most suitable model; When the current task meets any two of the requirements of accuracy, complexity, and load, among the models that meet the requirements, the remaining requirement scores are sorted in descending order, and the first model is selected as the most suitable model; Taking the low-load model as an example, when the CPU usage of the relevant model is less than 30%, the relevant model is marked as a low-load model. That is, the relevant model CPU usage ≤ 10% is 10 points, the relevant model 10% < CPU usage ≤ 20% is 9 points, and the relevant model 20% < CPU usage < 30% is 8 points. Based on the accuracy requirements, complexity requirements, and load requirements of the current task, the intelligent decision-making matching model is used to match the visual recognition model required for the current task, and the most suitable model is matched first to improve the response speed of the visual recognition system; The model optimization module collects samples of the visual recognition models in use, and based on the correct and incorrect samples obtained, incrementally updates the unused models, thereby continuously improving the visual recognition models: Based on the selected optimal visual recognition model, relevant parameters during use are recorded, including correct samples and error samples. Unused visual recognition models are updated and optimized in the cloud using correct and error samples: Collect samples that are correctly identified by the visual recognition model in use, along with their correct labels, as well as misclassified samples and their correct labels. Classify the types of misclassified samples into categories, including misclassified samples and missed samples. Based on the correct and incorrect samples obtained, an incremental learning dataset is constructed. The dataset includes all incorrectly recognized samples and one-third of correctly recognized samples. Using the online gradient descent method, the unused visual recognition model is fine-tuned to adapt to the new data: Among them, θ t represents the model parameters at the current time step t, θ t+1 That is, the updated model parameters are obtained by adjusting the current model parameters using new data. η represents the learning rate. Represents the loss function J on the model parameters θ t gradient; Misclassified samples are samples that the visual recognition model classifies into the wrong category, and missed samples are samples that the visual recognition model fails to recognize. η represents the learning rate, which controls the step size of each parameter update. A larger learning rate can easily lead to unstable training, while a smaller learning rate may lead to slow convergence. Flexible settings are made by consulting experts in related fields. Using the gradient descent algorithm, model parameters are adjusted according to the calculated gradient to minimize the loss function, thereby improving the model's performance on new data. The model optimization module is also provided with a self-updating optimization system, which establishes a specific database based on samples during each use of the visual recognition model. The specific steps are as follows: Collect the error samples obtained in each optimal recognition model recognition process, establish a specific database, and based on the type of samples, establish a misclassified sample database and a missed sample database in the specific database; Regularly conduct secondary specificity training for the visual recognition model added to the polymorphic model management module, including misclassification specificity training and missed detection specificity training. After each optimal visual model recognition, collect new error samples and label them. Add the newly collected error samples to the specificity database according to classification. Set up a cleanup mechanism to regularly check the data in the specificity database and remove samples before time point N. Regularly conduct secondary specific training for the visual recognition model added to the polymorphic model management module. The typical time period is 15 days, which can be flexibly set according to the specific usage environment. At the same time, remove samples before time point N. Time point N is usually set to 6 months, but can also be adjusted according to the specific usage environment requirements. The preprocessing module performs data preprocessing and feature extraction at the edge through data preprocessing algorithms and feature extraction, thereby reducing the amount of data transmitted to the cloud and improving the processing speed.

2. The edge cloud-based visual recognition algorithm polymorphic model system according to claim 1, characterized in that: The preprocessing module implements a fast data preprocessing algorithm and feature extraction method at the edge, including the compression algorithm JPEG compression to reduce the volume of image data, reduce the amount of data transmission and improve the processing speed, and removes noise in the data to be transmitted through median filtering to improve data quality.

3. The edge cloud-based visual recognition algorithm polymorphic model system according to claim 2, characterized in that: The pre-processing module also includes a resource optimization scheduling algorithm to dynamically allocate computing and storage resources of edge nodes. The specific steps are as follows: Collect the resource usage of relevant preprocessing tasks during the preprocessing process, including CPU and GPU core usage and memory capacity, classify the urgency of the preprocessing tasks, including deadlines and task priorities, and calculate the priority scores λ of different tasks: For the deadline t, based on the loose time threshold t1 and the urgent time threshold t2, the deadline score value is set based on the threshold: When t>t1, it means the related task is not urgent, and the score is set to 3 points; When t2<t<t1, it means the related task is generally urgent, and the score is set to 6 points; When t≤t2, it means the related task is very urgent and the score is set to 10 points; For task priority, it is directly set by the user, with a score of 1 to 10; λ=ω1·T+ω2·Y; Where λ represents the priority score of the task, T and Y represent the deadline score and task priority score of the relevant task respectively, ω1 and ω2 represent the weight ratio of the two, and the sum of ω1 and ω2 is 1; According to the priority scores λ of different preprocessing tasks, the tasks are processed in descending order.

4. The edge cloud-based visual recognition algorithm polymorphic model system according to claim 1, characterized in that: The cloud and edge are also equipped with heterogeneous computing acceleration modules, including GPU and TPU accelerators, to optimize the model inference process. The specific steps are as follows: Set up a hardware abstraction layer to uniformly manage different types of accelerators, hide the complexity of specific hardware, and provide a unified interface. Install drivers for related hardware and ensure system and application compatibility. Convert the model into a format suitable for the target accelerator, adjust the batch size or data parallelism to achieve optimal performance of the accelerator, integrate the acceleration module with the existing computing framework, and use different accelerators for calculations.

5. An application system, comprising an edge server, a cloud server, and a cloud computing platform, for implementing the edge cloud-based visual recognition algorithm polymorphic model system described in any one of claims 1-4.

Images