Visual identification algorithm polymorphic model microtechnology based on edge cloud and application system thereof

Abstract

The invention discloses a visual recognition algorithm polymorphic model microtechnology based on edge cloud and an application system thereof, and belongs to the technical field of computer vision, the visual recognition algorithm polymorphic model microtechnology comprises a cloud end and an edge end, the cloud end comprises a polymorphic model management module and a model optimization module, the edge end comprises a preprocessing module, and the polymorphic model management module is connected with the model optimization module. Comprising n related visual identification models, an optimal model is automatically matched for a current task through an intelligent decision-making mechanism based on calculation precision, complexity and load of different visual identification models, and the model optimization module collects visual identification model samples in use. According to the visual identification algorithm polymorphic model microtechnology based on the edge cloud and the application system thereof, model management, model screening, model optimization and resource optimization can be realized, the response speed and the resource utilization efficiency of a visual identification system are remarkably improved, and the visual identification algorithm polymorphic model microtechnology is suitable for various application scenes with high requirements on real-time performance, accuracy and safety.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically is a visual recognition algorithm polymorphic model micro-technology based on edge cloud and its application system. Background Art

[0002] The visual recognition model is a model based on computer vision technology, which aims to enable computers to recognize and understand the content in images or videos. It simulates the working mode of the human visual system and uses image processing and pattern recognition technology to recognize objects or scenes in images.

[0003] Traditional visual recognition systems often rely on centralized cloud computing, resulting in high latency and inflexible resource allocation. With the development of edge computing, migrating visual recognition tasks to edge nodes has become a trend, but it still faces problems such as poor model adaptability and insufficient resource optimization allocation. It is impossible to flexibly screen visual recognition models according to the needs of the actual use environment, lacks certain functionality, and has low practicality.

[0004] In response to the above, this case proposes a visual recognition algorithm polymorphic model micro-technology based on edge cloud and its application system to solve the above technical problems. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a visual recognition algorithm polymorphic model micro-technology based on edge cloud and its application system, which solves the above technical problems by improving the detection method and processing method.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A polymorphic model micro-technology of visual recognition algorithm based on edge cloud includes a cloud end and an edge end, wherein the cloud end includes a polymorphic model management module and a model optimization module, and the edge end includes a preprocessing module, wherein:

[0008] The polymorphic model management module includes n related visual recognition models, and automatically matches the most suitable model for the current task through an intelligent decision-making mechanism based on the calculation accuracy, complexity and load of different visual recognition models;

[0009] The model optimization module collects samples of the visual recognition models in use, and based on the correct and incorrect samples obtained, incrementally updates the unused models, thereby continuously improving the visual recognition models;

[0010] The preprocessing module performs data preprocessing and feature extraction at the edge through data preprocessing algorithms and feature extraction, thereby reducing the amount of data transmitted to the cloud and improving the processing speed.

[0011] Furthermore, the polymorphic model management module includes n related visual recognition models, and based on the calculation accuracy, complexity and load of different visual recognition models, automatically matches the most suitable model for the current task through an intelligent decision-making mechanism, including the following steps:

[0012] Based on the computational accuracy, complexity and load of different visual recognition models, the visual recognition models within the module are classified, and based on the computational accuracy, complexity and load of the model, specific classification is performed respectively;

[0013] Establish an intelligent decision-making matching model to automatically match the most suitable model for the current task based on the priority of the current visual recognition task.

[0014] Furthermore, based on the calculation accuracy, complexity and load of different visual recognition models, the visual recognition models within the module are classified, and based on the calculation accuracy, complexity and load of the model, specific classification is performed respectively, and the specific steps are as follows:

[0015] For the comprehensive visual recognition model, the computational accuracy of the model is specifically classified according to the F1 score of the model. The F1 score is an indicator that comprehensively measures the precision and recall rate of the model. The specific steps are as follows:

[0016] When the F1 score of the relevant model is less than 0.7, the relevant model is marked as a low-precision model;

[0017] When the F1 score of the relevant model is 0.7≤≤0.9, the relevant model is marked as a medium-precision model;

[0018] When the F1 score of the relevant model is greater than 0.9, the relevant model is marked as a high-precision model;

[0019] According to the number of floating point operations FLOPs of the model, the complexity of the model is specifically classified. The specific steps are:

[0020] When the FLOPs of the relevant model is less than 10 8 , then the relevant model is marked as a low-complexity model;

[0021] When the relevant model is 10 8 ≤FLOPs≤10 9 , then the relevant model is marked as a medium complexity model;

[0022] When the FLOPs of the relevant model is greater than 10 9 , then the relevant model is marked as a high-complexity model;

[0023] Based on the CPU usage of the model, the model load is specifically classified. The specific steps are as follows:

[0024] When the CPU usage of the relevant model is less than 30%, the relevant model is marked as a low-load model;

[0025] When 30% of the relevant model ≤ CPU usage ≤ 70%, the relevant model is marked as a medium load model;

[0026] When the CPU usage of the relevant model is greater than 70%, the relevant model is marked as a high-load model.

[0027] Furthermore, the intelligent decision-making matching model is established to automatically match the most suitable model for the current task based on the priority of the current visual recognition task. The specific steps are as follows:

[0028] Establish an intelligent decision-making matching priority mode, including accuracy priority, complexity priority, load priority and mixed priority, set different levels of model scores, and divide the judgment conditions into three equal parts for different levels of models based on the judgment conditions:

[0029] Among them, the scores of low-precision models are 2 points, 3 points, and 4 points respectively, the scores of medium-precision models are 5 points, 6 points, and 7 points respectively, the scores of high-precision models are 8 points, 9 points, and 10 points respectively, the scores of low-complexity models are 8 points, 9 points, and 10 points respectively, the scores of medium-complexity models are 5 points, 6 points, and 7 points respectively, the scores of high-complexity models are 2 points, 3 points, and 4 points respectively, the scores of low-load models are 8 points, 9 points, and 10 points respectively, the scores of medium-load models are 5 points, 6 points, and 7 points respectively, and the scores of high-load models are 2 points, 3 points, and 4 points respectively;

[0030] In the accuracy priority mode, based on the accuracy requirement of the current task, the models that meet the accuracy requirement of the current task are retained, and the first residual comprehensive score Z1 of each model in the retained model is calculated respectively, where:

[0031] Z1=B+C;

[0032] In the above formula, B represents the model complexity score, C represents the model load score, the first remaining comprehensive score Z1 is arranged in descending order, and the first model is selected as the most suitable model. When the first remaining comprehensive score Z1 is the same, the model load score C of the models with the same score is arranged in descending order, and the first model is selected as the most suitable model. When the model load score C is the same, the model complexity score B of the models with the same score is arranged in descending order, and the first model is selected as the most suitable model.

[0033] In the complexity priority mode, based on the complexity requirements of the current task, the models that meet the complexity requirements of the current task are retained, and the second residual comprehensive score Z2 of each model in the retained model is calculated respectively, where:

[0034] Z2=A+C;

[0035] In the above formula, A represents the model accuracy score, C represents the model load score, the second remaining comprehensive score Z2 is arranged in descending order, and the first model is selected as the most suitable model. When the second remaining comprehensive score Z2 is the same, the model accuracy scores A of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model load scores C of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model.

[0036] In the load priority mode, based on the load requirements of the current task, the models that meet the load requirements of the current task are retained, and the third residual comprehensive score Z3 of each model in the retained model is calculated respectively, where:

[0037] Z3=A+B;

[0038] In the above formula, A represents the model accuracy score, B represents the model complexity score, the third remaining comprehensive score Z3 is arranged in descending order, and the first model is selected as the most suitable model. When the third remaining comprehensive score Z3 is the same, the model accuracy scores A of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model complexity scores B of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model.

[0039] In the mixed priority mode, when the current task has accuracy requirements, complexity requirements, and load requirements, the model that meets the current task requirements is retained, and the comprehensive score Z4 is calculated for the retained model, where:

[0040] Z4=A+B+C;

[0041] Arrange the comprehensive scores Z4 in descending order and select the first model as the most suitable model;

[0042] When the current task meets any two of the accuracy requirements, complexity requirements, and load requirements, among the models that meet the requirements, the remaining requirement scores are arranged in descending order, and the first model is selected as the most suitable model.

[0043] Furthermore, the model optimization module collects samples of visual recognition models in use, and based on the correct and incorrect samples obtained, incrementally updates unused models to ensure continuous improvement of the algorithm. The specific steps are as follows:

[0044] Based on the selected optimal visual recognition model, the relevant parameters in the use process are recorded, including correct samples and error samples. The unused visual recognition models are updated and optimized in the cloud through correct samples and error samples. The specific steps are as follows:

[0045] Collect samples that are correctly identified and their correct labels, samples that are incorrectly classified and their correct labels of the visual recognition model in use, and classify the types of incorrectly classified samples into categories, including misclassified samples and missed samples;

[0046] Based on the correct and incorrect samples obtained, an incremental learning data set is constructed. The data set includes all incorrectly recognized samples and one-third of correctly recognized samples. The unused visual recognition model is fine-tuned through the online gradient descent method to adapt to the new data. The algorithm formula is:

[0047] θ t+1 =θ t -η·▽J(θ t ,x t ,y t );

[0048] Among them, θ t represents the model parameters at the current time step t, θ t+1 is the updated model parameter, which is obtained by adjusting the current model parameters using the new data. η represents the learning rate, ▽J(θ t ,x t ,y t ) represents the loss function J on the model parameters θ t gradient.

[0049] Furthermore, the model optimization module is also provided with a self-updating optimization system, which establishes a specific database based on samples in each use of the visual recognition model, and the specific steps are as follows:

[0050] Collect the erroneous samples obtained in each optimal recognition model recognition process, establish a specific database, and based on the type of samples, establish a misclassified sample database and a missed sample database in the specific database;

[0051] Regularly conduct secondary specific training for the visual recognition model added to the polymorphic model management module, including misclassification specific training and missed detection specific training. After each optimal visual model identification, collect new error samples and label the samples. Add the newly collected error samples to the specific database according to classification. Set up a cleaning mechanism, regularly check the data in the specific database, and remove samples before N time points.

[0052] Furthermore, the preprocessing module implements a fast data preprocessing algorithm and a feature extraction method at the edge, including a compression algorithm JPEG compression to reduce the volume of image data, reduce the amount of data transmission and increase the processing speed, and removes noise in the data to be transmitted through median filtering to improve data quality.

[0053] Furthermore, the preprocessing module also includes a resource optimization scheduling algorithm to dynamically allocate computing and storage resources of edge nodes, and the specific steps are as follows:

[0054] Collect the resource usage of relevant preprocessing tasks during the preprocessing process, including CPU and GPU core usage and memory capacity, classify the urgency of preprocessing tasks, including deadline and task priority, and calculate the priority scores λ of different tasks:

[0055] For the deadline t, based on the loose time threshold t1 and the emergency time threshold t2, the deadline score value is set based on the threshold:

[0056] When t>t1, it means the related task is not urgent, and the score is set to 3 points;

[0057] When t2<t<t1, it means the relevant task is generally urgent, and the score is set to 6 points;

[0058] When t≤t2, it means that the relevant task is very urgent, and the score is set to 10 points;

[0059] For task priority, it is directly set by the user, with a score of 1 to 10;

[0060] λ=ω1·T+ω2·Y;

[0061] Among them, λ represents the priority score of the task, T and Y represent the deadline score and task priority score of the relevant task respectively, ω1 and ω2 represent the weight ratio of the two, and the sum of ω1 and ω2 is 1;

[0062] According to the priority scores λ of different preprocessing tasks, the tasks are processed in descending order.

[0063] Furthermore, the cloud and edge are also provided with heterogeneous computing acceleration modules, including GPU and TPU accelerators, to optimize the model reasoning process, and the specific steps are as follows:

[0064] Set up a hardware abstraction layer to uniformly manage different types of accelerators, hide the complexity of specific hardware, and provide a unified interface, install drivers for related hardware, and ensure system and application compatibility, convert the model into a format suitable for the target accelerator, adjust the batch size or data parallelism to achieve the best performance of the accelerator, integrate the acceleration module with the existing computing framework, and use different accelerators for calculations.

[0065] An application system includes an edge server, a cloud server, and a cloud computing platform.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] 1. In the present invention, by setting a polymorphic model management module, based on the computing power of the cloud, according to the accuracy, complexity, and load requirements of the current task, all the visual recognition models under its jurisdiction are divided, and through an intelligent decision-making mechanism, the most suitable model is selected according to the actual situation to perform the visual recognition task at the edge, and the most suitable model is selected according to the accuracy requirements of the current task to ensure that the visual recognition task can be completed with the highest accuracy, reduce the situation of misidentification and missed identification, and select a suitable model according to the complexity and load requirements of the task, so as to efficiently use computing resources and avoid excessive use of high-complexity models or over-simplified models;

[0068] 2. In the present invention, by comprehensively analyzing the accuracy, complexity and load requirements of the current task, based on the decision-making effect of the intelligent decision-making module, the complexity and load of the selected model are reduced while meeting the task requirements, which can improve the processing efficiency of the system and reduce unnecessary waste of resources. Through different priority modes, the most suitable model can be flexibly selected according to the actual situation, thereby enhancing functionality;

[0069] 3. In the present invention, by setting a model optimization module to cooperate with a polymorphic model management module, by collecting samples of the selected optimal model in the visual recognition process, the existing visual recognition model is incrementally updated through the strong computing power of the cloud server for the unselected model, and the computing resources of the cloud are fully utilized to avoid overloading the edge devices, so that all models can be updated according to real-time sample data, thereby enhancing practicality;

[0070] 4. In the present invention, a specificity database is established in combination with the error sample database in the recognition process, and the cloud-based visual recognition model is specifically updated regularly, including misclassified samples and missed samples, so that the model is more adaptable to special situations in practical applications and optimizes performance and system robustness;

[0071] The entire edge cloud-based visual recognition algorithm polymorphic model micro-technology and its application system can realize model management, model screening, and model optimization, significantly improving the response speed and resource utilization efficiency of the visual recognition system, and is suitable for various application scenarios with high requirements for real-time, accuracy, and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A block diagram of a polymorphic model microtechnology of a visual recognition algorithm based on edge cloud and its application system of the present invention;

[0073] Figure 2 This is a schematic diagram of a polymorphic model microtechnology of a visual recognition algorithm based on edge cloud and its application polymorphic model management module of the present invention. DETAILED DESCRIPTION

[0074] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Embodiment 1:

[0076] like Figure 1-Figure 2 As shown, a visual recognition algorithm polymorphic model micro-technology based on edge cloud and its application system include a cloud end and an edge end, the cloud end includes a polymorphic model management module and a model optimization module, and the edge end includes a preprocessing module, wherein:

[0077] The polymorphic model management module includes n related visual recognition models. Based on the calculation accuracy, complexity and load of different visual recognition models, the module automatically matches the most suitable model for the current task through an intelligent decision-making mechanism, including the following steps:

[0078] Based on the calculation accuracy, complexity and load of different visual recognition models, the visual recognition models within the module are classified, and based on the calculation accuracy, complexity and load of the model, specific classification is performed respectively. The specific steps are as follows:

[0079] For the comprehensive visual recognition model, the computational accuracy of the model is specifically classified according to the F1 score of the model. The F1 score is an indicator that comprehensively measures the precision and recall rate of the model. The specific steps are as follows:

[0080] When the F1 score of the relevant model is less than 0.7, the relevant model is marked as a low-precision model;

[0081] When the F1 score of the relevant model is 0.7≤≤0.9, the relevant model is marked as a medium-precision model;

[0082] When the F1 score of the relevant model is greater than 0.9, the relevant model is marked as a high-precision model;

[0083] According to the number of floating point operations FLOPs of the model, the complexity of the model is specifically classified. The specific steps are:

[0084] When the FLOPs of the relevant model is less than 10 8 , then the relevant model is marked as a low-complexity model;

[0085] When the relevant model is 10 8 ≤FLOPs≤10 9 , then the relevant model is marked as a medium complexity model;

[0086] When the FLOPs of the relevant model is greater than 10 9 , then the relevant model is marked as a high-complexity model;

[0087] Based on the CPU usage of the model, the model load is specifically classified. The specific steps are as follows:

[0088] When the CPU usage of the relevant model is less than 30%, the relevant model is marked as a low-load model;

[0089] When 30% of the relevant model ≤ CPU usage ≤ 70%, the relevant model is marked as a medium load model;

[0090] When the CPU usage of the relevant model is greater than 70%, the relevant model is marked as a high-load model.

[0091] It should be noted that the F1 score is an indicator that comprehensively measures the precision and recall of the model, and its calculation formula is:

[0092]

[0093] Among them, P represents the precision of the model, R represents the recall rate of the model, and the F1 score ranges from 0 to 1, where 1 represents perfect precision and recall rate, and 0 represents the worst precision and recall rate. Through the F1 score, the precision and recall rate of related models can be comprehensively considered to avoid the one-sidedness that occurs when relying solely on precision or recall rate to evaluate the model.

[0094] Establish an intelligent decision-making matching model, and automatically match the most suitable model for the current task based on the priority of the current visual recognition task. The specific steps are as follows:

[0095] Establish an intelligent decision-making matching priority mode, including accuracy priority, complexity priority, load priority and mixed priority, set different levels of model scores, and divide the judgment conditions into three equal parts for different levels of models based on the judgment conditions:

[0096] Among them, the scores of low-precision models are 2 points, 3 points, and 4 points respectively, the scores of medium-precision models are 5 points, 6 points, and 7 points respectively, the scores of high-precision models are 8 points, 9 points, and 10 points respectively, the scores of low-complexity models are 8 points, 9 points, and 10 points respectively, the scores of medium-complexity models are 5 points, 6 points, and 7 points respectively, the scores of high-complexity models are 2 points, 3 points, and 4 points respectively, the scores of low-load models are 8 points, 9 points, and 10 points respectively, the scores of medium-load models are 5 points, 6 points, and 7 points respectively, and the scores of high-load models are 2 points, 3 points, and 4 points respectively;

[0097] In the accuracy priority mode, based on the accuracy requirement of the current task, the models that meet the accuracy requirement of the current task are retained, and the first residual comprehensive score Z1 of each model in the retained model is calculated respectively, where:

[0098] Z1=B+C;

[0099] In the above formula, B represents the model complexity score, C represents the model load score, the first remaining comprehensive score Z1 is arranged in descending order, and the first model is selected as the most suitable model. When the first remaining comprehensive score Z1 is the same, the model load score C of the models with the same score is arranged in descending order, and the first model is selected as the most suitable model. When the model load score C is the same, the model complexity score B of the models with the same score is arranged in descending order, and the first model is selected as the most suitable model.

[0100] In the complexity priority mode, based on the complexity requirements of the current task, the models that meet the complexity requirements of the current task are retained, and the second residual comprehensive score Z2 of each model in the retained model is calculated respectively, where:

[0101] Z2=A+C;

[0102] In the above formula, A represents the model accuracy score, C represents the model load score, the second remaining comprehensive score Z2 is arranged in descending order, and the first model is selected as the most suitable model. When the second remaining comprehensive score Z2 is the same, the model accuracy scores A of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model load scores C of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model.

[0103] In the load priority mode, based on the load requirements of the current task, the models that meet the load requirements of the current task are retained, and the third residual comprehensive score Z3 of each model in the retained model is calculated respectively, where:

[0104] Z3=A+B;

[0105] In the above formula, A represents the model accuracy score, B represents the model complexity score, the third remaining comprehensive score Z3 is arranged in descending order, and the first model is selected as the most suitable model. When the third remaining comprehensive score Z3 is the same, the model accuracy scores A of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model complexity scores B of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model.

[0106] In the mixed priority mode, when the current task has accuracy requirements, complexity requirements, and load requirements, the model that meets the current task requirements is retained, and the comprehensive score Z4 is calculated for the retained model, where:

[0107] Z4=A+B+C;

[0108] Arrange the comprehensive scores Z4 in descending order and select the first model as the most suitable model;

[0109] When the current task meets any two of the accuracy requirements, complexity requirements, and load requirements, among the models that meet the requirements, the remaining requirement scores are arranged in descending order, and the first model is selected as the most suitable model.

[0110] It should be noted that, taking the low-load model as an example, when the CPU usage rate of the relevant model is less than 30%, the relevant model is marked as a low-load model, that is, the relevant model CPU usage rate ≤10% is 10 points, the relevant model 10%<CPU usage rate ≤20% is 9 points, and the relevant model 20%<CPU usage rate<30% is 9 points. Based on the accuracy requirements, complexity requirements and load requirements of the current task, the visual recognition model required for the current task is matched through the intelligent decision-making matching model, and the most suitable model is matched first to improve the response speed of the visual recognition system.

[0111] Embodiment 2:

[0112] The model optimization module collects samples of visual recognition models in use, and based on the correct and incorrect samples obtained, incrementally updates unused models and continuously improves the visual recognition models. The specific steps are as follows:

[0113] Based on the selected optimal visual recognition model, the relevant parameters in the use process are recorded, including correct samples and error samples. The unused visual recognition models are updated and optimized in the cloud through correct samples and error samples. The specific steps are as follows:

[0114] Collect samples that are correctly identified and their correct labels, samples that are incorrectly classified and their correct labels of the visual recognition model in use, and classify the types of incorrectly classified samples into categories, including misclassified samples and missed samples;

[0115] Based on the correct and incorrect samples obtained, an incremental learning data set is constructed. The data set includes all incorrectly recognized samples and one-third of correctly recognized samples. The unused visual recognition model is fine-tuned through the online gradient descent method to adapt to the new data. The algorithm formula is:

[0116] θ t+1 =θ t -η·▽J(θ t ,x t ,y t );

[0117] Among them, θ t represents the model parameters at the current time step t, θ t+1 is the updated model parameter, which is obtained by adjusting the current model parameters using the new data. η represents the learning rate, ▽J(θ t ,x t ,y t ) represents the loss function J on the model parameters θ t gradient.

[0118] It should be noted that misclassified samples are samples that the visual recognition model classifies into the wrong category, and missed samples are samples that the visual recognition model fails to identify that should have been identified. η represents the learning rate, which controls the step size of each parameter update. A larger learning rate is likely to lead to unstable training, while a smaller learning rate may lead to slow convergence. You can consult experts in related fields for flexible settings. Through the gradient descent algorithm, the model parameters are adjusted according to the calculated gradient to minimize the loss function, thereby improving the performance of the model on new data.

[0119] The model optimization module is also equipped with a self-updating optimization system, which builds a specific database based on samples from each use of the visual recognition model. The specific steps are as follows:

[0120] Collect the erroneous samples obtained in each optimal recognition model recognition process, establish a specific database, and based on the type of samples, establish a misclassified sample database and a missed sample database in the specific database;

[0121] Regularly conduct secondary specific training for the visual recognition model added to the polymorphic model management module, including misclassification specific training and missed detection specific training. After each optimal visual model identification, collect new error samples and label the samples. Add the newly collected error samples to the specific database according to classification. Set up a cleaning mechanism, regularly check the data in the specific database, and remove samples before N time points.

[0122] It should be noted that the visual recognition model added to the polymorphic model management module is regularly trained for secondary specificity. The conventional time period is 15 days, which can also be flexibly set according to the specific usage environment. At the same time, samples before time point N are removed. Time point N is usually set to 6 months, and time point N can also be adjusted according to the specific usage environment requirements.

[0123] Embodiment 3:

[0124] The preprocessing module performs data preprocessing and feature extraction at the edge through data preprocessing algorithms and feature extraction, reducing the amount of data transmitted to the cloud and improving processing speed;

[0125] The preprocessing module implements fast data preprocessing algorithms and feature extraction methods at the edge, including compression algorithms such as JPEG compression to reduce the size of image data, reduce data transmission volume and increase processing speed, and remove noise from the data to be transmitted through median filtering to improve data quality;

[0126] The preprocessing module also includes a resource optimization scheduling algorithm to dynamically allocate computing and storage resources of edge nodes. The specific steps are as follows:

[0127] Collect the resource usage of relevant preprocessing tasks during the preprocessing process, including CPU and GPU core usage and memory capacity, classify the urgency of preprocessing tasks, including deadline and task priority, and calculate the priority scores λ of different tasks:

[0128] For the deadline t, based on the loose time threshold t1 and the emergency time threshold t2, the deadline score value is set based on the threshold:

[0129] When t>t1, it means the related task is not urgent, and the score is set to 3 points;

[0130] When t2<t<t1, it means the relevant task is generally urgent, and the score is set to 6 points;

[0131] When t≤t2, it means that the relevant task is very urgent, and the score is set to 10 points;

[0132] For task priority, it is directly set by the user, with a score of 1 to 10;

[0133] λ=ω1·T+ω2·Y;

[0134] Among them, λ represents the priority score of the task, T and Y represent the deadline score and task priority score of the relevant task respectively, ω1 and ω2 represent the weight ratio of the two, and the sum of ω1 and ω2 is 1;

[0135] According to the priority scores λ of different preprocessing tasks, the tasks are processed in descending order.

[0136] It should be noted that ω1 and ω2 represent the weight ratio of the two, which are generally set to ω1=0.65 and ω2=0.35. The values ​​of ω1 and ω2 can also be adjusted according to actual usage. By adjusting different values ​​of ω1 and ω2, the task priority in different situations can be flexibly adjusted.

[0137] Heterogeneous computing acceleration modules are also set up in the cloud and edge, including GPU and TPU accelerators, to optimize the model reasoning process. The specific steps are as follows:

[0138] Set up a hardware abstraction layer to uniformly manage different types of accelerators, hide the complexity of specific hardware, and provide a unified interface, install drivers for related hardware, and ensure system and application compatibility, convert the model into a format suitable for the target accelerator, adjust the batch size or data parallelism to achieve the best performance of the accelerator, integrate the acceleration module with the existing computing framework, and use different accelerators for calculations.

[0139] It should be noted that the GPU accelerator is an external GPU in the form of an independent graphics card, which is connected to the PCIe slot on the edge and cloud computer motherboards for auxiliary accelerated computing. TPU acceleration refers to the use of Tensor Processing Unit for computing acceleration.

[0140] The present invention discloses a visual recognition algorithm polymorphic model micro-technology based on edge cloud and its application system. When in use, the cloud-based task requirements based on the edge end are used to screen suitable visual recognition models to participate in visual recognition tasks, and samples in the recognition process are collected at the same time. The data is pre-processed and quickly transmitted back to the cloud, and all visual recognition models are incrementally updated. The system response, recognition, and training are accelerated based on GPU and TPU to improve the response speed and resource utilization efficiency of the visual recognition system.

[0141] In the embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation; the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the method of this embodiment.

[0142] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A multi-state model micro-technology for visual recognition algorithms based on edge cloud, characterized by: It includes the cloud and edge. The cloud includes a polymorphic model management module and a model optimization module, and the edge includes a preprocessing module, among which: The polymorphic model management module includes n related visual recognition models, and automatically matches the most suitable model for the current task through an intelligent decision-making mechanism based on the calculation accuracy, complexity and load of different visual recognition models; The model optimization module collects samples of the visual recognition models in use, and based on the correct and incorrect samples obtained, incrementally updates the unused models, thereby continuously improving the visual recognition models; The preprocessing module performs data preprocessing and feature extraction at the edge through data preprocessing algorithms and feature extraction, thereby reducing the amount of data transmitted to the cloud and improving the processing speed.

2. According to claim 1, a multi-state model micro-technology of visual recognition algorithm based on edge cloud is characterized in that: The polymorphic model management module includes n related visual recognition models, and automatically matches the most suitable model for the current task through an intelligent decision-making mechanism based on the calculation accuracy, complexity and load of different visual recognition models, including the following steps: Based on the computational accuracy, complexity and load of different visual recognition models, the visual recognition models within the module are classified, and based on the computational accuracy, complexity and load of the model, specific classification is performed respectively; Establish an intelligent decision-making matching model to automatically match the most suitable model for the current task based on the priority of the current visual recognition task.

3. The multi-state model micro-technology of visual recognition algorithm based on edge cloud according to claim 2, characterized in that: Based on the calculation accuracy, complexity and load of different visual recognition models, the visual recognition models within the module are classified, and based on the calculation accuracy, complexity and load of the model, specific classification is performed respectively. The specific steps are as follows: For the comprehensive visual recognition model, the computational accuracy of the model is specifically classified according to the F1 score of the model. The F1 score is an indicator that comprehensively measures the precision and recall rate of the model. The specific steps are as follows: When the F1 score of the relevant model is less than 0.7, the relevant model is marked as a low-precision model; When the F1 score of the relevant model is 0.7≤≤0.9, the relevant model is marked as a medium-precision model; When the F1 score of the relevant model is greater than 0.9, the relevant model is marked as a high-precision model; According to the number of floating point operations FLOPs of the model, the complexity of the model is specifically classified. The specific steps are: When the FLOPs of the relevant model is less than 10 8 , then the relevant model is marked as a low-complexity model; When the relevant model is 10 8 ≤FLOPs≤10 9 , then the relevant model is marked as a medium complexity model; When the FLOPs of the relevant model is greater than 10 9 , then the relevant model is marked as a high-complexity model; Based on the CPU usage of the model, the model load is specifically classified. The specific steps are as follows: When the CPU usage of the relevant model is less than 30%, the relevant model is marked as a low-load model; When 30% of the relevant model ≤ CPU usage ≤ 70%, the relevant model is marked as a medium load model; When the CPU usage of the relevant model is greater than 70%, the relevant model is marked as a high-load model.

4. The multi-state model micro-technology of visual recognition algorithm based on edge cloud according to claim 3 is characterized in that: The intelligent decision-making matching model is established to automatically match the most suitable model for the current task based on the priority of the current visual recognition task. The specific steps are as follows: Establish an intelligent decision-making matching priority mode, including accuracy priority, complexity priority, load priority and mixed priority, set different levels of model scores, and divide the judgment conditions into three equal parts for different levels of models based on the judgment conditions: Among them, the scores of low-precision models are 2 points, 3 points, and 4 points respectively, the scores of medium-precision models are 5 points, 6 points, and 7 points respectively, the scores of high-precision models are 8 points, 9 points, and 10 points respectively, the scores of low-complexity models are 8 points, 9 points, and 10 points respectively, the scores of medium-complexity models are 5 points, 6 points, and 7 points respectively, the scores of high-complexity models are 2 points, 3 points, and 4 points respectively, the scores of low-load models are 8 points, 9 points, and 10 points respectively, the scores of medium-load models are 5 points, 6 points, and 7 points respectively, and the scores of high-load models are 2 points, 3 points, and 4 points respectively; In the accuracy priority mode, based on the accuracy requirement of the current task, the models that meet the accuracy requirement of the current task are retained, and the first residual comprehensive score Z1 of each model in the retained model is calculated respectively, where: Z1=B+C; In the above formula, B represents the model complexity score, C represents the model load score, the first remaining comprehensive score Z1 is arranged in descending order, and the first model is selected as the most suitable model. When the first remaining comprehensive score Z1 is the same, the model load score C of the models with the same score is arranged in descending order, and the first model is selected as the most suitable model. When the model load score C is the same, the model complexity score B of the models with the same score is arranged in descending order, and the first model is selected as the most suitable model. In the complexity priority mode, based on the complexity requirements of the current task, the models that meet the complexity requirements of the current task are retained, and the second residual comprehensive score Z2 of each model in the retained model is calculated respectively, where: Z2=A+C; In the above formula, A represents the model accuracy score, C represents the model load score, the second remaining comprehensive score Z2 is arranged in descending order, and the first model is selected as the most suitable model. When the second remaining comprehensive score Z2 is the same, the model accuracy scores A of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model load scores C of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model. In the load priority mode, based on the load requirements of the current task, the models that meet the load requirements of the current task are retained, and the third residual comprehensive score Z3 of each model in the retained model is calculated respectively, where: Z3=A+B; In the above formula, A represents the model accuracy score, B represents the model complexity score, the third remaining comprehensive score Z3 is arranged in descending order, and the first model is selected as the most suitable model. When the third remaining comprehensive score Z3 is the same, the model accuracy scores A of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model. When the model accuracy scores A are the same, the model complexity scores B of the models with the same score are arranged in descending order, and the first model is selected as the most suitable model. In the mixed priority mode, when the current task has accuracy requirements, complexity requirements, and load requirements, the model that meets the current task requirements is retained, and the comprehensive score Z4 is calculated for the retained model, where: Z4=A+B+C; Arrange the comprehensive scores Z4 in descending order and select the first model as the most suitable model; When the current task meets any two of the accuracy requirements, complexity requirements, and load requirements, among the models that meet the requirements, the remaining requirement scores are arranged in descending order, and the first model is selected as the most suitable model.

5. The multi-state model micro-technology of visual recognition algorithm based on edge cloud according to claim 1, characterized in that: The model optimization module collects samples of visual recognition models in use, and incrementally updates unused models based on the correct and incorrect samples obtained to ensure continuous improvement of the algorithm. The specific steps are as follows: Based on the selected optimal visual recognition model, the relevant parameters in the use process are recorded, including correct samples and error samples. The unused visual recognition models are updated and optimized in the cloud through correct samples and error samples. The specific steps are as follows: Collect samples that are correctly identified and their correct labels, samples that are incorrectly classified and their correct labels of the visual recognition model in use, and classify the types of incorrectly classified samples into categories, including misclassified samples and missed samples; Based on the correct and incorrect samples obtained, an incremental learning data set is constructed. The data set includes all incorrectly recognized samples and one-third of correctly recognized samples. The unused visual recognition model is fine-tuned through the online gradient descent method to adapt to the new data. The algorithm formula is: i t+1 =θ t -η·▽J(θ t ,x t ,y t ); Among them, θ t represents the model parameters at the current time step t, θ t+1 is the updated model parameter, which is obtained by adjusting the current model parameters using the new data. η represents the learning rate, ▽J(θ t ,x t ,y t ) represents the loss function J on the model parameters θ t gradient.

6. The multi-state model micro-technology of visual recognition algorithm based on edge cloud according to claim 5, characterized in that: The model optimization module is also provided with a self-updating optimization system, which establishes a specific database based on samples in each use of the visual recognition model, and the specific steps are as follows: Collect the erroneous samples obtained in each optimal recognition model recognition process, establish a specific database, and based on the type of samples, establish a misclassified sample database and a missed sample database in the specific database; Regularly conduct secondary specific training for the visual recognition model added to the polymorphic model management module, including misclassification specific training and missed detection specific training. After each optimal visual model identification, collect new error samples and label the samples. Add the newly collected error samples to the specific database according to classification. Set up a cleaning mechanism, regularly check the data in the specific database, and remove samples before N time points.

7. The multi-state model micro-technology of visual recognition algorithm based on edge cloud according to claim 1, characterized in that: The preprocessing module implements a fast data preprocessing algorithm and a feature extraction method at the edge, including a compression algorithm JPEG compression to reduce the volume of image data, reduce the amount of data transmission and increase the processing speed, and removes noise in the data to be transmitted through median filtering to improve data quality.

8. The multi-state model micro-technology of visual recognition algorithm based on edge cloud according to claim 7, characterized in that: The preprocessing module also includes a resource optimization scheduling algorithm to dynamically allocate computing and storage resources of edge nodes. The specific steps are as follows: Collect the resource usage of relevant preprocessing tasks during the preprocessing process, including CPU and GPU core usage and memory capacity, classify the urgency of preprocessing tasks, including deadline and task priority, and calculate the priority scores λ of different tasks: For the deadline t, based on the loose time threshold t1 and the emergency time threshold t2, the deadline score value is set based on the threshold: When t>t1, it means the related task is not urgent, and the score is set to 3 points; When t2<t<t1, it means the relevant task is generally urgent, and the score is set to 6 points; When t≤t2, it means that the relevant task is very urgent, and the score is set to 10 points; For task priority, it is directly set by the user, with a score of 1 to 10; λ=ω1·T+ω2·Y; Among them, λ represents the priority score of the task, T and Y represent the deadline score and task priority score of the relevant task respectively, ω1 and ω2 represent the weight ratio of the two, and the sum of ω1 and ω2 is 1; According to the priority scores λ of different preprocessing tasks, the tasks are processed in descending order.

9. The multi-state model micro-technology of visual recognition algorithm based on edge cloud according to claim 1, characterized in that: The cloud and edge are also provided with heterogeneous computing acceleration modules, including GPU and TPU accelerators, to optimize the model reasoning process. The specific steps are as follows: Set up a hardware abstraction layer to uniformly manage different types of accelerators, hide the complexity of specific hardware, and provide a unified interface, install drivers for related hardware, and ensure system and application compatibility, convert the model into a format suitable for the target accelerator, adjust the batch size or data parallelism to achieve the best performance of the accelerator, integrate the acceleration module with the existing computing framework, and use different accelerators for calculations.

10. An application system, comprising an edge server, a cloud server, and a cloud computing platform, for implementing the edge cloud-based visual recognition algorithm polymorphic model microtechnology described in any one of claims 1-9.

Images