An image joint inference recognition system and method based on a cloud edge architecture

Abstract

The application relates to an image joint inference recognition system and method based on a cloud edge end architecture, which comprises a cloud platform layer, a communication network layer, an edge layer and a terminal layer; the cloud platform layer comprises a cloud computing center, a database, a cloud file storage system and a mirror image warehouse; the edge layer comprises various devices integrated in a target image recognition scene, including edge devices and a local file storage system; and the terminal layer is a terminal device with a data acquisition function in the target image recognition scene, which is used for transmitting real-time images and videos collected by monitoring to the edge layer through wired or wireless networks for detection. The cloud edge joint inference algorithm is adopted, the shortcomings of single-side model deployment on the cloud or the edge are overcome, multi-target detection and recognition can be carried out more quickly, cloud edge resources are fully utilized, and the application is more suitable for complex multi-target image recognition and detection work.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence processing and joint reasoning of images or video stream data to be detected, based on edge computing. Specifically, it involves an image joint reasoning system and method based on artificial intelligence models, a cloud-edge-device architecture, and joint reasoning algorithms. Background Technology

[0002] Artificial intelligence has been widely used in target recognition and has been extended to multi-target recognition fields such as marine fish recognition, pathological image recognition, and autonomous driving through cloud computing and edge computing technologies. However, as the expanded scenarios become more complex, higher requirements are placed on recognition accuracy and resource allocation.

[0003] Currently, there are some target recognition systems based on cloud servers or edge devices, but they have the following shortcomings when applied to complex scenarios of multi-target detection: First, edge-based target recognition systems deploy models at the edge, with the cloud responsible for decision-making and the edge responsible for data processing and recognition. However, edge devices have limited resources, and the deployed models are lightweight, making it difficult for them to work independently in complex scenarios of multi-target detection. Second, cloud-based target recognition systems deploy models on cloud servers, receiving data uploaded from the edge devices for data processing and recognition. However, cloud servers are usually far from the work environment, resulting in high data transmission costs and long processing times, making them unsuitable for multi-target image recognition scenarios. Summary of the Invention

[0004] To address the aforementioned needs and the shortcomings of existing technologies, this invention proposes an image joint reasoning and recognition system based on a cloud-edge-device architecture.

[0005] Based on the Kubeedge-Sedna management platform, this invention provides a stable cloud-edge architecture. The cloud center can effectively manage a large number of image recognition edge devices, supporting large-scale deployment and centralized management.

[0006] This invention also proposes a joint inference algorithm, which overcomes the limitations of limited resources at the edge by offloading high-load tasks at the edge to the cloud, thereby improving the efficiency of dynamic resource allocation and image recognition, and reducing deployment costs.

[0007] Terminology Explanation:

[0008] Kubeedge-Sedna Management Platform: Kubeedge is an open-source system that supports edge computing and extends the orchestration capabilities of containerized applications to the edge. Sedna is an edge-cloud collaborative AI project incubated by KubeEdge SIG AI, which can extend the functionality of Kubeedge and provide cross-cloud-edge collaborative training and collaborative inference capabilities.

[0009] The technical solution of this invention is as follows:

[0010] An image joint reasoning and recognition system based on a cloud-edge-device architecture includes a cloud platform layer, a communication network layer, an edge layer, and a terminal layer;

[0011] The cloud platform layer includes a cloud computing center, database, cloud file storage system, and image repository;

[0012] The cloud computing center is used for training cloud and edge AI image recognition models and processing image recognition tasks offloaded from the edge to the cloud computing center; the database is used to store addressable directory addresses; the cloud file storage system is used to store cloud and edge AI image recognition models, as well as training and test set image files used for training the models; the mirror repository is used to store mirror files generated by cloud and edge AI image recognition models trained in the cloud computing center.

[0013] The communication network layer is used for information interaction, data uploading, and model and image downloading between the edge layer and the cloud platform layer;

[0014] The edge layer includes various devices integrated into the target image recognition scenario, including edge devices and local file storage systems;

[0015] The edge device carries an edge-end artificial intelligence image recognition model, which is used to process monitoring data uploaded from the terminal layer in the target image recognition scene, and calculates whether the monitoring data should be detected by the cloud-based artificial intelligence image recognition model through a joint inference algorithm; the local file storage system is used to store the image files, executable programs, image datasets uploaded from the terminal layer, and detection results of the historical and latest versions of the edge-end artificial intelligence image recognition model.

[0016] The terminal layer is a terminal device with data acquisition function in the target image recognition scenario, used to transmit real-time images and videos collected by monitoring to the edge layer through wired or wireless networks for detection.

[0017] The process of calculating whether the monitoring data is detected using a cloud-based artificial intelligence image recognition model through a joint inference algorithm includes: obtaining the current CPU remaining rate of the edge device, the current memory remaining rate of the edge device, the latency of the gateway between the edge device and the cloud computing center network connection, the model startup time, and the current data size of the current operation; and then using a joint inference algorithm to calculate whether the monitoring data is detected using a cloud-based artificial intelligence image recognition model.

[0018] According to a preferred embodiment of the present invention, the cloud computing center deploys the open-source Kubeedge-Sedna management platform;

[0019] Edge devices are added to the cloud platform layer through the Join Token key and certificate verification method of the Kubeedge-Sedna management platform, and the cloud computing center manages and controls the edge devices;

[0020] The cloud computing center receives access from terminal devices and designates the edge devices to which the terminal devices belong. The cloud computing center manages and controls the edge devices.

[0021] According to a preferred embodiment of the present invention, the edge device is an Nvidia Nx Xavier integrated device; the terminal device includes a camera or a monitor.

[0022] According to a preferred embodiment of the present invention, the cloud computing center, database, cloud file storage system, and mirror repository are all located in a local area network; the edge device, local file storage system, and terminal layer are located in a local area network.

[0023] A cloud-edge-device-based image joint reasoning and recognition method, implemented based on the aforementioned image joint reasoning and recognition system, includes:

[0024] Step 1: The cloud computing center trains the required cloud and edge AI image recognition models according to different target image recognition scenario requirements, packages these two models into an image, and stores it in the image repository; and sends the image file of the edge AI image recognition model to the local file storage system.

[0025] Step 2: The cloud computing center starts the Pod container, runs the target image recognition service, and deploys the Sedna plugin on the specified edge device through the YAML file. The Sedna plugin already contains the joint inference algorithm.

[0026] Step 3: The edge device pulls the image required to run the Pod container from the local file storage system. The Pod container starts successfully and returns the Pod container status to the cloud computing center.

[0027] Step 4: After the edge device runs the target image recognition service through the YAML file, it uses the Sedna plugin to perform joint inference algorithms in the target image recognition scenario.

[0028] Step 5: The terminal device connects to the edge device via an IP address, and the edge device acquires target image recognition data through the real-time image transmitted from the terminal device;

[0029] Step 6: The edge device reads the target image recognition data to be detected, and simultaneously obtains the current edge device CPU remaining rate, current edge device memory remaining rate, latency of the gateway between the edge device and the cloud computing center network connection, model startup time, and current data size of this operation. It then uses a joint inference algorithm to calculate whether the monitoring data should be used to enable the cloud-based artificial intelligence image recognition model for detection, and performs target image recognition service to obtain the target image recognition result.

[0030] According to a preferred embodiment of the present invention, in step 1, the cloud computing center trains the required cloud-based and edge-based artificial intelligence image recognition models according to different target image recognition scenario requirements, specifically including:

[0031] Step 1.1: Construct cloud-based and edge-based AI image recognition models;

[0032] The cloud-based AI image recognition model includes the backbone neural network CSP-Darknet53, the neural network LeNet-5, and the neural network Darknet-19;

[0033] The edge AI image recognition model adopts the CSP-Darknet53 neural network architecture;

[0034] Step 1.2: Train the cloud-based artificial intelligence image recognition model;

[0035] Obtain the dataset;

[0036] The training set in the dataset is labeled with features and class labels are added, and then exported as XML format to obtain the XML dataset.

[0037] Normalize the coordinates of the images in the XML dataset;

[0038] Training cloud-based artificial intelligence image recognition models;

[0039] Step 1.3: Train the edge AI image recognition model;

[0040] Obtain the dataset;

[0041] The training set in the dataset is labeled with features and class labels are added, and then exported as XML format to obtain the XML dataset.

[0042] Normalize the coordinates of the images in the XML dataset;

[0043] Training an edge AI image recognition model.

[0044] According to a preferred embodiment of the present invention, step 6, calculating whether the monitoring data is detected using a cloud-based artificial intelligence image recognition model via a joint inference algorithm, includes:

[0045] Step 6.1: Calculate the relationship between edge device model load and resource utilization D(r,h) to obtain the value of resource pressure load parameter n1:

[0046] D(r,h) is shown in equation (1):

[0047]

[0048] In equation (1), R a With R b These represent the current CPU and memory remaining rates of the edge device, respectively; V1 and V2 are adjustment constants; a is the average constant of the model startup time; h i This represents the current data size for this task.

[0049] If the value of D(r,h) is less than 0, n1 = 0, proceed to step 6.2; otherwise, n1 = 1, proceed to step 6.3.

[0050] Step 6.2: When n1 = 0, it means that the remaining resources of the current edge device are insufficient to support the load of this detection; calculate the network status W(t) between the edge device and the cloud computing center to obtain the network status parameter n2, W(t) as shown in equation (2):

[0051]

[0052] In equation (2), T i Let n be the latency between the edge device and the cloud computing center network, and m be the number of gateways on the connection path. When W(t) is greater than 150ms, n2 = 0; otherwise, n2 = 1.

[0053] If n2 = 1, it means that the current network connection between the edge device and the cloud computing center is good, so the image is uploaded to the cloud and the cloud artificial intelligence image recognition model is started for recognition; if n2 = 0, it means that the current network connection between the edge device and the cloud computing center is poor, so the image is marked as unsuitable for uploading to the cloud for recognition, and the process returns to step 6.1. When n2 is calculated again and the condition is met, the image is uploaded to the cloud artificial intelligence image recognition model for detection.

[0054] Step 6.3: When n1 = 1, it means that the remaining resources of the current edge device are sufficient to support the load of this detection. Start the artificial intelligence image recognition model to perform detection and calculate the detection accuracy B(y) of the edge model; B(y) is shown in equation (3):

[0055]

[0056] In equation (3), C represents the minimum expected accuracy, and y iThis represents the actual accuracy of the test. When B(y)≥0, the accuracy parameter n3 = 1, and proceed to step 6.4; when B(y)<0, n3 = 0, and proceed to step 6.4.

[0057] Step 6.4: If n3 = 1, it means that the edge image recognition model has a high accuracy in recognizing the image. Mark the image as recognized and store it in the local file storage system.

[0058] Step 6.5: If n3 = 0, it means that the edge image recognition model has low accuracy in recognizing the image. Calculate W(t) to get the value of n2. If n2 = 0, mark the image as unsuitable for cloud recognition and wait for the network to be good before uploading it to the cloud. If n2 = 1, upload the image to the cloud and start the cloud artificial intelligence image recognition model for recognition.

[0059] According to a preferred embodiment of the present invention, in step 6, if the cloud-based artificial intelligence image recognition model is enabled, the detection results and images are stored in the cloud file storage system, and a backup copy of the detection results is sent to the local file storage system; otherwise, the edge-based artificial intelligence image recognition model is enabled, the detection results and images are stored in the local file storage system, and the data from the local file storage system is packaged and sent to the cloud platform layer periodically using the network idle time transmission mode.

[0060] The beneficial effects of this invention are as follows:

[0061] 1. Structured Approach. This invention provides a mature Kubeedge-Sedna management platform as a method for cloud-based decision-making and management of the edge and terminal layers, supporting large-scale node access and management, task initiation and distribution.

[0062] 2. Lightweight. This invention adopts a mature, lightweight, and open-source edge computing management platform, and preferably uses Nvidia NXXavier integrated devices as edge devices. It features low power consumption, small size, powerful computing power, rapid deployment, and can support large-scale deployment, making it more suitable for multi-target detection scenarios.

[0063] 3. Cloud-edge collaboration. This system employs a joint inference algorithm, overcoming the shortcomings of deploying the model only on the cloud or edge. It can detect and identify multiple targets more frequently and quickly, making it more suitable for complex multi-target image recognition and detection tasks.

[0064] 4. Continuous updates. This system adopts a dataset feedback method, separating model training and model inference. This allows the cloud to continuously train new models, correct parameters and feature values, and maintain model updates, providing continuous and efficient detection efficiency and better meeting the needs of multi-object detection scenarios.

[0065] 5. Improved cloud-edge resource optimization strategies. Utilizing edge computing closer to the data collection end reduces data transmission latency, while cloud-edge collaborative inference fully leverages cloud-edge resources to improve task execution efficiency. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the image joint reasoning and recognition system based on the cloud-edge-device architecture of the present invention;

[0067] Figure 2 This is a schematic diagram of a cloud-edge-device architecture.

[0068] Figure 3 This is a diagram illustrating image packaging and distribution.

[0069] Figure 4 This is a schematic diagram of cloud-edge-device information interaction.

[0070] Figure 5 This is a schematic diagram of the joint inference algorithm;

[0071] Figure 6 This is a schematic diagram of the fish species to be tested in Example 4;

[0072] Figure 7 This is a schematic diagram of the neural network architecture of a cloud-based artificial intelligence image recognition model. Detailed Implementation

[0073] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0074] Example 1

[0075] An image joint reasoning and recognition system based on a cloud-edge-device architecture, such as Figure 1 As shown, it includes the cloud platform layer, communication network layer, edge layer, and terminal layer;

[0076] The cloud platform layer includes cloud computing centers, databases, cloud file storage systems, and image repositories;

[0077] The cloud computing center integrates software such as the Go environment, Kubeedge platform, Docker, and Sedna plugin. Figure 3 As shown; used for training AI image recognition models in the cloud and edge, and processing image recognition tasks offloaded from the edge to the cloud computing center; the database is used to store addressable directory addresses; the cloud file storage system is used to store the AI ​​image recognition models in the cloud and edge, as well as the training and test set image files used for training the models; the mirror repository is used to store the mirror files generated by the AI ​​image recognition models trained in the cloud computing center in the cloud and edge; such as Figure 3 As shown.

[0078] The communication network layer is used for information exchange, data uploading, and model and image downloading between the edge layer and the cloud platform layer; it consists of wired / wireless communication lines and various gateways between the edge layer and the cloud platform layer.

[0079] The edge layer includes various devices integrated into the target image recognition scenario, including edge devices and local file storage systems;

[0080] The edge device carries an edge-end artificial intelligence image recognition model, which is used to process monitoring data uploaded from the terminal layer in the target image recognition scene, and calculates whether the monitoring data should be detected by the cloud-based artificial intelligence image recognition model through a joint inference algorithm; the local file storage system is used to store the image files, executable programs, image datasets uploaded from the terminal layer, and detection results of the historical and latest versions of the edge-end artificial intelligence image recognition model.

[0081] The terminal layer is a terminal device with data acquisition function in the target image recognition scenario. It is used to transmit real-time images and videos collected by monitoring to the edge layer through wired or wireless networks for detection.

[0082] The process of calculating whether the monitoring data is detected using a cloud-based artificial intelligence image recognition model through a joint inference algorithm includes: obtaining the current CPU remaining rate of the edge device, the current memory remaining rate of the edge device, the latency of the gateway between the edge device and the cloud computing center network connection, the model startup time, and the current data size of the current operation; and then using a joint inference algorithm to calculate whether the monitoring data is detected using a cloud-based artificial intelligence image recognition model.

[0083] Example 2

[0084] The image joint reasoning and recognition system based on a cloud-edge-device architecture, as described in Example 1, differs in that:

[0085] The cloud computing center deploys the open-source Kubeedge-Sedna management platform;

[0086] Edge devices join the cloud platform layer via Join Token keys and certificate verification on the Kubeedge-Sedna management platform. The cloud computing center manages and controls the edge devices. The cloud computing center receives access from terminal devices, specifically by creating resource objects such as devices and object models through Yaml configuration files, adding terminal devices via URLs in device resource groups, and specifying the edge device to which each terminal device belongs. The cloud computing center then manages and controls the edge devices. This forms a cloud-edge-device architecture, such as... Figure 2 As shown.

[0087] Edge devices are devices with high floating-point computing capabilities, using Nvidia Nx Xavier processors; terminal devices include cameras or monitors.

[0088] The cloud computing center, database, cloud file storage system, and image repository are all located on the same local area network (LAN); edge devices, local file storage system, and terminal layer are located on the same LAN.

[0089] Example 3

[0090] An image joint reasoning and recognition method based on a cloud-edge-device architecture, implemented based on the image joint reasoning and recognition system described in Embodiment 1 or 2, includes:

[0091] Step 1: The cloud computing center trains the required cloud and edge AI image recognition models according to different target image recognition scenario requirements, packages these two models into an image and stores it in the image repository; and sends the edge AI image recognition model to the local file storage system.

[0092] Step 2: The cloud computing center starts Pod containers, runs the target image recognition service, and deploys the Sedna plugin on the specified edge device using a YAML file. The Sedna plugin already contains a joint inference algorithm; for example... Figure 4 As shown;

[0093] Step 3: The edge device pulls the image required to run the Pod container from the local file storage system. The Pod container starts successfully and returns the Pod container status to the cloud computing center.

[0094] The cloud computing center defines Pod containers, target image recognition services, and Sedna plugins through YAML files. The communication network layer transmits the startup command to the edge layer. After receiving the startup command, the edge device starts the Pod container according to the contents of the YAML file, runs the target image recognition service, pulls the image required by the Pod from the local file storage system, and returns edge device information and Pod information to the cloud.

[0095] Step 4: After the edge device runs the target image recognition service through the YAML file, it uses the Sedna plugin to perform joint inference algorithms in the target image recognition scenario.

[0096] Step 5: The terminal device connects to the edge device via IP address, and the edge device obtains target image recognition data through the real-time image transmitted from the terminal device;

[0097] Step 6: The edge device reads the target image recognition data to be detected, and simultaneously obtains the current edge device CPU remaining rate, current edge device memory remaining rate, gateway latency between the edge device and the cloud computing center network connection, model startup time, and current data size for this operation. It then uses a joint inference algorithm to calculate whether the monitoring data warrants the use of a cloud-based AI image recognition model for detection, and performs target image recognition service to obtain the target image recognition result. For example... Figure 5 As shown.

[0098] Example 4

[0099] The image joint reasoning and recognition method based on a cloud-edge-device architecture, as described in Example 3, differs in that:

[0100] In step 1, the cloud computing center trains the required cloud-based and edge-based AI image recognition models according to the needs of marine fish image recognition scenarios, specifically including:

[0101] Step 1.1: Construct cloud-based and edge-based AI image recognition models;

[0102] The cloud-based AI image recognition model includes the YOLOv5 official backbone neural network CSP-Darknet53, the LeNet-5 neural network, and the Darknet-19 neural network. This model employs a multi-scale, multi-neural network training approach, with different network models corresponding to targets of different fish sizes. The YOLOv5 official backbone neural network CSP-Darknet53 corresponds to large-scale feature datasets, while LeNet-5 and Darknet-19 correspond to medium- and small-scale feature datasets, respectively, forming a multi-neural network structure. Figure 7 As shown;

[0103] The edge AI image recognition model adopts the CSP-Darknet53 neural network architecture provided by YOLO5.

[0104] Step 1.2: Train the cloud-based artificial intelligence image recognition model;

[0105] Obtain the dataset; use image labeling software to label the features of the training set in the dataset and add category labels, export it as XML format to obtain the XML dataset;

[0106] The training set file is read and returned using the os.listdir(xmlfilePath) function, imagew = 1. / size[0], imageh = 1. / size[1], and the coordinates of the images in the xml dataset are normalized.

[0107] Training process: The normalized dataset is fed into the CSP-Darknet53 neural network for training, where the Backbone performs feature extraction. The cloud-based AI image recognition model employs a multi-scale training strategy, feeding features of different scales into different neural networks for training; details are as follows:

[0108] The feature image input size of CSP-Darknet53 is (N, C). in H in W in ) = (N, 3, 640, 640), corresponding to large-scale features, and using convolutional layers with kernel = 6, stride = 2, and padding = 2; in SPPF, multiple 5x5 MaxPool layers are used for spatial pyramid pooling to transform the previously obtained feature map into a fixed-size feature vector.

[0109] The feature image input size of Darknet-19 is (N, C). in H in W in = (N, 3, 224, 224), corresponding to mesoscale features, and using convolutional layers with kernel = 4, stride = 2, padding = 2 and alternating 3x3 and 1x1 convolutional kernels;

[0110] The feature image input size of LeNet-5 is (N, C). in H in W in = (N,3,32,32), corresponding to small-scale features, all convolutional kernels are 5*5, stride is 1, and all pooling methods are average pooling;

[0111] Step 1.2: Train the edge AI image recognition model;

[0112] Obtain the dataset;

[0113] Image labeling software was used to label the features of the training set in the dataset and add category labels, which were then exported as XML format to obtain the XML dataset.

[0114] The training set file is read and returned using the os.listdir(xmlfilePath) function, imagew = 1. / size[0], imageh = 1. / size[1], and the coordinates of the images in the xml dataset are normalized.

[0115] Training process: The normalized dataset is fed into the CSP-Darknet53 neural network for training, and the Backbone performs feature extraction; specifically, the input size of the feature images for CSP-Darknet53 is (N, C... in H in W in = (N, 3, 640, 640), corresponding to large-scale features, and uses convolutional layers with kernel = 6, stride = 2, and padding = 2; in SPPF, multiple 5x5 MaxPool layers are used for spatial pyramid pooling to transform the previously obtained feature map into a fixed-size feature vector.

[0116] The cloud-based AI image recognition model uses YOLOv5x as its core, employing a multi-neural network structure. Different network models correspond to targets of different fish sizes, and the model is trained and tested using a fusion approach.

[0117] The edge model was trained and tested using a single YOLOv5x neural network architecture.

[0118] Underwater surveillance cameras transmit real-time video and frame images to edge devices to acquire datasets; targets to be detected, such as... Figure 6 As shown;

[0119] In step 6, such as Figure 5 As shown, the joint inference algorithm is used to calculate whether the monitoring data is detected using a cloud-based artificial intelligence image recognition model, and to obtain the current CPU remaining rate (R). a ) and memory remaining rate (R b The percentages are: 36% and 42%; the model startup time constant is 2.5 s (a), and the image data size is 8.6 M (h). i Network latency (T) between edge devices and cloud servers at various gateways i ): T0=90ms, T1=136ms, T2=170ms, T3=128ms, T4=112ms, T5=143ms, T6=101ms, T7=121ms, m=8. include:

[0120] Step 6.1: Calculate the relationship between edge device model load and resource utilization D(r,h) to obtain the value of resource pressure load parameter n1:

[0121] D(r,h) is shown in equation (1):

[0122]

[0123] In equation (1), R a With R bV1 and V2 represent the current CPU and memory remaining rates of the edge device, respectively. V1 and V2 are adjustment constants. Resource configurations vary across different application scenarios for edge devices, and the size of data uploaded by terminal devices can differ due to factors such as resolution, leading to an imbalance between CPU and memory remaining rates. For example, a sudden increase in uploaded data can lower the memory remaining rate while the CPU remaining rate remains considerable. V1 and V2 are adjustment constants to balance this relationship. When the data size h... i When the length is 4m-9m, V1 and V2 are 0.2 and 0.25 respectively; when the data size h i When the length is 9m-15m, V1 and V2 are 0.36 and 0.55 respectively; when the data size h... i When the length is 15m-25m, V1 and V2 are taken as 0.34 and 0.5 respectively; when the data size h i When the time exceeds 25m, V1 and V2 are taken as 0.32 and 0.55 respectively; a is the model start-up time constant, h i This represents the current data size for this task.

[0124]

[0125] Therefore, the model load and remaining resource usage function D(r,h)≥0, resulting in n1=1.

[0126] Step 6.2: Activate the edge-end marine fish artificial intelligence image recognition model to perform recognition, obtain the recognition results and accuracy. The current accuracy is y. i =0.8329, calculate B(y). C represents the minimum expected accuracy, which is 0.8 in this example.

[0127]

[0128] Therefore, the edge end has high accuracy in recognizing the image, and if n3=1, there is no need to calculate the value of W(t).

[0129] Step 6.3: As Figure 5 As shown, the edge device stores the image file and the image recognition result image and result in the local file storage system.

Claims

1. An image joint reasoning and recognition system based on a cloud-edge-device architecture, characterized in that, It includes the cloud platform layer, communication network layer, edge layer, and terminal layer; The cloud platform layer includes a cloud computing center, database, cloud file storage system, and image repository; The cloud computing center is used for training cloud and edge AI image recognition models and processing image recognition tasks offloaded from the edge to the cloud computing center; the database is used to store addressable directory addresses; the cloud file storage system is used to store cloud and edge AI image recognition models, as well as training and test set image files used for training the models; the mirror repository is used to store mirror files generated by cloud and edge AI image recognition models trained in the cloud computing center. The communication network layer is used for information interaction, data uploading, and model and image downloading between the edge layer and the cloud platform layer; The edge layer includes various devices integrated into the target image recognition scenario, including edge devices and local file storage systems; The edge device carries an edge-end artificial intelligence image recognition model, which is used to process monitoring data uploaded from the terminal layer in the target image recognition scene, and calculates whether the monitoring data should be detected by the cloud-based artificial intelligence image recognition model through a joint inference algorithm; the local file storage system is used to store the image files, executable programs, image datasets uploaded from the terminal layer, and detection results of the historical and latest versions of the edge-end artificial intelligence image recognition model. The terminal layer is a terminal device with data acquisition function in the target image recognition scenario, used to transmit real-time images and videos collected by monitoring to the edge layer through wired or wireless networks for detection. The monitoring data is analyzed using a joint inference algorithm to determine whether a cloud-based AI image recognition model is being used for detection, including: Step 1: Calculate the relationship between edge device model load and resource utilization D(r,h) to obtain the value of resource pressure load parameter n1: D(r,h) is shown in equation (1): In equation (1), R a With R b These represent the current CPU and memory remaining rates of the edge device, respectively; V1 and V2 are adjustment constants; a is the average constant of the model startup time; h i This represents the current data size for this task. If the value of D(r,h) is less than 0, n1 = 0, proceed to step 6.2; otherwise, n1 = 1, proceed to step 6.

3. Step 2: When n1 = 0, it means that the remaining resources of the current edge device are insufficient to support the load of this detection; calculate the network status W(t) between the edge device and the cloud computing center to obtain the network status parameter n2, W(t) as shown in equation (2): In equation (2), T / Let n be the latency between the edge device and the cloud computing center network, and m be the number of gateways on the connection path. When W(t) is greater than 150ms, n2 = 0; otherwise, n2 = 1. If n2 = 1, it means that the current network connection between the edge device and the cloud computing center is good, so the image is uploaded to the cloud and the cloud artificial intelligence image recognition model is started for recognition; if n2 = 0, it means that the current network connection between the edge device and the cloud computing center is poor, so the image is marked as unsuitable for uploading to the cloud for recognition, and the process returns to step 6.

1. When n2 is calculated again and the condition is met, the image is uploaded to the cloud artificial intelligence image recognition model for detection. Step 3: When n1 = 1, it means that the remaining resources of the current edge device are sufficient to support the load of this detection. Start the artificial intelligence image recognition model to perform detection and calculate the detection accuracy B(y) of the edge model; B(y) is shown in equation (3): In equation (3), C represents the minimum expected accuracy, and y i This represents the actual accuracy of the test. When B(y)≥0, the accuracy parameter n3 = 1, and proceed to step 6.4; when B(y)<0, n3 = 0, and proceed to step 6.

4. Step 4: If n3 = 1, it means that the edge image recognition model has a high accuracy in recognizing the image. Mark the image as recognized and store it in the local file storage system. Step 5: If n3 = 0, it means that the edge image recognition model has low accuracy in recognizing the image. Calculate W(t) to get the value of n2. If h2 = 0, mark the image as unsuitable for cloud recognition and wait for the network to be good before uploading it to the cloud. If n2 = 1, upload the image to the cloud and start the cloud artificial intelligence image recognition model for recognition.

2. The image joint reasoning and recognition system based on a cloud-edge-device architecture according to claim 1, characterized in that, The cloud computing center deploys the open-source Kubeedge-Sedna management platform; Edge devices are added to the cloud platform layer through the Join Token key and certificate verification method of the Kubeedge-Sedna management platform, and the cloud computing center manages and controls the edge devices; The cloud computing center receives access from terminal devices and designates the edge devices to which the terminal devices belong. The cloud computing center manages and controls the edge devices.

3. The image joint reasoning and recognition system based on a cloud-edge-device architecture according to claim 1, characterized in that, The edge device is an Nvidia Nx Xavier integrated device; the terminal device includes a camera or monitor.

4. The image joint reasoning and recognition system based on a cloud-edge-device architecture according to claim 1, characterized in that, The cloud computing center, database, cloud file storage system, and image repository are all located on the same local area network (LAN); edge devices, local file storage system, and terminal layer are located on the same LAN.

5. An image joint reasoning and recognition method based on a cloud-edge-device architecture, implemented based on the image joint reasoning and recognition system according to any one of claims 1-4, characterized in that, include: Step 1: The cloud computing center trains the required cloud and edge AI image recognition models according to different target image recognition scenario requirements, packages these two models into an image, and stores it in the image repository; and sends the image file of the edge AI image recognition model to the local file storage system. Step 2: The cloud computing center starts the Pod container, runs the target image recognition service, and deploys the Sedna plugin on the specified edge device through the YAML file. The Sedna plugin already contains the joint inference algorithm. Step 3: The edge device pulls the image required to run the Pod container from the local file storage system. The Pod container starts successfully and returns the Pod container status to the cloud computing center. Step 4: After the edge device runs the target image recognition service through the YAML file, it uses the Sedna plugin to perform joint inference algorithms in the target image recognition scenario. Step 5: The terminal device connects to the edge device via an IP address, and the edge device acquires target image recognition data through the real-time image transmitted from the terminal device; Step 6: The edge device reads the target image recognition data to be detected, and simultaneously obtains the current edge device CPU remaining rate, current edge device memory remaining rate, latency of the gateway between the edge device and the cloud computing center network connection, model startup time, and current data size of this operation. It then uses a joint inference algorithm to calculate whether the monitoring data should be used to enable the cloud-based artificial intelligence image recognition model for detection, and performs target image recognition service to obtain the target image recognition result.

6. The image joint reasoning and recognition method based on a cloud-edge-device architecture according to claim 5, characterized in that, In step 1, the cloud computing center trains the required cloud-based and edge-based artificial intelligence image recognition models according to different target image recognition scenario requirements, specifically including: Step 1.1: Construct cloud-based and edge-based AI image recognition models; The cloud-based AI image recognition model includes the backbone neural network CSP-Darknet53, the neural network LeNet-5, and the neural network Darknet-19; The edge AI image recognition model adopts the CSP-Darknet53 neural network architecture; Step 1.2: Train the cloud-based artificial intelligence image recognition model; Obtain the dataset; The training set in the dataset is labeled with features and class labels are added, and then exported as XML format to obtain the XML dataset. Normalize the coordinates of the images in the XML dataset; Training cloud-based artificial intelligence image recognition models; Step 1.3: Train the edge AI image recognition model; Obtain the dataset; The training set in the dataset is labeled with features and class labels are added, and then exported as XML format to obtain the XML dataset. Normalize the coordinates of the images in the XML dataset; Train an edge AI image recognition model.

7. The image joint reasoning and recognition method based on a cloud-edge-device architecture according to any one of claims 5-6, characterized in that, In step 6, if the cloud-based AI image recognition model is enabled, the detection results and images are stored in the cloud file storage system, and a backup copy of the detection results is sent to the local file storage system. Otherwise, the edge AI image recognition model is enabled, the detection results and images are stored in the local file storage system, and the data from the local file storage system is packaged and sent to the cloud platform layer periodically using the network idle time transmission mode.

Images