An AI multi-modal visual recognition system based on edge computing

Abstract

The application relates to the field of AI multi-modal visual recognition technology, and specifically discloses an AI multi-modal visual recognition system based on edge computing, which comprises an edge perception and processing subsystem arranged at an edge node and a cloud-side coordination and optimization subsystem arranged at a cloud side. The edge subsystem dynamically loads a light-weighted recognition model according to real-time computing power and scene complexity through a dynamic model adaptation engine, and fuses and decides multi-modal recognition results; the cloud-side subsystem continuously optimizes a model library and provides strategy guidance for the edge side through incremental model distillation and a scene knowledge graph. The application realizes efficient and self-adaptive multi-modal recognition in a resource-limited edge environment, and continuously evolves the system recognition ability through cloud-edge coordination.

Description

Technical Field

[0001] This invention belongs to the field of AI multimodal visual recognition technology, specifically relating to an AI multimodal visual recognition system based on edge computing. Background Technology

[0002] Artificial intelligence (AI) technology, as a core driving force for the intelligent transformation of society, is being widely applied in numerous visual perception fields such as security monitoring, autonomous driving, and industrial quality inspection. Multimodal visual recognition, by fusing perception data from multiple modalities such as images, videos, infrared, and depth, aims to build a more comprehensive and robust scene understanding capability, and is a key technological direction for improving the environmental perception and decision-making capabilities of intelligent systems.

[0003] A centralized cloud-based processing architecture is adopted, uploading multimodal perception data to a cloud server for model inference. However, this architecture faces significant bottlenecks in applications with high real-time requirements: the data transmission process introduces substantial network latency, making it difficult to meet the millisecond-level response requirements of tasks such as emergency obstacle avoidance in autonomous driving and real-time defect detection in industrial production lines. Simultaneously, the continuous uploading of massive amounts of raw data puts enormous pressure on network bandwidth and raises the risk of user data privacy breaches.

[0004] While edge computing alleviates these problems by offloading computational tasks to devices at the network edge, the inherently limited computing resources and storage space of edge devices make it difficult to directly deploy multimodal recognition models with large numbers of parameters and complex computations. Existing lightweight techniques, such as fixed model pruning or quantization strategies, can reduce model complexity, but when faced with dynamically changing and complex scenarios, they often lead to an unacceptable decline in recognition accuracy due to insufficient model capacity and expressive power. Summary of the Invention

[0005] The purpose of this invention is to provide an AI multimodal visual recognition system based on edge computing, so as to solve the technical contradictions in the existing cloud-based centralized processing architecture, which suffers from high latency, high bandwidth pressure, and high privacy risks, as well as the difficulty of deploying high-precision multimodal recognition models on edge devices due to limited resources.

[0006] The technical solution of this invention is an AI multimodal visual recognition system based on edge computing, deployed in a distributed computing environment containing at least one edge computing node and a cloud server. The system includes an edge perception and processing subsystem deployed on the edge computing node, and a cloud collaboration and optimization subsystem deployed on the cloud server. The edge perception and processing subsystem and the cloud collaboration and optimization subsystem interact with each other through a preset secure communication protocol.

[0007] The edge perception and processing subsystem comprises a multimodal sensor array, a front-end feature extraction module, a dynamic model adaptation engine, and a local decision fusion module. The multimodal sensor array simultaneously acquires perception data of at least two different physical properties of the target scene, including visible light images, infrared thermal images, and depth point clouds. The front-end feature extraction module preprocesses and performs preliminary feature extraction on the raw perception data acquired by the multimodal sensor array, generating intermediate feature representations for each modality. The dynamic model adaptation engine dynamically loads and instantiates a lightweight multimodal recognition sub-model from a pre-defined model library based on the available computing resources of the real-time monitored edge computing nodes and the complexity assessment results of the current scene. The local decision fusion module fuses the preliminary recognition results of each modality output by the lightweight multimodal recognition sub-model instantiated by the dynamic model adaptation engine according to a pre-defined confidence weighting rule, generating the final local recognition decision.

[0008] The cloud-based collaboration and optimization subsystem comprises a model repository and version management module, a scene knowledge graph module, and an incremental model distillation module. The model repository and version management module stores, versions, and distributes a series of lightweight multimodal recognition sub-models with varying computational complexity and recognition accuracy. The scene knowledge graph module constructs and maintains a structured knowledge network centered on scene semantics. This network associates typical objects, events, and environmental attributes in different scenes with their corresponding preferred perceptual modal combinations and model configuration strategies. The incremental model distillation module periodically receives anonymized hard sample features and their corresponding local fusion decision results uploaded by the edge perception and processing subsystem. Using these as supervisory signals, it incrementally trains and distills the teacher models stored in the model repository, generating a new generation of higher-performing lightweight student models and updating the model repository.

[0009] Furthermore, the dynamic model adaptation engine's operating mechanism specifically includes a resource monitoring unit, a scene parsing unit, and a model selection and loading unit. The resource monitoring unit continuously monitors the current available memory, CPU utilization, GPU utilization, and estimated battery life of the edge computing node, and comprehensively calculates a quantified real-time available computing power score. The scene parsing unit receives the intermediate feature representation output by the front-end feature extraction module and performs semantic parsing of the current scene based on a pre-trained lightweight scene classification network, outputting a scene category identifier and a confusion index representing scene complexity. The model selection and loading unit receives the real-time available computing power score, scene category identifier, and confusion index; it has a built-in predefined strategy mapping table that maps different computing power score ranges, scene categories, and confusion indices to specific lightweight multimodal recognition sub-model version identifiers in the model repository; based on the mapping results, the model selection and loading unit initiates a model retrieval request to the cloud collaboration and optimization subsystem, and after receiving the model data, completes the instantiation and initialization of the model in the memory of the edge computing node.

[0010] Furthermore, the front-end feature extraction module processes the multimodal sensing data as follows: For visible light image data, preprocessing operations including white balance correction, gamma correction, and resolution normalization are performed, followed by the extraction of spatial texture features using a lightweight convolutional neural network. For infrared thermal imaging data, preprocessing operations including non-uniformity correction and temperature value calibration are performed, followed by the extraction of temperature distribution features using a convolutional layer network focused on gradient feature extraction. For deep point cloud data, preprocessing operations including outlier filtering and voxel grid downsampling are performed, followed by the extraction of geometric structure features using a lightweight point-convolution-based network. The feature extraction networks for each modality share some of the underlying convolutional kernel parameters to reduce the overall parameter count and promote feature alignment.

[0011] Furthermore, the confidence weighting rule generation process of the local decision fusion module is as follows: For the preliminary recognition result of each target object output by the lightweight multimodal recognition sub-model, the maximum value in the probability vector output by the softmax layer of the corresponding model is extracted as the original confidence of the modality for the target recognition result. Simultaneously, based on the modality priority configuration table for the current scene category identifier issued by the scene knowledge graph module, a static weight coefficient is assigned to each perceptual modality. The local decision fusion module multiplies the original confidence of each modality with its corresponding static weight coefficient to obtain the weighted confidence. Finally, the local decision fusion module sums the weighted confidence of all modalities for the same target object and determines the target category with the highest sum as the final local recognition decision.

[0012] Furthermore, the incremental model distillation module's workflow specifically includes a sample screening and desensitization unit, a teacher model fine-tuning unit, and a student model distillation unit. The sample screening and desensitization unit receives difficult sample data proactively reported by the edge perception and processing subsystem. The criterion for determining a difficult sample is that the weighted confidence sum of the final output from the local decision fusion module is lower than a preset confidence threshold. This unit desensitizes the original perception data of the difficult samples, retaining only the intermediate feature representation output by the front-end feature extraction module and the corresponding true or high-confidence pseudo-labels. The teacher model fine-tuning unit uses the desensitized difficult sample features and their labels to fine-tune the full-parameter, high-precision teacher model stored in the cloud with small batches and low learning rates, adapting the teacher model to difficult patterns in these edge scenarios. The student model distillation unit uses the fine-tuned teacher model to perform knowledge distillation on a lightweight student model selected from the model repository. The distillation process employs a multi-task loss function combining a soft-label loss function and an intermediate layer feature map matching loss function to transfer knowledge from the teacher model to the student model, thereby generating a new version of the student model with improved performance.

[0013] Furthermore, the construction and updating mechanism of the scene knowledge graph module specifically includes a graph construction unit and a strategy recommendation unit. The graph construction unit is built based on massive historical multimodal recognition task logs. Through entity extraction and relationship mining techniques, it associates scenes, objects, events, environmental conditions, effective modal combinations, and historically optimal model configurations to form a queryable graph-structured database. The strategy recommendation unit responds to the scene category identifiers uploaded by the edge perception and processing subsystem, searches the graph, identifies a list of typical objects strongly associated with the scene, the main modalities relied upon for historical successful recognition, and the identifiers of the lightweight model versions that perform best under different computing power conditions in that scene. This information is then encapsulated into a modality priority configuration table and a model recommendation strategy, and distributed to the corresponding edge computing nodes.

[0014] Furthermore, the system operates within a hierarchical decision-making and collaborative update framework. On a millisecond-level timescale, the edge perception and processing subsystem autonomously completes the entire process from data acquisition, feature extraction, model inference to decision fusion, ensuring exceptional real-time response. On a second- to minute-level timescale, the edge perception and processing subsystem communicates asynchronously with the cloud-based collaboration and optimization subsystem, reporting difficult samples and receiving updated model recommendation strategies and modal configurations. On an hour- to day-level timescale, the cloud-based collaboration and optimization subsystem, based on aggregated multi-edge node data, evolves the scene knowledge graph and iteratively updates the model library, enabling continuous evolution of the system's overall recognition capabilities.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0016] 1. This invention achieves precise matching between edge-side recognition models and real-time available computing resources and scene complexity by designing a dynamic model adaptation engine. Instead of using a fixed lightweight model, this engine dynamically loads the most suitable model version from a cloud-based model repository based on real-time monitored computing power scores and scene analysis results. This mechanism fundamentally resolves the contradiction between the limited resources of edge devices and the diverse needs of tasks. While ensuring millisecond-level real-time response, it maximizes the utilization of the limited computing power of edge devices to maintain the highest possible recognition accuracy, avoiding latency caused by model overload or a sharp drop in accuracy due to an overly simplified model.

[0017] 2. This invention establishes a sustainably evolving system ecosystem by constructing a cloud-based collaborative and optimization subsystem, particularly its incremental model distillation module and scene knowledge graph module. The incremental model distillation module continuously optimizes the cloud-based teacher model and distills stronger, lightweight student models using difficult samples reported from the edge, enabling the edge model's capabilities to iteratively improve with data accumulation. The scene knowledge graph module provides the edge with prior knowledge guidance based on historical experience, optimizing modality fusion strategies and model selection logic. This collaborative paradigm of "real-time execution at the edge and continuous evolution in the cloud" ensures the system not only possesses high efficiency in the present but also adaptability and growth capabilities for future complex scenarios.

[0018] 3. This invention achieves efficient and robust fusion of multimodal information at the edge by combining a local decision fusion module with confidence-weighted and scenario-based modality priority fusion rules. This rule comprehensively considers the instantaneous confidence of the model output and the prior importance of modalities based on scenario knowledge, making the fusion decision more reliable. Especially when the quality of a certain modality's data deteriorates due to environmental interference, the system can automatically reduce its weight and rely on other high-quality modalities to make judgments, significantly improving the overall robustness and reliability of the system in complex and changing real-world environments. Simultaneously, all raw sensing data is processed and fused at the edge, with only a very small amount of desensitized features or model parameters transmitted when necessary, greatly reducing network bandwidth pressure and fundamentally protecting user data privacy. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall technical architecture of the AI ​​multimodal visual recognition system based on edge computing proposed in this invention.

[0020] Figure 2 This is a schematic diagram of the core principle framework of the dynamic model adaptation engine in this invention;

[0021] Figure 3 This is a flowchart of the main stages of the edge sensing and processing subsystem in this invention.

[0022] Figure 4 This is a schematic diagram of the multi-level interaction relationship and data flow between the edge perception and processing subsystem and the cloud collaboration and optimization subsystem in this invention;

[0023] Figure 5 This is a schematic diagram of the core principle framework of the incremental model distillation module in this invention. Detailed Implementation

[0024] This invention provides an AI multimodal visual recognition system based on edge computing. Please refer to the appendix. Figures 1 to 5 This system is deployed in a distributed computing environment containing at least one edge computing node and a cloud server. It consists of an edge sensing and processing subsystem deployed on the edge computing node and a cloud collaboration and optimization subsystem deployed on the cloud server. All data interaction between the edge sensing and processing subsystem and the cloud collaboration and optimization subsystem is conducted through a pre-defined secure communication protocol. This secure communication protocol employs a two-way authentication mechanism based on digital certificates and uses advanced encryption standards (AACSBs) for encryption at the data transmission layer, ensuring the confidentiality and integrity of instructions and data during transmission over public networks. Edge computing nodes can be embedded devices, industrial gateways, or mobile terminals with certain computing capabilities. Their hardware platforms typically integrate a central processing unit (CPU), a graphics processing unit (GPU), a dedicated neural network processing unit (NNZ), memory, and various sensor interfaces. The cloud server is a cluster of multiple high-performance computing nodes, providing massive storage space and powerful batch computing capabilities.

[0025] Please refer to the attached document. Figure 1 The edge perception and processing subsystem is the core of multimodal perception and real-time decision-making. It includes a multimodal sensor array, a front-end feature extraction module, a dynamic model adaptation engine, and a local decision fusion module. The multimodal sensor array is physically integrated on edge computing nodes for high-precision synchronous acquisition of perception data of at least two different physical properties of the target scene. This array includes at least a visible light image sensor, an infrared thermal imaging sensor, and a depth sensor. The visible light image sensor captures the intensity of reflected light from the scene, generating RGB or grayscale image data streams. The infrared thermal imaging sensor captures the intensity of infrared radiation emitted by the surface of objects in the scene and converts it into grayscale or pseudo-color image data streams characterizing temperature distribution. The depth sensor uses structured light, time-of-flight, or binocular vision principles to acquire distance information from objects in the scene to the sensor, generating a 3D point cloud data stream. To achieve strict temporal and spatial alignment of multimodal data, the system equips all sensors with a unified hardware synchronization trigger signal, ensuring that data frames of all modalities are acquired within the same microsecond-level timestamp. Meanwhile, during the deployment phase, the extrinsic and intrinsic parameter matrices of each sensor are obtained through a calibration procedure, and a unified coordinate system is established, enabling data from different modes to be accurately registered in space.

[0026] The front-end feature extraction module directly receives the raw, synchronized data stream from the multimodal sensor array. The core task of this module is to preprocess the raw sensing data and extract preliminary features, generating intermediate feature representations for each modality that are dimensionally regular and have higher information density. Please refer to the appendix. Figure 3 The front-end feature extraction module employs a parallel, partially parameter-shared processing pipeline designed for different modalities of data. For visible light image data streams, the preprocessing stage first performs automatic white balance correction to eliminate the influence of ambient light color temperature. This algorithm calculates the mean of the image's RGB channels and adjusts the gain. Next, gamma correction is performed to linearize the image's brightness response curve and enhance details in dark areas. Finally, resolution normalization is performed, scaling or cropping all input images to a fixed size, such as 640 pixels × 480 pixels. The preprocessed images are then fed into a lightweight convolutional neural network for feature extraction. This convolutional neural network uses depthwise separable convolutions to construct its backbone, containing five convolutional blocks. Each block consists of a depthwise convolutional layer, a pointwise convolutional layer, a batch normalization layer, and a modified linear unit activation function in sequence. The network outputs a 512-dimensional feature vector that encodes the spatial texture and contour information of the image.

[0027] For infrared thermal imaging data streams, the preprocessing stage first performs non-uniformity correction to eliminate fixed-pattern noise caused by inconsistent responses among sensor pixels. The correction employs a two-point correction method, using pre-calibrated high-temperature and low-temperature blackbody radiation source data to calculate the gain and bias coefficients for each pixel. Subsequently, temperature calibration is performed, converting grayscale values ​​to true Celsius temperatures based on the sensor's radiation response model and mapping them to an integer range of 0 to 255 for subsequent processing. The preprocessed temperature distribution image is then fed into a feature extraction network. This network uses a convolutional layer structure focused on gradient feature extraction, with a core of 3×3 Sobel operator convolutional layers to highlight temperature abrupt changes in the image. Two standard convolutional layers are then connected for feature integration. The network output is a 256-dimensional feature vector, primarily encoding the temperature gradient distribution and heat source contour information of the scene.

[0028] For deep point cloud data streams, the preprocessing stage first performs outlier filtering using a statistical filtering method. This involves calculating the average distance between each point and its 50 nearest neighbors and removing points whose distance exceeds the global mean plus or minus two standard deviations to eliminate noise. Next, voxel mesh downsampling is performed, dividing the 3D space into sections with sides of length... A grid is constructed, where all points within each grid are replaced by their centroids, significantly reducing point cloud density while preserving geometry. The preprocessed point cloud is then fed into a lightweight network based on point convolutions. This network employs a hierarchical structure similar to PointNet++, constructing multi-scale features through farthest-point sampling and grouping operations, and using a multilayer perceptron for feature learning within local regions. The network outputs a 256-dimensional feature vector encoding the point cloud's geometry, normal orientation, and spatial distribution information. To reduce the overall parameter count and promote semantic alignment of features from different modalities, the three feature extraction networks share the same convolutional kernel weights in the initial one or two convolutional or point convolutional layers. These shared low-level convolutional kernels learn to extract general low-level features across modalities, such as edges and corners.

[0029] The dynamic model adaptation engine is an intelligent scheduling center that enables resource and scene adaptation at the edge. Please refer to the appendix. Figure 2 Its operation mechanism is specifically completed collaboratively by a resource monitoring unit, a scene parsing unit, and a model selection and loading unit. The resource monitoring unit runs continuously as a background daemon, actively collecting real-time hardware status data of the edge computing nodes 10 times per second. The collected data includes the current available memory size in megabytes; the average utilization percentage of all CPU cores; the utilization percentage of the graphics processing unit (GPU) or dedicated neural network processing unit; and the estimated remaining battery life based on the current discharge current and total battery capacity, in minutes. The resource monitoring unit has a built-in quantitative scoring function that maps the above four indicators to a real-time available computing power score between 0 and 100. The specific calculation logic of the scoring function is as follows: First, a baseline value is set for each indicator, for example, the available memory baseline is 512 megabytes, the CPU utilization baseline is 50%, the GPU utilization baseline is 70%, and the battery life baseline is 120 minutes. Then, the proportion or inverse proportion of each indicator relative to its baseline value is calculated, and a weighted sum is performed. The weighting coefficients are preset according to the device type; for example, for mobile devices, battery life has a higher weight; for fixed-power equipment, computing unit utilization has a higher weight. Finally, the weighted sum is normalized to a score between 0 and 100 using the Sigmoid function.

[0030] The scene parsing unit receives intermediate feature representations of various modalities from the front-end feature extraction module. This unit loads a pre-trained lightweight scene classification network, which takes the concatenated result of the multimodal feature vectors output by the front-end feature extraction module as input. Specifically, it sequentially concatenates the 512-dimensional feature vector from visible light images, the 256-dimensional feature vector from infrared thermal imaging, and the 256-dimensional feature vector from depth point clouds to form a 1024-dimensional fused feature vector. The scene classification network consists of two fully connected layers, with a dropout layer in between to prevent overfitting. The output layer of this network is a Softmax layer, with the number of neurons equal to the total number of preset scene categories, such as 15 categories including indoor, outdoor roads, warehouses, parking lots, and wilderness. The scene parsing unit selects the category with the highest probability in the Softmax output as the current scene category identifier. Furthermore, the scene parsing unit calculates a disorder index characterizing scene complexity, defined as the entropy value of the probability distribution of the Softmax output. A higher entropy value indicates greater uncertainty in the network's judgment of the scene category, indirectly reflecting the complexity and atypicality of the scene content. Scene category identifiers and confusion index together constitute the semantic analysis results of the current environment.

[0031] The model selection and loading unit is the decision-making and execution endpoint of the dynamic model adaptation engine. This unit receives real-time available computing power scores from the resource monitoring unit, and scene category identifiers and disorder indices from the scene parsing unit. Internally, the model selection and loading unit maintains a predefined policy mapping table, which is a three-dimensional lookup table. The first dimension index is the computing power score range; for example, 0 to 30 points defines a low computing power range, 31 to 70 points a medium computing power range, and 71 to 100 points a high computing power range. The second dimension index is the scene category identifier. The third dimension index is the disorder index range; for example, 0 to 1.0 represents low disorder, and 1.1 to 2.5 represents high disorder. Each cell in the policy mapping table stores triple information, including the version identifier of the recommended lightweight multimodal recognition sub-model in the cloud model repository, the estimated memory usage of the model, and its estimated inference latency. This mapping table is pre-generated and distributed in the cloud through offline simulation and historical performance data analysis. The model selection and loading unit searches the mapping table based on the input triples to determine the target model version identifier. Subsequently, it initiated a model retrieval request to the model repository and version management module in the cloud collaboration and optimization subsystem.

[0032] The request message contains the edge node's identity, the target model version identifier, and a local model cache checksum. If the requested model version already exists in the edge node's local cache and the checksum matches, it is loaded directly. Otherwise, the cloud transmits the corresponding model file serialized data stream to the edge. The model file contains the model structure definition, weight parameters, and initialization configuration. The model selection and loading unit parses this data stream in memory, instantiates the model, and loads the weights into the graphics processor or dedicated neural network processing unit, preparing for the inference task.

[0033] The local decision fusion module receives preliminary recognition results for each modality from a lightweight multimodal recognition sub-model instantiated by the dynamic model adaptation engine. This lightweight multimodal recognition sub-model takes the intermediate multimodal feature representations generated by the front-end feature extraction module as input. Internally, it contains multiple parallel branch networks that process features from different modalities, performing feature interaction and fusion in intermediate layers, ultimately outputting an independent preliminary recognition result for each modality. For each detected target object, the preliminary recognition result for each modality includes a target category label and a probability vector output by the model's Softmax layer. The fusion process of the local decision fusion module consists of three steps.

[0034] Step 1: Extract the raw confidence scores for each modality. For each modality, take the maximum value in the modality's probability vector as the raw confidence score of the target recognition result for that modality. This value is a floating-point number between 0 and 1.

[0035] Step 2: Obtain and apply modal static weight coefficients. The local decision fusion module maintains a modal priority configuration table issued by the cloud-based scene knowledge graph module. This table uses the scene category identifier as an index and assigns static weight coefficients to each perceptual modality. For example, for an indoor scene, the weight coefficient for the visible light modality might be 0.5, the depth modality 0.3, and the infrared modality 0.2; for a nighttime outdoor scene, the weight coefficient for the infrared modality might increase to 0.6, the visible light modality decrease to 0.2, and the depth modality remain at 0.2. The sum of these coefficients is 1. The local decision fusion module looks up the corresponding weight coefficient in the table based on the current scene category identifier.

[0036] Step 3: Calculate the weighted confidence score and final decision. Multiply the original confidence score of each modality by its corresponding static weight coefficient to obtain the weighted confidence score for that modality. Then, sum the weighted confidence scores of all modalities for the same target object to obtain the overall confidence score.

[0037] Finally, the local decision fusion module selects the target category with the highest comprehensive confidence score as the final local identification decision for the object, and outputs the decision along with its comprehensive confidence score, target bounding box or location information for use by subsequent edge applications.

[0038] Please refer to the attached document. Figure 1 The cloud-based collaboration and optimization subsystem serves as the system's intelligent hub and evolutionary engine. It comprises a model repository and version management module, a scene knowledge graph module, and an incremental model distillation module. The model repository and version management module is a distributed file database and version control system. It stores a series of lightweight multimodal recognition sub-models with varying computational complexity and recognition accuracy. Each model has a unique version identifier, with version numbers following a semantic versioning rule of major version number, minor version number, and revision number. The model repository records metadata for each model, including creation time, source of the underlying teacher model, number of parameters, accuracy metrics on the standard test set, and the expected minimum resource configuration requirements for edge devices. The version management module handles model retrieval requests from edge nodes, performing version comparisons and incremental updates. When a new model is added to the repository, the version management module pushes an update notification to all edge nodes that have subscribed to the relevant model series.

[0039] The scene knowledge graph module is used to build and maintain a structured knowledge network centered on scene semantics. Please refer to the appendix. Figure 4 This module is specifically operated by a graph construction unit and a strategy recommendation unit. The graph construction unit builds the graph offline based on massive historical multimodal recognition task logs. These logs come from numerous edge nodes deployed globally, and the log entries include the feature hash values ​​of scene image snapshots, recognition task types, sensor modality combinations used, loaded model versions, recognition results and their confidence levels, as well as the final manual verification results or high-confidence automatic annotation results.

[0040] The knowledge graph construction unit first employs entity extraction technology to automatically identify frequently occurring scene entities, object entities, event entities, and environmental condition entities from the logs. For example, scene entities include "intersection" and "supermarket shelf"; object entities include "pedestrian" and "cardboard box"; event entities include "people gathering" and "vehicle changing lanes"; and environmental condition entities include "low light" and "rain / snow". Subsequently, through relation mining technology, the co-occurrence frequency and conditional probability of these entities are statistically analyzed to establish directed relation edges. Relationship types include "occurred at", "contains", "affected by environment", "preferred perceptual modality is", and "optimal model configuration is". All entities and relations constitute a massive attribute graph, stored in a graph database. This knowledge graph continuously evolves, incrementally updated daily based on new log data, adding new entities and adjusting relation weights.

[0041] The strategy recommendation unit provides online services. It is triggered when the edge perception and processing subsystem uploads its scene category identifier. Within the knowledge graph, it performs a multi-hop retrieval using the scene category identifier as the root node. The retrieval process identifies a list of all typical object entities directly associated with the scene, and ranks the perceptual modalities that historically contributed the most to successfully identifying these objects in that scene, i.e., modal priority. Simultaneously, it retrieves the identifier of the lightweight model version with the highest average recognition accuracy across different computing power score ranges within that scene.

[0042] The strategy recommendation unit encapsulates these search results into a structured strategy recommendation package. This package mainly consists of two parts: first, a modality priority configuration table for the specific scenario, explicitly providing suggested static weight coefficient values ​​for each modality; and second, a simplified model recommendation strategy table, which may make scenario-specific fine-tuning or supplements to the global strategy mapping table. This strategy recommendation package is distributed to the requesting edge node via a secure link, updating the relevant configurations of its local decision fusion module and dynamic model adaptation engine, thus enabling cloud-based knowledge to reach the edge.

[0043] The incremental model distillation module is the core of achieving continuous model optimization. Please refer to the appendix. Figure 5 Its workflow specifically includes a sample screening and desensitization unit, a teacher model fine-tuning unit, and a student model distillation unit. The sample screening and desensitization unit runs a message queue service in the cloud, receiving data proactively reported by the edge perception and processing subsystem. The reporting trigger condition is that the final comprehensive confidence score output by the local decision fusion module is lower than a preset confidence threshold, such as 0.7. Such samples are defined as hard samples. The edge nodes do not report the original image or point cloud, but rather a desensitized data packet. This data packet contains three parts: first, the scene category identifier that triggers the reporting; second, the multimodal intermediate feature representation output by the front-end feature extraction module corresponding to the target object that leads to low confidence; and third, the object's true label or high-confidence pseudo-label. The true label may come from subsequent manual review processes, while the high-confidence pseudo-label may come from cross-validation of other high-confidence recognition results within the same time period or from parallel inference results of high-precision models in the cloud. This design ensures that the original sensitive data does not leave the edge; only abstract feature vectors are transmitted.

[0044] The teacher model fine-tuning unit maintains a fully parameterized, high-precision multimodal recognition teacher model. This model has a large architecture, high accuracy, and runs on powerful cloud computing resources. Once the sample screening and desensitization unit has accumulated a certain batch of difficult sample features and their labels, the teacher model fine-tuning unit initiates fine-tuning training. Training employs mini-batch gradient descent with a low learning rate, such as 0.0001, to prevent catastrophic forgetting of existing knowledge. The loss function used is the standard cross-entropy loss function. The purpose of fine-tuning is to enable the teacher model to correctly classify these difficult patterns that are difficult to distinguish at the margins, thereby updating its decision boundary.

[0045] The student model distillation unit is crucial for generating next-generation lightweight models. It selects a lightweight student model from the model repository as the distillation target. The student model is typically a highly compressed version of the teacher model, employing pruning, quantization, or a more efficient neural network architecture. The distillation process is guided by the fine-tuned teacher model. Specifically, feature representations of the same batch of difficult samples are simultaneously input into both the teacher and student models. The distillation loss function consists of two parts. The first part is the soft-label loss function, using Kullback-Leibler divergence to measure the difference between the class probability distributions output by the student model and those output by the teacher model. The teacher model's Softmax output is softened by a high temperature parameter, containing rich information about inter-class similarity. The second part is the intermediate layer feature map matching loss function, which selects certain corresponding intermediate layer output feature maps from the teacher and student models and calculates their mean squared error loss, forcing the student model to mimic the internal feature representations of the teacher model. The total loss function is a weighted sum of these two losses:

[0046]

[0047] in, and These represent the softened probability distributions of the teacher model and the student model outputs, respectively. This represents the Kullback-Leibler divergence. and Representing the teacher model and student model respectively The selected intermediate layer feature maps, This is the number of selected intermediate layers. and The weights are used to balance the two losses and are typically determined through adjustment on the validation set. By minimizing this total loss function, the student model is trained to mimic the behavior and internal representation of the teacher model, thereby transferring knowledge learned from difficult samples. After training, a new version of the student model with improved performance is generated and released via the model repository and version management module. Finally, it is deployed to edge nodes through policy recommendation or model pull mechanisms, completing an iterative upgrade of the model's capabilities.

[0048] Please refer to the attached document. Figure 4 The entire system operates within a precisely defined hierarchical decision-making and collaborative update framework. On a millisecond-level timescale (less than 10 milliseconds), the edge perception and processing subsystem, as an autonomous unit, independently and continuously executes a complete closed loop from sensor data acquisition, front-end feature extraction, dynamic model inference to local decision fusion. This closed loop does not rely on any real-time cloud feedback, ensuring the extremely low-latency real-time response required for applications such as autonomous driving obstacle avoidance and industrial robot grasping. On a second- to minute-level timescale, the system initiates asynchronous collaboration. During local decision-making intervals, edge nodes, through background threads, package and upload difficult sample features to the message queue of the incremental model distillation module in the cloud.

[0049] Meanwhile, edge nodes periodically, or when the scene undergoes significant changes, query and receive updated modality priority configuration tables and model recommendation strategies from the cloud-based scene knowledge graph module. These interactions are non-blocking and do not affect the main recognition process, which operates on a millisecond-level timescale. On a timescale ranging from hours to days, the cloud-based collaboration and optimization subsystem initiates batch computing tasks.

[0050] The incremental model distillation module uses hard samples collected over a period of time to fine-tune the teacher model and distill the student model, producing new models. The scene knowledge graph module performs offline analysis on millions of new recognition logs, updating entities, relationships, and statistical weights in the graph. New model versions are then released in the model repository. These updates are gradually pushed through periodic strategies or actively pulled by edge nodes, gradually permeating the globally deployed edge node network, driving the continuous and collaborative evolution of the entire system's recognition capabilities. This multi-timescale coupled architecture perfectly balances real-time performance, adaptability, and continuous learning capabilities.

Claims

1. An AI multimodal visual recognition system based on edge computing, characterized in that, Deployed in a distributed computing environment containing at least one edge computing node and a cloud server, including: An edge sensing and processing subsystem deployed on an edge computing node, and a cloud collaboration and optimization subsystem deployed on a cloud server; the edge sensing and processing subsystem and the cloud collaboration and optimization subsystem interact with each other through a preset secure communication protocol; The edge perception and processing subsystem includes a multimodal sensor array, a front-end feature extraction module, a dynamic model adaptation engine, and a local decision fusion module. The multimodal sensor array is used to simultaneously acquire sensing data of at least two different physical properties of the target scene, including visible light images, infrared thermal imaging, and depth point clouds. The front-end feature extraction module is used to preprocess and extract preliminary features from the raw sensing data collected by the multimodal sensor array to generate intermediate feature representations corresponding to each modality. The dynamic model adaptation engine is used to dynamically load and instantiate a lightweight multimodal recognition sub-model that matches the available computing resources of the edge computing nodes monitored in real time and the complexity assessment results of the current scene from a preset model library. The local decision fusion module is used to fuse the preliminary identification results of each modality output by the lightweight multimodal recognition sub-model instantiated by the dynamic model adaptation engine according to the preset confidence weighting rules to generate the final local recognition decision. The cloud-based collaboration and optimization subsystem includes a model repository and version management module, a scene knowledge graph module, and an incremental model distillation module. The model repository and version management module is used to store, version manage, and distribute a series of lightweight multimodal recognition sub-models with different computational complexity and recognition accuracy. The scene knowledge graph module is used to construct and maintain a structured knowledge network with scene semantics as its core. This network associates typical objects, events, environmental attributes and their corresponding preferred perceptual modal combinations and model configuration strategies in different scenes. The incremental model distillation module is used to periodically receive the desensitized features of difficult samples and their corresponding local fusion decision results uploaded by the edge perception and processing subsystem, and use them as supervision signals to perform incremental training and knowledge distillation on the teacher models stored in the model repository, thereby generating a new generation of lightweight student models and updating them to the model repository. The dynamic model adaptation engine includes a resource monitoring unit, a scene parsing unit, and a model selection and loading unit. The resource monitoring unit is used to continuously monitor the current available memory, CPU utilization, GPU utilization, and estimated battery life of the edge computing node, and to calculate a quantitative real-time available computing power score. The scene parsing unit is used to receive the intermediate feature representation output by the front-end feature extraction module, and perform semantic parsing on the current scene based on the pre-trained lightweight scene classification network, and output the scene category identifier and the confusion index representing the complexity of the scene. The model selection and loading unit is used to receive the real-time available computing power score, the scene category identifier, and the disorder index. The model selection and loading unit has a built-in predefined strategy mapping table, which combines different computing power score ranges with scene categories and disorder indices, and maps them to specific lightweight multimodal recognition sub-model version identifiers in the model repository. The model selection and loading unit initiates a model retrieval request to the cloud collaboration and optimization subsystem according to the mapping result, and after receiving the model data, completes the instantiation and initialization of the model in the memory of the edge computing node.

2. The AI ​​multimodal visual recognition system based on edge computing according to claim 1, characterized in that, The specific process by which the front-end feature extraction module processes multimodal sensing data is as follows: For visible light image data, preprocessing operations including white balance correction, gamma correction and resolution normalization are performed, and then a lightweight convolutional neural network is used to extract its spatial texture features. For infrared thermal imaging data, preprocessing operations including non-uniformity correction and temperature value calibration are performed, and then a convolutional network focused on gradient feature extraction is used to extract its temperature distribution features. For deep point cloud data, preprocessing operations including outlier filtering and voxel mesh downsampling are performed, followed by the extraction of its geometric features using a lightweight point convolution-based network. Among them, the feature extraction networks of each modality share some of the underlying convolutional kernel parameters.

3. The AI ​​multimodal visual recognition system based on edge computing according to claim 2, characterized in that, The confidence-weighted rule generation process of the local decision fusion module is as follows: For each target object's preliminary recognition result output by the lightweight multimodal recognition sub-model, the maximum value in the probability vector output by the softmax layer of the model corresponding to the result is extracted as the original confidence level of the recognition result of the target object for that modality. Based on the modality priority configuration table for the current scene category identifier issued by the scene knowledge graph module, a static weight coefficient is assigned to each perceptual modality; The weighted confidence level is obtained by multiplying the original confidence level of each modality by its corresponding static weight coefficient. The weighted confidence scores of all modalities for the same target object are summed, and the target category with the highest summation value is determined as the final local recognition decision.

4. The AI ​​multimodal visual recognition system based on edge computing according to claim 3, characterized in that, The incremental model distillation module includes a sample screening and desensitization unit, a teacher model fine-tuning unit, and a student model distillation unit. The sample screening and desensitization unit is used to receive difficult sample data actively reported by the edge perception and processing subsystem. The criterion for judging difficult samples is that the weighted confidence summation value finally output by the local decision fusion module is lower than the preset confidence threshold. The sample screening and desensitization unit performs desensitization processing on the original perception data of difficult samples, retaining only the intermediate feature representation output by the front-end feature extraction module and the corresponding real or high-confidence pseudo-labels. The teacher model fine-tuning unit is used to perform small-batch, low-learning-rate fine-tuning training on the full-parameter teacher model stored in the cloud using the de-identified features and labels of difficult samples. The student model distillation unit is used to perform knowledge distillation on the lightweight student model selected from the model repository using the fine-tuned teacher model. The knowledge distillation process adopts a multi-task loss function that combines the soft label loss function and the intermediate layer feature map matching loss function to transfer the knowledge of the teacher model to the student model.

5. The AI ​​multimodal visual recognition system based on edge computing according to claim 4, characterized in that, The multi-task loss function is composed of a weighted sum of a soft label loss function term and an intermediate layer feature map matching loss function term. The soft-label loss function term is used to measure the difference between the class probability distribution output by the student model and the class probability distribution output by the teacher model; The intermediate layer feature map matching loss function term is used to measure the difference between the intermediate layer output feature maps corresponding to the student model and the teacher model.

6. The AI ​​multimodal visual recognition system based on edge computing according to claim 5, characterized in that, The scenario knowledge graph module includes a graph construction unit and a strategy recommendation unit; The graph construction unit is built based on massive historical multimodal recognition task logs. Through entity extraction and relationship mining techniques, it associates scenes, objects, events, environmental conditions, effective modal combinations, and historical optimal model configurations to form a queryable graph structure database. The strategy recommendation unit responds to the scene category identifier uploaded by the edge perception and processing subsystem by searching the graph structure database to find a list of typical objects strongly associated with the scene, the main modalities on which historical successful identifications depend, and the lightweight model version identifier that performs best under different computing power conditions in the scene. It then encapsulates this information into a modality priority configuration table and a model recommendation strategy and sends it to the corresponding edge computing nodes.

7. The AI ​​multimodal visual recognition system based on edge computing according to claim 6, characterized in that, The resource monitoring unit has a built-in quantitative scoring function that maps four indicators—available memory, CPU utilization, GPU utilization, and estimated battery life—to a real-time available computing power score between 0 and 100. The calculation process of the quantitative scoring function is as follows: a benchmark value is set for each indicator, the proportion or inverse proportion of each indicator relative to its benchmark value is calculated, and a weighted sum is performed. Finally, the weighted sum is normalized to 0 to 100 points through the Sigmoid function.

Images