Edge vision model optimization method and system based on dynamic reasoning kernel

Abstract

The invention discloses an edge vision model optimization method and system based on a dynamic reasoning kernel, and the method specifically comprises the steps: firstly, generating a cross-modal task prototype for a small number of new target example images; then, directly mapping the prototype through a parameter projection network based on a neural radiation field principle, and generating a task-exclusive dynamic deformable reasoning kernel; and then, combining a pseudo tag generated by self-supervision, adopting a parameter isolation updating strategy, binding the reasoning kernel as a hard updating parameter to a specified layer of a pre-training model, and only carrying out rapid gradient updating on a small number of key parameters of the model. According to the method, the edge end closed loop of'example input-local evolution 'is constructed, the second-level evolution of the model can be realized without data backflow and manual annotation, the new task detection precision is remarkably improved, meanwhile, the forgetting of old knowledge is effectively inhibited, and the method has the advantages of low adaptation cost, high response speed and high safety.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and computer vision, in particular to an edge visual model optimization method and system based on a dynamic inference kernel, which is used for quickly and efficiently optimizing visual models on edge computing devices. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, intelligent vision technology has been widely applied in industrial quality inspection, security monitoring, autonomous driving, smart retail and many other fields. In these scenarios, models are usually deployed on edge devices with limited computing power, storage and power consumption to meet the needs of low latency and data privacy. However, traditional intelligent vision systems face serious challenges in edge applications.

[0003] The pain points of existing technologies mainly lie in the following aspects: Small sample detection accuracy bottleneck: Traditional target detection models such as YOLOv8, Faster R-CNN, etc. usually rely on large-scale labeled data sets for training. When a new class needs to be quickly adapted on site (such as a new type of industrial product defect), only a small number of samples can be collected. Under such small sample conditions, the average precision (AP) of the model will decrease significantly after fine-tuning, making it difficult to meet production requirements. Usually, at least 100 labeled samples are needed to achieve a relatively ideal fine-tuning effect.

[0004] Data backflow dependence and risk: Existing incremental learning or continuous learning frameworks such as MARS, while capable of handling new data, generally adopt the "edge collection-cloud training-edge update" mode. This mode is heavily dependent on network connection and has at least 200ms of network delay, which cannot meet the real-time response requirement. More importantly, the data collected on site (such as industrial defect images containing commercial secrets) is transmitted back to the cloud, posing a huge risk of data privacy leakage and incurring high cloud computing costs.

[0005] Poor timeliness in on-site adaptation: In actual delivery scenarios, when encountering new targets or new tasks (such as discovering a new type of equipment defect in a substation inspection), the traditional process requires "on-site data collection-> manual labeling-> data transmission-> cloud retraining-> model deployment". The entire response cycle takes an average of more than 7 days, which completely fails to meet the real-time and agility requirements of modern production.

[0006] In addition, some related technologies proposed by the academic community also have limitations: Meta-learning approaches (e.g., MAML, Reptile): Although MAML is designed for fast adaptation to new tasks, it usually requires dozens of iterations (more than 10 seconds) to converge, which is difficult to achieve real-time application on edge devices with limited computing power. While Reptile and other algorithms can quickly adapt, they can also cause catastrophic forgetting of old tasks, with a forgetting rate of more than 15%.

[0007] Lightweight fine-tuning frameworks (e.g., TinyCLIP): Although these frameworks compress the model, they still require a complete backpropagation process during fine-tuning. On typical embedded devices such as Raspberry Pi 4B, a single parameter update takes more than 5 seconds, which cannot meet the requirements of second-level evolution.

[0008] Parameter-free example inference (e.g., CLIP): CLIP models rely on their powerful pre-trained semantic alignment capabilities for zero-shot inference, but for fine-grained categories that have not been seen during training and lack explicit textual descriptions (e.g., "industrial gear 2 crack"), there is a semantic gap, and the direct application of detection accuracy is extremely low (e.g., 32%), which cannot be reliably used in industrial scenarios.

[0009] Therefore, there is an urgent need for a new technical solution that does not require data backflow, can utilize a small amount of samples, achieve second-level model evolution on edge devices, and effectively avoid forgetting old knowledge. SUMMARY

[0010] The present application mainly solves the technical problems of existing technologies, such as small sample detection precision bottleneck, data backflow dependence, and poor timeliness of on-site adaptation, and provides an edge visual model optimization method and system based on dynamic inference kernel with zero data backflow adaptation, second-level evolution capability, lightweight, and low forgetting.

[0011] The core of the present application is to construct a "example input-real-time inference-local evolution" closed-loop application paradigm, which is mainly supported by three major innovative points: dynamic example feature binding engine (DSFBE), edge incremental parameter surgery mechanism, and self-supervised example effectiveness verification framework.

[0012] Firstly, the present application proposes a dynamic example feature binding engine (DSFBE) for efficiently injecting information of a small number of examples into a pre-trained model. The engine constructs a three-stage processing pipeline: Semantic-visual dual prototype generation: Use DINOv2 lightweight version to extract example visual features, and combine CLIP text encoder to generate cross-modal prototype pair (P_v, P_t).

[0013] Deformable inference kernel generation: Based on the principle of Neural Radiance Fields (NeRF), prototype pairs are encoded into task-specific deformable convolution kernels K_task and bound to the ROIAlign layer of the model. This binding process is achieved through a "dynamic parameter injection mechanism": K_task is parsed into a spatial offset matrix (Δx, Δy) and a weight adjustment coefficient W; during the feature sampling stage of the ROIAlign layer, (Δx, Δy) is injected into the sampling point coordinate calculation, and the original coordinates (x, y) are adjusted to (x+Δx_i,j, y+Δy_i,j), and the feature values of the adjusted sampling points are weighted W. This binding process achieves real-time parameter loading through memory mapping files, without the need to recompile the network structure, and the binding time is ≤10 ms.

[0014] Dual-branch inference routing: The global pre-trained model and the example subgraph branch loaded with K_task are run in parallel, and the outputs are fused through an attention gating mechanism. The fusion process consists of four steps: feature extraction, attention weight calculation, feature fusion (using the weighted fusion formula , and output correction (applying batch normalization BN layer). This mechanism can improve the detection confidence of new classes by 25%, and the total fusion time is ≤30 ms.

[0015] Secondly, in order to quickly adapt to new tasks while avoiding catastrophic forgetting, the invention designs an edge incremental parameter surgery mechanism and proposes a ternary parameter isolation update strategy: Frozen layer (80% parameters): The pre-trained visual backbone network (such as Swin-T) is fixed, retaining its powerful general feature extraction capability.

[0016] Soft update layer (17% parameters): The cross-modal fusion module is updated using exponential moving average (EMA) smoothing to ensure smooth transition of new and old knowledge, with a forgetting rate <3%.

[0017] Hard update layer (3% parameters): Only the deformable convolution kernel K_task bound to the example and the final classification head are updated. Through local gradient masking technology, the update is achieved in seconds.

[0018] Furthermore, to achieve automatic evolution without human intervention, the invention constructs a self-supervised example effectiveness verification framework. The framework designs an entropy-value and diversity double-threshold filtering mechanism and integrates an automatic annotation generator: Entropy screening: When the predicted entropy value is >1.8, it is determined to be an effective new class, avoiding repeated learning of known classes.

[0019] Feature diversity measurement: Redundant examples with a similarity >0.9 are removed through local feature PCA projection to ensure that 5 examples cover 85% of the feature space.

[0020] Automatic label generation: a pseudo-label generator based on teacher-student distillation, which can generate high-quality labels for effective examples without manual annotation for model updating. The teacher model is a pre-trained general-purpose model for edge deployment (such as DINOv2+FPN), and the student model is a lightweight inference model of the system. The student model is optimized by the distillation loss L_distill=L1(B_stu,B_tea)+CrossEntropy(P_stu,P_tea).

[0021] The specific scheme of the present application is: an edge visual model optimization method based on a dynamic inference kernel, comprising the following steps: S1: cross-modal task prototype generation: For at least one example image containing a new target and its corresponding optional text description, use a pre-trained visual model (such as Vision Transformer, ViT) to extract N-dimensional visual features, and use a pre-trained text encoder (such as CLIP-Text) to extract M-dimensional text features; the N-dimensional visual features are clustered using a Gaussian Mixture Model (GMM-Clustering) to generate a visual prototype P_v representing the core visual attributes of the new target; the visual prototype P_v and the text prototype P_t generated from the M-dimensional text features are combined to form a cross-modal task prototype (P_v, P_t); generally, N can be 2048, M can be 768, and P_v and P_t are concatenated to form a 2816-dimensional cross-modal task model; S2: Dynamic deformable inference kernel generation: The cross-modal task prototype (P_v, P_t) is input into a parameter projection network based on the principle of Neural Radiance Field (NeRF); the parameter projection network maps the task prototype to a task-specific spatial grid parameter θ_grid, and parses a dynamic deformable inference kernel K_task from the θ_grid; the K_task includes a spatial offset matrix (Δx, Δy) for adjusting feature sampling position and a weight coefficient matrix W for adjusting feature value; S3: Self-supervised example verification and pseudo-label generation: inputting the example image into a visual model currently deployed on the edge device, calculating an entropy value H of a predicted class probability distribution of the visual model, and determining that the example image is a valid new class example image when the entropy value H is greater than a preset entropy value threshold (e.g., 1.8); generating a pseudo label for the valid new class example image by using a Teacher-Student distillation framework, where a Teacher model is a general model with high performance but slow inference that is pre-stored on the edge, and a Student model is a lightweight edge visual model to be optimized; and the Teacher model outputs a high-confidence bounding box coordinate B_tea and a class c_tea as the pseudo label; S4: Incremental update based on parameter isolation strategy: According to a predefined parameter mask matrix M, the parameters of the Student model are divided into frozen layers (M[i]=0), soft update layers (M[i]=0.3), and hard update layers (M[i]=1); the dynamic deformable inference kernel K_task generated in step S2 is bound to a region of interest alignment (ROIAlign) layer of the Student model as a hard update parameter; and based on the pseudo label B_tea and c_tea generated in step S3 and the prediction result B_stu and P_stu of the example image by the Student model, the overall loss function L(θ;D_new) is calculated according to the following formula: L(θ;D_new)=α·L_cls(P_stu,c_tea)+β·L_reg(B_stu,B_tea); where L_cls is a classification loss, L_reg is a regression loss, and α and β are preset weight coefficients. The parameters θ of the Student model are updated according to the following update rule: ; where Δθ is a parameter update amount, η is a learning rate, is a gradient of the loss function; only the parameters of the soft update layers and the hard update layers (M[i]≠0) are updated, so that the general knowledge is retained while quickly adapting to the new target.

[0022] Preferably, in step S2, the parameter projection network includes at least three fully connected layers, and the parameter projection network maps the input cross-modal task prototype into a spatial grid parameter θ_grid with a dimension of 4×4×(3+1), where 3 represents the dimension of the convolution kernel weight, and 1 represents the dimension of the offset; and the resolved spatial offset matrix (Δx, Δy) has a size of 4×4×2.

[0023] Preferably, in step S3, the entropy value H is calculated by the following formula: H = -∑(p_i×log(p_i)); where p_i is the predicted probability of the model for the i-th class.

[0024] As preferred, the step S3 further includes a feature diversity measurement step: performing principal component analysis (PCA) dimension reduction on the local feature vectors of the plurality of valid new class examples, and calculating the cosine similarity S between the reduced features, when S is less than a preset similarity threshold (such as 0.9), the example is retained to avoid sample redundancy.

[0025] As preferred, in the step S4, the parameter mask matrix M defines no more than 3% of the model parameters as a hard update layer, and 10%-20% of the parameters as a soft update layer; the soft update layer is updated by exponential moving average (EMA) smoothing, and the hard update layer is updated by gradient using the RMSprop optimizer.

[0026] An edge visual model optimization system based on a dynamic inference kernel, comprising: A cross-modal task prototype generation module is configured to extract visual features and text features from at least one example image containing a new target and its corresponding optional text description, and generate a cross-modal task prototype (P_v, P_t) through Gaussian mixture model clustering and other processing. A dynamic deformable inference kernel generation module is configured with a parameter projection network constructed based on the principle of neural radiance field (NeRF), and is configured to receive the cross-modal task prototype and map it to generate a dynamic deformable inference kernel K_task containing a spatial offset matrix (Δx, Δy) and a weight coefficient matrix W. A self-supervised verification and labeling module is configured to calculate the prediction entropy of the input example to verify whether it is a new class, and to use a built-in Teacher-Student distillation framework to automatically generate pseudo-labels (B_tea, c_tea) for supervised learning for the verified examples. A parameter isolation update module is configured to divide the Student model parameters to be optimized into three categories: frozen, soft update, and hard update, according to a predefined parameter mask matrix M; and bind the dynamic deformable inference kernel K_task as a hard update parameter to a specified layer of the model, calculate the loss according to the pseudo-labels, and only perform gradient update on the non-frozen parameters to complete the optimization of the model.

[0027] As preferred, the parameter projection network in the dynamic deformable inference kernel generation module takes the cross-modal task prototype as input and outputs a task-specific spatial grid parameter θ_grid, and the module further parses the dynamic deformable inference kernel K_task from θ_grid.

[0028] Preferably, in the parameter isolation update module, the learning rate of the hard update parameter (e.g., 0.01) is higher than that of the soft update parameter (e.g., 0.0001), and the hard update parameter is updated directly through gradient descent, while the soft update parameter is updated through exponential moving average (EMA).

[0029] From a system architecture perspective, this system can be divided into four hierarchical functional layers: the input layer, the DSFBE engine, the edge evolution layer, and the inference decision layer.

[0030] Input layer: Supports real-time video streaming via USB / industrial camera, or batch import of sample images in JPEG / PNG format. Text descriptions (e.g., "gear tooth surface crack") can be included.

[0031] The DSFBE engine includes: Example processor: Extracts 2048-dimensional visual features and 768-dimensional text embeddings, aligning cross-modal sequences via Dynamic Time Warping (DTW).

[0032] Prototype generator: Generates 32 cross-modal prototypes by clustering example features, and uses a topology protection mechanism to prevent feature drift.

[0033] Inference kernel generator: Encodes prototype pairs into 4×4 deformable convolution kernels (parameters only 128KB) and loads them directly into the inference network.

[0034] The marginal evolutionary layer includes: Parameter isolator: The frozen layer, soft update layer, and hard update layer are divided by a binary mask matrix M (M[i]∈{0,0.3,1}) with the same dimension as the model parameters. This mask matrix is ​​determined through offline sensitivity analysis.

[0035] Incremental optimizer: The RMSprop algorithm is used, with a learning rate of 0.01 for hard update layers and 0.0001 for other layers, and ≤5 iterations per update.

[0036] Hardware acceleration unit: Adapted to NPU operators such as Ascend 310 and NVIDIA Jetson AGX Orin, achieving update time ≤200ms.

[0037] The reasoning and decision-making layer includes: Dual-branch fusion: The outputs of the global model branch and the example subgraph branch are fused using attention-gated weighted fusion.

[0038] Uncertainty estimator: Calculates the predicted entropy value and feature outlier, and automatically triggers example collection prompts (such as "New category detected, suggest taking 3 examples").

[0039] This invention enables zero-data backflow adaptation: model optimization can be completed directly on an edge device using only 5-10 sample images collected on-site, without any cloud intervention or manual annotation.

[0040] Second-level capability evolution: After receiving a new example image, the model parameters can be updated within 200 milliseconds (ms), which improves the average accuracy (AP) of the model for new categories by more than 15%.

[0041] Lightweight and low forgetting rate: Only about 3% of the key parameters in the model are updated, ensuring that the forgetting rate of old tasks is less than 3% while adapting to new tasks, and the memory usage of the entire system on edge devices does not exceed 1GB, making it suitable for embedded systems.

[0042] The substantial effects of this invention are: Achieving true edge-closed-loop evolution: This invention pioneered the construction of an application paradigm of "example input - real-time inference - local evolution," where all computations are completed on edge devices, completely eliminating dependence on the cloud and fundamentally solving the problems of network latency, data privacy, and cloud costs.

[0043] This significantly reduces adaptation costs and time: requiring only 5-10 sample images taken by on-site personnel, without any professional annotation, the system can automatically complete model optimization within 200ms. Compared to the traditional solution's time-consuming manual annotation and retraining cycle of several days, this invention reduces on-site adaptation manpower costs by more than 90%, and compresses time costs from days to seconds.

[0044] Significantly improves the performance and efficiency of small-sample learning: By dynamically generating task-specific inference kernels and combining them with a precise parameter isolation and update strategy, this invention can improve the detection AP of new categories by more than 15% with only a few samples, while ensuring that the forgetting rate of the original task is less than 3%, thus achieving an effective balance between "fast learning" and "long-term memory".

[0045] The solution is highly versatile and portable: the optimization method of this invention can be used as a "plugin" or "adapter" to be combined with various mainstream pre-trained visual models, and the system design is lightweight and compatible with mainstream embedded hardware (such as NVIDIA Jetson series), with broad application prospects. Attached Figure Description

[0046] Figure 1 This is a flowchart of an edge vision model optimization method based on a dynamic inference kernel according to the present invention. Detailed Implementation

[0047] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0048] Example: This example provides an edge vision model optimization system based on a dynamic inference kernel. The system is deployed on an edge computing device, such as a terminal consisting of an industrial smart camera and an edge computing box (e.g., Intel NUC + NVIDIA RTX A2000). Figure 1 As shown, the optimization method includes four steps: cross-modal task prototype generation, dynamic deformable inference kernel generation, self-supervised example verification and pseudo-label generation, and incremental update based on parameter isolation strategy. The system also includes the following modules: cross-modal task prototype generation module, dynamic deformable inference kernel generation module, self-supervised verification and labeling module, and parameter isolation update module.

[0049] The function of the cross-modal task prototype generation module is to receive new task examples from external input and transform them into highly condensed feature representations that the model can understand—that is, cross-modal task prototypes.

[0050] In one specific implementation, when the user inputs N=5 example images of a new category (e.g., "novel insulator crack"), the cross-modal task prototyping module performs the following operations: Feature Extraction: For each example image, a powerful pre-trained visual model, preferably a lightweight version of VisionTransformer (ViT) (such as DINOv2), is used to extract its global features. ViT effectively captures the contextual information of the image by dividing the image into multiple patches and utilizing the self-attention mechanism of the Transformer architecture. In this embodiment, the extracted visual features are 2048-dimensional vectors. If the user also inputs the text description "insulator crack," the CLIP model's text encoder (CLIP-Text) is used to convert it into a 768-dimensional text feature vector.

[0051] Visual Prototype Generation: Only a few samples may have visual features that are noisy or have different viewpoints. To obtain a stable and representative visual center, the cross-modal task prototype generation module uses a Gaussian Mixture Model (GMM-Clustering) to cluster the five 2048-dimensional visual feature vectors. GMM can fit complex data distributions with a weighted sum of multiple Gaussian distributions, and compared to hard clustering methods such as K-Means, it is better able to capture the uncertainty of features. The result of clustering is the generation of one or more visual prototypes P_v, which are the centers of the distribution and represent the core visual patterns of the new category.

[0052] Prototype Combination: The generated visual prototype P_v is concatenated with the textual prototype P_t (if any) to form a 2048+768=2816-dimensional cross-modal task prototype (P_v, P_t). This prototype contains both the "appearance" (visual information) and "definition" (semantic information) of the new target, providing comprehensive guidance for the subsequent generation of customized inference kernels.

[0053] To describe this process more formally, the following key formulas and model definitions are introduced: Formula: P_v=GMM-Clustering(ViT(x_i)),P_t=CLIP-Text(y_i); Usage steps: For the input example image x_i, extract the image features using a Visual Transformer (ViT).

[0054] The extracted visual features are processed using Gaussian mixture model clustering (GMM-Clustering) to generate a visual prototype P_v.

[0055] The input text description y_i is processed by the CLIP model's text encoder (CLIP-Text) to generate the text prototype P_t.

[0056] Related model description: GMM-Clustering (Gaussian Mixture Model Clustering): Gaussian mixture model clustering generates visual prototypes by fitting feature distributions through multiple Gaussian probability density functions.

[0057] ViT (Vision Transformer): A visual Transformer model used to extract image patch embedding features.

[0058] CLIP-Text (CLIP Text Encoder): A text encoder for the CLIP model, based on the Transformer architecture, that converts text descriptions into 768-dimensional semantic embedding vectors.

[0059] The core innovation of this invention lies in the dynamic deformable inference kernel generation module. Its function is to "compile" the abstract task prototype into concrete model parameters that can be directly used to modify the model's behavior—the dynamic deformable inference kernel K_task.

[0060] This module creatively draws inspiration from Neural Radiation Fields (NeRF). NeRF was originally used to synthesize new 3D scenes from multi-view 2D images, its core being the mapping of spatial coordinates (x, y, z) to color and density. This invention transfers this principle: treating a high-dimensional task prototype vector as a "scene description," and "rendering" it into low-dimensional convolutional kernel parameters with a specific spatial structure.

[0061] Parametric Projection Network: The Dynamic Deformable Inference Kernel Generation Module incorporates a lightweight Multilayer Perceptron (MLP), namely the parametric projection network. This network contains three fully connected layers with a hidden layer dimension of 256. Its input is a 2816-dimensional cross-modal task prototype.

[0062] Spatial parameter mapping: The network outputs a one-dimensional vector, whose dimensions are designed to be reshaped into a 4×4×(3+1)=64-dimensional spatial grid parameter θ_grid. Here, 4×4 corresponds to the size of the convolutional kernel that will be used for subsequent operations on the feature map, 3 represents the weight at each position, and 1 represents the offset. This θ_grid can be understood as a "radiation field" of the task prototype in the parameter space.

[0063] Inference kernel parsing: The dynamic deformable inference kernel generation module 120 parses two parts from θ_grid: Spatial offset matrix (Δx, Δy): A 4×4×2 matrix used to finely adjust the position of standard grid sampling points during subsequent feature sampling.

[0064] Weight coefficient matrix W: A 4×4×C_in×C_out matrix (for simplification, it can be regarded as 4x4x3), used to weight the feature values ​​sampled after the position is adjusted.

[0065] Together, these two constitute the dynamically deformable inference kernel K_task. This process is a pure forward computation, which takes very little time (≤10ms).

[0066] The core operations of this module can be defined using the following key formulas and models: Formula: K_task=NeRF-Encode(P_v,P_t,θ_grid) Usage steps: Obtain the visual prototype P_v, the text prototype P_t, and the spatial grid parameter θ_grid.

[0067] The above three elements are input into the Neural Radiation Field Encoding (NeRF-Encode) module.

[0068] The NeRF-Encode module generates a task-specific deformable convolution kernel K_task.

[0069] Related model description: NeRF-Encode (Neural Radiance Field Encoding): Based on voxel rendering encoding using the Neural Radiance Field, it projects cross-modal prototypes into spatial grid parameters θ_grid to generate dynamic convolutional kernels. The NeRF-Encode used in this invention is a lightweight improved version (70% reduction in parameters), and its network structure includes 3 fully connected layers (256 hidden dimensions) + 1 output layer. The input is a concatenated vector of cross-modal prototypes (P_v, P_t) (dimension 2048+768=2816), and the output is a spatial grid parameter θ_grid (dimension 4×4×(3+1)=64, where 3 is the convolution kernel weight and 1 is the offset). The projection process is as follows: P_v and P_t are concatenated into an input vector V; a three-dimensional voxel feature field Φ(V) is obtained through NeRF network mapping (each voxel contains "feature value + density" information); Φ(V) is voxel rendered along the spatial dimension (2×2×2 voxels) to obtain the two-dimensional grid parameter θ_grid (containing the weight and offset of each convolution kernel position); the dynamic convolution kernel is generated by parsing a 4×4 weight matrix W (size 4×4×C_in×C_out) and an offset matrix Δ (size 4×4×2) from θ_grid, where C_in and C_out are the number of input / output channels (adapting to the feature dimension of the ROIAlign layer).

[0070] The self-supervised validation and labeling module automatically determines the validity of input samples and generates high-quality training labels for them without human intervention.

[0071] Example Validation (Entropy Filtering): When a new image is input, it is first predicted using the current model, resulting in a probability distribution P=[p_1,...,p_K] covering all known categories. Then, the Shannon entropy of this distribution is calculated as H=-∑(p_i×log(p_i)). The entropy measures the uncertainty of the prediction. If the model's prediction for the image is highly uncertain (i.e., the probability is uniformly distributed across multiple categories), the entropy H will be high. This invention sets a threshold, such as 1.8. When H>1.8, the system determines that the image likely belongs to a new category not seen by the model, and is a "valid" example that should be used for model evolution. This effectively avoids using difficult or repetitive samples of known categories for unnecessary updates.

[0072] Example Diversity Validation (Optional): To avoid the five valid input example images being highly similar (e.g., taken from the same location and angle), the self-supervised validation and annotation module can also perform diversity measurement. Local features are extracted from each image, and after dimensionality reduction using Principal Component Analysis (PCA), the cosine similarity S between feature vectors is calculated. If S > 0.9 for two images, they are considered redundant samples, and only one is retained. This ensures that the small number of samples used for optimization covers as many feature variations as possible.

[0073] Pseudo-label generation (Teacher-Student distillation): For valid examples that pass the validation, the self-supervised validation and annotation module 130 initiates the Teacher-Student framework to generate labels.

[0074] Teacher model: A larger, more accurate general vision model (such as DINOv2+FPN) pre-built at the edge. It has a slow inference speed and is not used for real-time detection, but it has strong feature extraction capabilities.

[0075] Student model: A lightweight model currently deployed for real-time inference that needs optimization.

[0076] The example image is input into the Teacher model to obtain its predicted bounding box B_tea and class c_tea. Predictions with a confidence score higher than 0.85 are selected as pseudo-labels, serving as the "gold standard." This process eliminates all manual annotation work.

[0077] The parameter isolation update module is responsible for performing the final model parameter update. To balance rapid adaptation and avoid forgetting, this module adopts a refined update strategy of "ternary parameter isolation".

[0078] Parameter partitioning: During system initialization, an offline sensitivity analysis (analyzing the impact of each parameter gradient on the loss of the new task) generates a mask matrix M with the same dimensionality as the Student model parameters. This matrix partitions the model parameters into three parts: Frozen layers (M[i]=0, approximately 80% of parameters): These are mainly the backbone network of the model (such as the first few layers of Swin-T), responsible for extracting general low-level features. These parameters remain unchanged during updates, thus preserving the model's generalization ability and are key to avoiding catastrophic forgetting.

[0079] Soft update layer (M[i]=0.3, approximately 17% of parameters): This mainly consists of the fusion layer for the neck and part of the head of the model. These parameters have a certain impact on both new and old tasks. An exponential moving average (EMA) is used for smooth updates, resulting in a slower update rate and a stable transition.

[0080] Hard update layer (M[i]=1, approximately 3% parameters): mainly the final classification head of the model and the parts directly related to the new task. In this invention, the dynamic deformable inference kernel K_task generated by module 120 is directly regarded as a hard update parameter and bound to the ROIAlign layer of the Student model.

[0081] Loss Calculation and Gradient Update: Substitute the pseudo-labels (B_tea, c_tea) and the predictions from the Student model (B_stu, P_stu) into the loss function L(θ; D_new) = α·L_cls + β·L_reg. Here, L_cls preferably uses cross-entropy loss, and L_reg preferably uses smoothed L1 loss. The weight coefficients are determined experimentally, for example, α = 1.0, β = 0.5. Then, calculate the gradient ∇L of the loss function with respect to all parameters.

[0082] Mask update: Finally, apply the update rule Δθ=M·η· Due to the mask M, only parameters with M[i] ≠ 0 (in both soft and hard update layers) are updated. Hard update layers use a higher learning rate (e.g., 0.01) and the RMSprop optimizer for fast adjustments, while soft update layers use a lower learning rate (e.g., 0.0001) for a smooth transition. The entire update process involves very few iterations (≤5 times), achieving update times of ≤200ms on NPUs such as the NVIDIA Jetson AGX Orin.

[0083] The core mathematical principle of parameter updating can be described by the following formula: formula: ; Usage steps: Determine the local gradient mask M, the learning rate η, and the gradient of the loss function based on the new example dataset D_new. (θ;D_new).

[0084] The local gradient mask M, the learning rate η, and the gradient of the loss function are used. (θ;D_new) Multiply the three together.

[0085] The result obtained is the parameter update amount Δθ.

[0086] Detailed explanation of each part of the formula: M (Local Gradient Mask): Set to 1 only for 3% of the hard update layer parameters, and set to 0 or 0.3 for the rest according to the strategy.

[0087] η (learning rate): 0.01 for hard update layers, and 0.0001 for other layers.

[0088] (Loss Function Gradient): The loss function L is a weighted sum of the classification loss and the regression loss: L(θ;D_new)=α⋅L_cls+β⋅L_reg. Where L_cls is the improved cross-entropy loss, and L_reg is the smoothed L1 loss. The weight coefficients α and β are determined through grid search, with optimal values ​​of α=1.0 and β=0.5.

[0089] The following is the flowchart of the method of the present invention.

[0090] Step A1: The system enters standby mode and loads the pre-trained Student and Teacher models, as well as the predefined parameter mask matrix M.

[0091] Step A2: Receive N (e.g., N=3-5) new category example images and optional text descriptions from the user.

[0092] Step A3: Perform cross-modal task prototype generation. Extract visual and textual features, generate stable visual prototypes through GMM clustering, and combine them into a cross-modal task prototype (P_v, P_t).

[0093] Step A4: Generate a dynamic deformable inference kernel. The task prototype is forward computed through a NeRF-based parametric projection network to generate a task-specific dynamic deformable inference kernel K_task.

[0094] Step A5: Perform self-supervised example validation and pseudo-label generation. The input examples are filtered for entropy and diversity to remove invalid and redundant samples. Then, the Teacher-Student framework is used to generate high-precision pseudo-labels (B_tea, c_tea) for the valid samples.

[0095] Step A6: Perform incremental updates based on a parameter isolation strategy. Bind K_task to the ROIAlign layer of the Student model. Calculate the loss based on the pseudo-labels and apply the mask matrix M and the differential learning rate, performing gradient updates only on the hard and soft update layers with a small number of iterations.

[0096] Step A7: Update complete. The Student model now has high-precision detection capabilities for the new category. The system switches to real-time inference mode and uses the optimized model for online detection.

[0097] Application Scenario Example 1: Real-time Adaptation for Fresh Produce Sorting The intelligent camera system of this invention was deployed on an automated fresh produce sorting line. The initial model was able to identify apples, bananas, and oranges.

[0098] A new task arises: One day, a new batch of "black plums" arrives on the assembly line. The system has never seen this type of fruit before, and the initial detection accuracy is only about 50%.

[0099] Example data collection: The operator removes three "black plums" from the production line at three different angles, takes three example images in front of the camera, and inputs them into the system.

[0100] Second-level evolution: The system automatically executes the complete process of S203-S206 mentioned above.

[0101] Extract the visual features of the "black plum" and generate a prototype.

[0102] Generate a dynamic inference kernel K_task specifically for recognizing the texture, color, and shape of the "black plum".

[0103] The Teacher model generates accurate bounding box pseudo-labels for these three images.

[0104] The parameter isolation update module completed model optimization in less than one minute (including multiple iterations and hardware processing time).

[0105] Results Verification: The optimized model was immediately put into use, and the accuracy rate for detecting "black plums" on the production line surged from 50% to 92%. The entire process required no line downtime and no data experts, achieving seamless integration with the production workflow. At night, the system automatically integrates all new examples from the day for a more comprehensive incremental learning, further improving sorting efficiency by 30% the following day.

[0106] Application Scenario Example 2: Substation Equipment Defect Detection In unattended substations, the system of this invention is deployed on inspection robots or fixed cameras to monitor the status of critical equipment (such as insulators, transformers, switchgear, etc.) in real time. The initial model library contains common equipment types and known defect patterns.

[0107] A new type of defect was discovered: With equipment aging or environmental changes, a novel type of insulator series crack emerged, a crack pattern not previously observed in the initial training data. During inspection, the inspection robot equipped with this invention consistently exhibited low detection confidence in this area, with an initial false negative rate as high as 28%.

[0108] Rapid on-site adaptation: After receiving the alert, maintenance personnel arrived at the site and confirmed the existence of the new defect. Using handheld devices or a remote-controlled inspection robot, they captured five clear images of the new crack from different angles and lighting conditions. Due to the isolation of the substation's internal network, the maintenance personnel imported these five sample images into the invention's system deployed locally on the robot via USB drive.

[0109] Autonomous evolution and inference kernel generation: After receiving the image, the system immediately executes the optimization process locally. The system identified this as a "high entropy" event and determined it to be a valid new category.

[0110] Within 200ms, the system generates a dedicated dynamic inference kernel K_task for the fine-grained category of "novel insulator series crack".

[0111] The Teacher-Student framework automatically generated pixel-level pseudo-labels for these 5 images.

[0112] The parameter isolation update module completes the fine-tuning of the local model within seconds.

[0113] Performance improvements and continuous optimization: Immediate results: After the model was updated, the inspection robot immediately re-inspected the same area, and the missed detection rate of the new type of crack was reduced from 28% to 5%, with a false alarm rate of less than 1%, achieving a high reliability standard for industrial applications.

[0114] Continuous learning: During subsequent routine inspections, the system will automatically aggregate newly detected, high-confidence images of this type of defect as new samples. For example, during low-load periods at night, the system will automatically integrate all new examples collected that day for an incremental learning round, further consolidating and optimizing model performance, achieving continuous evolution without human intervention and without the need for secondary intervention from maintenance personnel.

[0115] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0116] Although this document uses a variety of terms, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of the invention; interpreting them as any additional limitation would contradict the spirit of the invention.

Claims

1. An edge visual model optimization method based on a dynamic inference core, characterized in that, The method comprises the following steps: S1: cross-modal task prototype generation: For at least one example image containing a new target and its corresponding optional text description, N-dimensional visual features are extracted using a pre-trained visual model, and M-dimensional text features are extracted using a pre-trained text encoder; The N-dimensional visual features are clustered using a Gaussian mixture model to generate a visual prototype P_v representing the core visual attributes of the new target; The visual prototype P_v is combined with the text prototype P_t generated from the M-dimensional text features to form a cross-modal task prototype; S2: Dynamic deformable inference kernel generation: The cross-modal task prototype is input into a parameter projection network based on the principle of neural radiation field; The parameter projection network maps the task prototype into a task-specific spatial grid parameter θ_grid, and resolves a dynamic deformable inference kernel K_task from the θ_grid; The K_task includes a spatial offset matrix for adjusting feature sampling positions and a weight coefficient matrix W for adjusting feature values; S3: Self-supervised example verification and pseudo-label generation: The example image is input into the visual model currently deployed on the edge device to calculate the entropy value H of its predicted class probability distribution. When H is greater than a preset entropy threshold, the example image is determined to be an effective new class example image; A pseudo-label is generated for the effective new class example image using a Teacher-Student distillation framework, where the Teacher model is a high-performance general model pre-installed on the edge, and the Student model is a lightweight edge visual model to be optimized; The Teacher model outputs high-confidence bounding box coordinates B_tea and class c_tea as pseudo-labels; S4: Incremental update based on parameter isolation strategy: According to a pre-defined parameter mask matrix M, the parameters of the Student model are divided into frozen layers, soft update layers and hard update layers; The dynamic deformable inference kernel K_task generated in step S2 is bound to the region of interest alignment layer of the Student model as a hard update parameter; Based on the pseudo-labels B_tea and c_tea generated in step S3 and the prediction results B_stu and P_stu of the example image by the Student model, the overall loss function L(θ;D_new) is calculated according to the following formula: L(θ;D_new)=α·L_cls(P_stu,c_tea)+β·L_reg(B_stu,B_tea); Where L_cls is the classification loss, L_reg is the regression loss, and α and β are preset weight coefficients; The parameters θ of the Student model are updated according to the following update rule: ； where Δθ is the parameter update, and η is the learning rate, is the gradient of the loss function; only the parameters of the soft and hard update layers are updated.

2. The edge visual model optimization method based on dynamic inference core according to claim 1, characterized in that, In step S2, the parameter projection network includes at least three fully connected layers, and the parameter projection network maps the input cross-modal task prototype into a spatial grid parameter θ_grid with dimensions 4×4×(3+1), where 3 represents the convolution kernel weight dimension and 1 represents the offset dimension; The size of the resolved spatial offset matrix is 4×4×2.

3. The method of claim 1, wherein the method further comprises: In the step S3, the entropy value H is calculated by the following formula: H=-∑(p_i×log(p_i)); where p_i is the prediction probability of the model for the i-th class.

4. The edge visual model optimization method based on dynamic inference core according to claim 1 or 3, characterized in that, In the step S3, a feature diversity measurement step is further included: performing principal component analysis dimension reduction on the local feature vectors of the plurality of valid new class examples, and calculating the cosine similarity S between the reduced features, when S is less than a preset similarity threshold, the example is retained.

5. The method of claim 1, wherein the method further comprises: In the step S4, the parameter mask matrix M defines no more than 3% of the model parameters as a hard update layer, and 10%-20% of the parameters as a soft update layer; the soft update layer is updated by exponential moving average, and the hard update layer is updated by gradient using the RMSprop optimizer.

6. An edge visual model optimization system based on dynamic inference core, characterized in that, The method comprises the steps of: a cross-modal task prototype generation module configured to extract visual features and text features from at least one example image containing a new target and its corresponding optional text description, and generate a cross-modal task prototype (P_v, P_t) through Gaussian mixture model clustering and other processing; a dynamic deformable inference kernel generation module configured with a parameter projection network constructed based on the principle of neural radiation field, for receiving the cross-modal task prototype and mapping it to generate a dynamic deformable inference kernel K_task containing a spatial offset matrix (Δx, Δy) and a weight coefficient matrix W; a self-supervised verification and labeling module for calculating the prediction entropy value of the input example to verify whether it is a new class, and using the built-in Teacher-Student distillation framework to automatically generate pseudo-labels (B_tea, c_tea) for supervised learning for the verified examples; a parameter isolation update module for dividing the Student model parameters to be optimized into three categories of frozen, soft update and hard update according to a predefined parameter mask matrix M; and binding the dynamic deformable inference kernel K_task as a hard update parameter to the specified layer of the model, calculating the loss according to the pseudo-labels, and only performing gradient update on the non-frozen parameters to complete the optimization of the model.

7. The edge visual model optimization system based on dynamic inference core according to claim 6, wherein, The parameter projection network in the dynamic deformable inference kernel generation module takes the cross-modal task prototype as input and outputs a task-specific spatial grid parameter θ_grid, and the module further parses the dynamic deformable inference kernel K_task from θ_grid.

8. The edge visual model optimization system based on dynamic inference core according to claim 6, wherein, In the parameter isolation update module, the learning rate of the hard update parameter is higher than that of the soft update parameter, and the hard update parameter is directly updated by gradient descent, and the soft update parameter is updated by exponential moving average.

Images