Image processing model optimization method and related device

Abstract

The invention discloses an image processing model optimization method and a related device, and the method comprises the steps: obtaining an optimization data source of a to-be-optimized image processing model, carrying out the multi-modal feature fusion and root cause diagnosis through a pre-training defect analysis model, generating a model defect positioning result and a parameter optimization instruction, and carrying out the optimization of the to-be-optimized image processing model. The parameter optimization instruction is compiled into a parameter updating operation code, and the to-be-optimized image processing model is updated, so that real-time updating, automatic optimization, safe evolution and data privacy protection of the model can be realized in a resource-limited edge environment; and the applicability and reliability of the edge video image analysis model in scenes such as intelligent security and protection, industrial visual inspection and intelligent oil stations are improved.

Description

Technical Field

[0001] This invention pertains to model optimization technology, and particularly relates to an image processing model optimization method and related apparatus. Background Technology

[0002] With the widespread deployment of video surveillance and intelligent sensing devices in urban management, industrial manufacturing, and retail, massive amounts of video data are increasingly being analyzed and reasoned at the edge, thereby reducing cloud pressure and improving real-time performance. However, due to the limitations of computing power, storage, and network bandwidth on edge devices, the lightweight video analytics models running on them face numerous challenges in practical applications.

[0003] Existing edge video analytics systems generally adopt the traditional "cloud training + edge deployment" model, which has significant shortcomings in model maintenance and updates. On the one hand, data needs to be collected manually and sent back to the cloud for processing, and model updates rely on full OTA push notifications, resulting in response cycles typically exceeding 24 hours, which cannot meet the real-time requirements of sudden events. On the other hand, edge-side models struggle to quickly adapt to long-tail scene changes, such as extreme weather, sudden changes in lighting, and occlusion, often resulting in 30%–40% false alarms and a lack of effective online adaptive capabilities. Furthermore, existing update mechanisms mostly rely on directly replacing or incrementally training model parameters, which can easily lead to catastrophic amnesia and system instability, posing significant security risks. Summary of the Invention

[0004] Based on this, the present invention aims to propose an image processing model optimization method and related apparatus, which constructs a data source reflecting the faults of the image processing model, uses a defect analysis model to locate the cause of the fault and generate optimization instructions, and compiles the parameter optimization model into operation codes that the image processing model can understand, thereby realizing timely response to model optimization and improving the performance and stability of the model.

[0005] In a first aspect, the present invention provides an image processing model optimization method, comprising:

[0006] The erroneous inference data of the image processing model to be optimized is denoted as the optimization data source, which includes the structural metadata of the image processing model to be optimized.

[0007] Using a pre-trained defect analysis model, defects are analyzed in the image processing model to be optimized based on the optimized data source, resulting in defect localization results and model parameter optimization instructions.

[0008] The model parameter optimization instructions are compiled into parameter update opcodes, which are then pushed to the image processing model to be optimized for model optimization.

[0009] Furthermore, the erroneous inference data of the image processing model to be optimized, denoted as the optimization data source, includes:

[0010] Obtain the image samples that are incorrectly inferred by the image processing model to be optimized and their corresponding image features;

[0011] Record the current structural metadata of the image processing model to be optimized;

[0012] Combine image samples, image features, and structural metadata into an optimized data source.

[0013] Furthermore, the pre-trained defect analysis model is used to perform defect analysis on the image processing model to be optimized based on the optimized data source, and the model defect localization results are obtained, including:

[0014] The image samples, image features, and structural metadata in the optimized data source are encoded to obtain corresponding feature vectors;

[0015] Cross-modal feature fusion is performed on each feature vector using an attention mechanism to obtain fused features;

[0016] The fusion features are matched with a predefined defect database to determine the model defects corresponding to the fusion features;

[0017] The multi-task prediction head outputs model defect localization results and model parameter optimization instructions in parallel.

[0018] Furthermore, compiling the model parameter optimization instructions into parameter update opcodes includes:

[0019] The parametric compilation model is used to compile model parameter optimization instructions into initial values ​​of parameter update opcodes based on regular expression matching rules.

[0020] Calculate the parameter increment based on the initial value of the parameter update opcode;

[0021] The parameter increment is validated based on preset constraints to obtain the validation parameter increment.

[0022] The update accuracy is tested based on the increment of the verification parameters and the initial value of the parameter update opcode. If the model accuracy does not decrease after the image processing model to be optimized is updated based on the initial value of the parameter update opcode, the initial value of the parameter update opcode is determined to be the final parameter update opcode.

[0023] Furthermore, the preset constraints include:

[0024] The norm of the parameter increment does not exceed the preset norm threshold, and the change in the output of the image model to be optimized under the effect of the parameter increment does not exceed the preset output change range.

[0025] Furthermore, when multiple image processing models to be optimized are in the same system, the above method also includes:

[0026] For each image processing model to be optimized, the corresponding parameter update opcode is used to update the model parameters to obtain the optimized image processing model.

[0027] Obtain scene fingerprints for each optimized image processing model;

[0028] The update contribution of each optimized image processing model is calculated based on the fingerprint of each scenario;

[0029] The global parameter update amount is calculated based on the update contribution of each optimized image processing model, and the global parameter update amount is pushed to each optimized image processing model for global update.

[0030] Furthermore, the calculation of the global parameter update amount based on the update contribution of each optimized image processing model includes:

[0031] The increments of the verification parameters of each optimized image processing model are weighted and calculated based on the update contribution of each optimized image processing model to obtain the weighted parameter increments.

[0032] Gaussian noise is introduced into the weighted parameter increment to obtain the global parameter update.

[0033] In a second aspect, the present invention provides an image processing model optimization apparatus, comprising:

[0034] The data acquisition module is used to acquire the erroneous inference data of the image processing model to be optimized, which is referred to as the optimization data source. The optimization data source includes the structural metadata of the image processing model to be optimized.

[0035] The parameter optimization module is used to perform defect analysis on the image processing model to be optimized based on the optimization data source using a pre-trained defect analysis model, and obtain the model defect location results and model parameter optimization instructions.

[0036] The instruction compilation module is used to compile model parameter optimization instructions into parameter update opcodes, which are then pushed to the image processing model to be optimized for model optimization.

[0037] Thirdly, the present invention provides an electronic device including a memory storing computer-executable instructions and a processor, wherein when the computer-executable instructions are executed by the processor, the device performs the steps of the image processing model optimization method provided in the first aspect.

[0038] Fourthly, the present invention provides a readable storage medium storing a computer-executable program that, when executed, can implement the various steps of the image processing model optimization method provided in the first aspect.

[0039] The present invention has the following beneficial effects:

[0040] This invention proposes an image processing model optimization method and related apparatus. When erroneous inference occurs, it records image samples, feature data, and model structure metadata. It then uses a pre-trained defect analysis model to locate defects and generate parameter optimization instructions, compiling these instructions into opcodes for accurate model understanding and execution. This invention can accurately identify the source of model errors and provide customized parameter optimization schemes without human intervention. Further embodiments employ parameter compilation, parameter incremental constraint verification, and update accuracy testing to effectively avoid over-updating or catastrophic forgetting, ensuring safe and controllable incremental optimization of the model on structurally constrained edge devices. In a more optimized embodiment, in multi-model collaborative scenarios, this invention further introduces a scene fingerprint and contribution evaluation mechanism, enabling the system to integrate optimization experience across multiple edge devices. It generates global parameter updates through weighted increments and noise perturbations, enhancing cross-device generalization and privacy-friendly knowledge sharing. Therefore, this invention not only significantly reduces model optimization latency and improves model adaptability and accuracy in complex scenarios but also maintains overall system stability and security in resource-constrained scenarios, demonstrating significant practical value and potential for widespread adoption. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 The flowchart illustrates the implementation of the image processing model optimization method provided in this embodiment of the invention.

[0043] Figure 2 This is a schematic diagram of the image processing model optimization device provided in an embodiment of the present invention;

[0044] Figure 3 This is an electronic device architecture diagram provided for an embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] See Figure 1An embodiment of the present invention provides an image processing model optimization method, comprising the following steps:

[0047] Step S110. Obtain the erroneous inference data of the image processing model to be optimized, denoted as the optimization data source. The optimization data source includes the structural metadata of the image processing model to be optimized.

[0048] In this step, we first acquire the erroneous inference data of the image processing model to be optimized during the actual inference process. This data serves as the optimization data source, characterizing the model's behavioral patterns at three levels: input, internal computation process, and structural configuration. This provides a quantifiable basis for subsequent defect diagnosis. The erroneous inference data can be automatically collected by the online model during real business processing, or it can be recorded after batch inference of the model using an offline validation set.

[0049] Specifically, optimizing the data source first includes image samples that are incorrectly inferred. The image data that is incorrectly identified by the model is preprocessed and converted into a pixel tensor with dimensions H×W×3. Further pixel normalization is performed, mapping the RGB components of each pixel from 0–255 to a floating-point range of [0,1] to eliminate the influence of the input distribution scale on the defect analysis model and ensure that the input tensor maintains a consistent dynamic range.

[0050] In a further embodiment, the system can also simultaneously collect different versions of the same misjudged image, such as the original resolution frame, the frame scaled to the model input size, and the frames before and after enhancement, to compare the model's sensitivity to images of different scales and qualities during the defect analysis phase. This multi-version input supplementation can be used to determine whether the model has significant defects in multi-scale features, texture variations, or input noise.

[0051] In a more optimized embodiment, contextual description metadata, such as lighting conditions (low light, backlight), scene category, shooting device parameters, motion blur level, etc., is further added during the input data preparation stage to form weak label information of the input data. These descriptions can be automatically generated by external image quality evaluation algorithms or video metadata to help the defect analysis model establish the correlation between input conditions and model error performance.

[0052] Secondly, obtain the image features mapped in the image processing model to be optimized from the image samples that were incorrectly inferred. For example, collect the activation feature maps and attention heatmaps of the intermediate layers. The intermediate feature maps reflect the model's capabilities in terms of local receptive field, channel space, and semantic expression. Their abnormal shapes are often the root cause of output errors.

[0053] In a further embodiment, the acquisition of intermediate feature maps can be extended to more levels, including shallow texture features, deep semantic features, fusion layer outputs, and the logits distribution of the detection head or classification head. By constructing multi-level feature combination signals, the defect analysis model can perform a more refined evaluation of the feature gradient flow, attention allocation, and semantic fusion mechanism within the model.

[0054] In a more optimized embodiment, statistical features of the feature maps, such as channel mean, channel variance, spatial entropy, and response sparsity, are introduced to provide compressed structural feature metrics for the defect analysis model. This statistical supplement significantly reduces the storage and communication overhead of intermediate feature maps in the model, making it particularly suitable for edge device deployments or low-bandwidth scenarios.

[0055] The structural metadata of the model, including descriptions of its hierarchical structure, parameter distribution, operator types, activation function types, kernel size, and feature map dimensions, serves to provide an interpretable structural cross-section for subsequent defect analysis. This allows the analysis to extend beyond input and output, enabling inferences about the internal causes of errors. Furthermore, annotations can be added to each layer regarding the number of parameters, weight distribution range, whether learnable biases are included, and whether learnable normalized parameters are present. This helps the defect analysis model determine whether the structure suffers from insufficient capacity, gradient anomalies, or parameter imbalances.

[0056] In a further embodiment, the network structure metadata may also include model deployment environment information (such as hardware type, memory size, operator acceleration library version, inference framework version, etc.) to determine whether the model suffers from structural bottlenecks due to hardware compatibility differences. For example, some convolutional structures may be inefficient on DSPs or specific NPUs, leading to a decrease in real-time performance and thus interfering with the model's correct inference.

[0057] In a more optimized embodiment, in order to further enhance the expressive power of structural metadata, a graph embedding representation of structural metadata can be constructed. That is, a graph encoder is used to encode the model topology into a fixed-length vector, which, together with the intermediate feature map and the input data, serves as a unified input to the defect analysis model, thereby achieving deeper defect detection at the level of linkage between structure, feature, and decision.

[0058] Step S120. Utilize the pre-trained defect analysis model to perform defect analysis on the image processing model to be optimized based on the optimized data source, and obtain the model defect localization results and model parameter optimization instructions.

[0059] In this step, the pre-trained defect analysis model automatically extracts key features from the optimization data source and performs defect analysis on the image processing model to be optimized. The defect analysis model is typically a deep network structure with cross-modal representation capabilities, capable of processing images, feature maps, and structural metadata simultaneously, thereby analyzing the source of model errors from multiple dimensions. Its architecture can be graph neural networks, multimodal Transformers, structural analysis networks, etc. Its input includes error sample data and model structural metadata, and its output includes model defect localization results and parameter optimization instructions.

[0060] Specifically, the defect analysis model first performs special encoding on the three types of data (original image, intermediate feature map, and structural metadata) in the optimized data source to extract their deep semantic features.

[0061] The original image is first passed through a convolutional neural network or a visual Transformer encoder to generate image feature vectors of fixed length or fixed spatial scale. The encoder extracts information such as target appearance, background texture, and image quality related to erroneous results through hierarchical convolution and attention mechanisms. Image features are then extracted using a lightweight convolutional network or a channel attention network. The vectors encoded from the feature maps characterize the model's response patterns at different semantic levels (e.g., texture layer, semantic layer), and are used to check for anomalies in the model's internal inference, such as features that are too weak, too strong, or have abnormal channel distribution. Network topology information in the structural metadata can be encoded using a graph neural network or a structured Transformer, and the encoding results are used to infer the structural constraints of the model. All three types of encoding output comparable feature vectors, providing a unified input space for subsequent cross-modal fusion.

[0062] The defect analysis model uses a multi-head attention mechanism to fuse three types of features across modalities. The attention allocation process can dynamically adjust the weights of different modalities. For example, when the lighting is normal but the feature distribution is abnormal, the weight of the intermediate feature map will be higher; when there is a significant bottleneck in the model structure, the attention weight of the structural metadata will be higher.

[0063] In a further embodiment, a contrastive learning loss can be introduced during the cross-modal fusion process. By setting positive and negative sample pairs, the defect analysis model can learn more robust matching relationships under different modal combinations, thereby improving the accuracy of defect identification. For example, normal inference samples can be used as a control, enabling the model to learn to recognize the differences between "normal feature flow" and "abnormal feature flow".

[0064] In a more optimized embodiment, cross-modal fusion introduces a hierarchical fusion strategy, which involves performing fusion at different levels of abstraction, including a low-level visual layer, a mid-level semantic layer, and a high-level structural layer. This approach allows defect analysis to occur not only at the global level but also at specific levels, making the output defect localization results more physically meaningful, such as explicitly indicating that "insufficient extraction of shallow features leads to loss of edge information."

[0065] After completing cross-modal fusion, the system performs similarity matching between the fused features and a predefined defect database. The defect database defines common model error patterns, such as insufficient receptive field leading to delayed target detection, insufficient channel representation leading to missed detection of small objects, attention shift leading to background interference, and multi-scale fusion failure leading to feature expression fragmentation. Each type of defect in the database corresponds to a feature template. By matching the fused features with these templates, the system can identify which type of internal fault the current misjudgment belongs to.

[0066] The defect analysis model uses a multi-task prediction head to output the model defect localization results and model parameter optimization instructions in parallel. The model defect localization results include information such as defect category, structural location of defect occurrence, and feature level of impact. The model parameter optimization instructions are expressions predicted directly from the fused features to adjust the parameters, such as increasing or decreasing the channels of a convolution kernel, or resetting or enhancing the weights of an attention head.

[0067] In a further embodiment, the multi-task prediction head adopts a shared feature encoding but independent output layer structure, so that the defect localization and parameter optimization instructions can share context information while maintaining the independence of the output space, thus avoiding gradient conflicts between the two tasks.

[0068] In a more optimized embodiment, parameter optimization instructions are described by interpretable expressions, such as "enhancement of gradient direction of the Kth channel in the Lth layer" and "recalibration of the detection head anchor encoding range", which enables the parameter compilation stage to directly generate update opcodes based on the expressions, thereby improving the overall automation level of the system.

[0069] In a further embodiment, the defect database is dynamically updated. That is, during long-term operation, the system will re-cluster the fusion features generated by real misjudgments to form new defect categories, thereby achieving "database self-evolution" and making the knowledge base of the defect analysis model closer to the actual business distribution.

[0070] In a more optimized embodiment, the defect database matching process outputs not only the defect type, but also the defect location (e.g., which layer and which convolutional block it occurs in), so that the parameter update instruction can achieve "local optimization" rather than global perturbation, significantly reducing the risk of model instability.

[0071] In a further embodiment, the defect analysis model also infers causal relationships based on the distribution characteristics of erroneous samples. For example, by clustering misjudged samples in the feature space, the analysis model can identify abnormal associations between certain input patterns (such as local high-frequency textures, low-light images, and partially occluded objects) and specific network layers. At this point, the model can infer that a certain convolutional module or attention head performs poorly for a specific type of input and locate that module as the source of the defect.

[0072] In a more optimized embodiment, the defect analysis model incorporates a parameter sensitivity analysis mechanism during diagnosis. This involves subtly perturbing the model parameters and observing the resulting output changes on erroneous samples to quantify the contribution of different parameters to the erroneous behavior. This mechanism can generate sensitivity scores based on gradients, Jacobian matrices, or black-box perturbations, leading to more accurate defect localization results and providing a basis for subsequent parameter optimization instructions.

[0073] Step S130. Compile the model parameter optimization instructions into parameter update opcodes, and push the parameter update opcodes to the image processing model to be optimized for model optimization.

[0074] In this step, the model parameter optimization instructions generated in step S120 are converted into executable parameter update opcodes through parameter compilation model, and the image processing model to be optimized is safely and controllably optimized and updated on the edge device.

[0075] Specifically, the parametric compilation model uses natural language or high-level parameter optimization instructions (such as "enhance the weight of the 5th channel of Block3.conv" or "adjust the kernel size of conv2d_2 to 5") to generate corresponding opcode initial values ​​through regular expression matching and rule parsing. The information of the opcode initial value includes the target layer identifier, operation type, operation parameters, etc., where the target layer identifier indicates the network layer to be modified, the operation type indicates types such as weight increase (WEIGHT_INC), kernel adjustment (KERNEL_RESIZE), and channel pruning (CHANNEL_PRUNE), and the operation parameters indicate the corresponding updated values, such as weight increment, kernel size adjustment value, channel ratio, etc.

[0076] By compiling, high-level optimization strategies are transformed into low-level opcodes that can be directly executed by the system, ensuring that subsequent operations can accurately correspond to the structure and parameters of the model to be optimized.

[0077] In a further embodiment, opcode compilation may introduce semantic verification, such as checking whether the target layer exists or whether the operation parameters exceed physical constraints or hardware support range, thereby avoiding illegal operations that could lead to model crashes or hardware execution errors.

[0078] In a more optimized embodiment, the system introduces operation dependency analysis during the compilation phase, that is, analyzes the possible dependencies and conflicts between opcodes. For example, adjusting the convolution kernel and channel pruning at the same time may cause abnormal gradient distribution. The system will automatically reorder or execute the opcodes in batches to ensure update safety.

[0079] After the initial opcode value is generated, the system calculates the corresponding parameter increment Δθ according to the operation type. For example, for WEIGHT_INC, the system adds the convolution weights or fully connected layer weights according to the increment value specified by the opcode; or for KERNEL_RESIZE, the system re-initializes the increment parameters after adjusting the convolution kernel size or generates a new weight tensor using an interpolation algorithm; or for CHANNEL_PRUNE, the system calculates the impact of zeroing or redistributing the weights of the pruned channels on the remaining channels.

[0080] The calculation of parameter increment Δθ not only considers the operation value, but also combines the current weight distribution and gradient direction to ensure that the increment direction conforms to the overall model optimization objective.

[0081] In a further embodiment, gradient normalization or scaling mechanisms can be incorporated into the incremental calculation to adaptively adjust the incremental magnitude of different levels and channels, making the weight adjustment of key layers more refined and avoiding excessive perturbation of low-response channels that could affect overall performance.

[0082] In a more optimized embodiment, incremental pre-evaluation can be performed by combining simulation inference feedback: small batches of synthetic data are generated locally, and the update effect is predicted through rapid inference testing, thereby correcting the direction and magnitude of parameter increments and realizing a closed loop of "pre-tuning - fine-tuning - safe update".

[0083] In a further embodiment, to ensure model stability, the parameter increment is checked based on preset constraints to obtain the check parameter increment. If the increment does not meet the constraints, Δθ can be trimmed proportionally or the initial value of the opcode can be adjusted to meet the security update requirements.

[0084] Specifically, the preset constraints are set as a dual constraint mechanism, including: the norm of the parameter increment does not exceed the preset norm threshold, and the output change of the image model to be optimized under the action of the parameter increment does not exceed the preset output change range.

[0085] The F-norm of the parameter increment Δθ must not exceed a preset threshold (e.g., 0.1) to prevent excessively large single updates from causing drastic changes in the model output. After the update, the change in the model output with respect to the same input must be within a controllable range, for example, through Lipschitz continuity testing or actual simulation tests to ensure this. Less than the preset upper limit.

[0086] In a further embodiment, the constraint mechanism can dynamically adjust the threshold by incorporating historical update records. For example, for certain layers with good stability, the amplitude constraint can be relaxed; for critical output layers or gradient-sensitive layers, the upper limit of output change can be tightened to improve update security.

[0087] In a more optimized embodiment, a multi-round simulation test and incremental rolling strategy are introduced: Δθ is broken down into multiple small increments, and a fast inference verification is performed after each update to ensure that the cumulative effect of continuous small updates reaches the expected level, while avoiding the risk of accuracy degradation caused by a large one-time disturbance.

[0088] Finally, the update accuracy is tested based on the verification parameter increment and the initial value of the parameter update opcode. If the model accuracy does not decrease after the image processing model to be optimized is updated based on the initial value of the parameter update opcode, the initial value of the parameter update opcode is determined to be the final parameter update opcode.

[0089] Specifically, if the accuracy after the update decreases by less than a preset threshold (e.g., ≤5%) compared to the original model, then Δθ is confirmed to be usable for formal update; if the accuracy decreases by more than the threshold, then the update is rolled back to restore the original model weights.

[0090] In a further embodiment, the simulation test can use a multi-scenario comprehensive test set to cover different lighting, target occlusion and motion blur conditions to ensure the robustness of the update under various business scenarios.

[0091] In a more optimized embodiment, a progressive update strategy can be introduced into the simulation test, first testing on local scenarios or a subset of channels, and then gradually extending to a global update. This strategy can significantly reduce update risks, and is particularly suitable for scenarios where edge devices have limited computing resources and high real-time requirements.

[0092] Once the parameter increment passes the constraints and test verification, the system pushes the final parameter update opcode to the image processing model mounted on the edge device (such as a camera, drone, vehicle-mounted device, etc.) to complete the actual update.

[0093] In a further embodiment, batch model updates can be supported, with similar opcodes executed in parallel on multiple devices, while generating local update logs for subsequent federated aggregation and global optimization.

[0094] In a more optimized embodiment, the update process is combined with real-time performance monitoring of edge devices. If resource shortages or abnormal latency are detected, the update can be delayed or performed in stages, further ensuring the reliability of the edge model in low computing power or low bandwidth environments.

[0095] In some alternative embodiments, when there are multiple image processing models to be optimized in the system, the local updates of a single edge device are difficult to fully utilize the experience of multiple devices. A federated evolution mechanism can be introduced to achieve cross-device knowledge sharing and global optimization updates. This mechanism specifically includes scene fingerprint generation, update contribution calculation, global parameter aggregation, and differential privacy protection.

[0096] Specifically, each optimized edge model first generates a privacy-preserving scene identifier using a scene fingerprint algorithm. Multi-scale information is extracted from key features at each layer (such as shallow convolutional features, Grad-CAM activation heatmaps, and intermediate convolutional block features). These features are flattened and concatenated to form a unified vector representation. This vector comprehensively reflects multi-dimensional feature information such as lighting conditions, target type, and background complexity of the scene in which the edge device is located, providing a foundation for scene feature matching. The concatenated vector is then symbolically binarized: elements greater than 0 are marked as 1, and elements less than or equal to 0 are marked as 0, resulting in a hash binary string. This step enhances the robustness of the features and reduces storage and computation costs. The generated hash binary string is processed using the SHA-256 hash algorithm, and the first 16 bits are used as the scene fingerprint. This fingerprint can characterize the main features of the scene in which the device is located and can perform cross-device matching without leaking the original data, ensuring privacy and security.

[0097] In a further embodiment, scene fingerprint generation can incorporate multiple hashing and Locality Sensitive Hash (LSH) algorithms to improve matching accuracy while enhancing robustness to abnormal scenarios.

[0098] In a more optimized embodiment, time-series feature encoding, such as dynamic feature changes in continuous video frames, can be incorporated into the fingerprint generation process to form a "spatiotemporal scene fingerprint," thereby providing more refined guidance for the scene adaptability of model updates.

[0099] The federated side calculates the update contribution of each model based on the similarity between the scene fingerprint of each model and the global scene fingerprint. For example, the update contribution of each model is calculated as follows:

[0100]

[0101] Where represents the update contribution of the i-th image processing model to be optimized. For the scene fingerprint of the i-th image processing model to be optimized, This represents the global scene fingerprint, where β is the similarity decay coefficient, used to control the rate at which the contribution diminishes with scene differences. Through... Quantify the weight of each model's optimization experience in the global update.

[0102] In a further embodiment, the β value can be dynamically adjusted based on the device's historical performance records. For highly reliable devices, the contribution weight can be appropriately increased to improve the accuracy of global updates.

[0103] In a more optimized embodiment, the contribution calculation can be extended to a multi-dimensional weighted mode, such as taking into account device computing power, data quality and scenario representativeness, so that the global parameter update is more in line with actual optimization needs.

[0104] Local verification parameter increments for each device The global weighted parameter increment is obtained by weighting the values ​​based on contribution, and Gaussian noise is introduced onto it to generate the global parameter update. And introduce Gaussian noise accomplish - Differential privacy, for example, at σ=1.2, conditions, This ensures that data from a single model cannot be inferred during knowledge sharing and global aggregation, thus preventing privacy leaks.

[0105] Final global parameter update It is distributed to various optimized image processing models, achieving global model optimization across devices. Each edge device then applies it locally. After the update is completed, the model gains local optimization capabilities based on local experience and global adaptability after cross-model knowledge aggregation.

[0106] In the above embodiments, the image processing model to be optimized is typically deployed on edge devices, such as cameras used for video acquisition and analysis, industrial inspection terminals, or video processing nodes in smart gas stations. The edge device is responsible for real-time acquisition of video data and local inference. When the model makes a misjudgment, it can instantly generate an optimized data source and perform local parameter updates, thereby ensuring high real-time performance and low latency of the model in resource-constrained environments.

[0107] The pre-trained defect analysis model, along with federated learning and global parameter aggregation mechanisms, can be deployed in the cloud or on a server. The pre-trained defect analysis model receives optimization data from edge devices, performs multimodal feature fusion and root cause analysis, and generates model defect localization results and optimization instructions, providing targeted model optimization suggestions for edge devices. The federated learning mechanism utilizes the optimization experience of multiple edge devices for scene fingerprint matching, contribution calculation, and global parameter aggregation, thereby achieving cross-device knowledge sharing and global optimization while ensuring data privacy. Through this cloud-edge collaborative deployment approach, the system can meet the real-time inference needs of edge devices while leveraging the powerful computing resources of the cloud for complex analysis and global optimization, achieving efficient, secure, and controllable model evolution.

[0108] The disclosed method can be implemented using various types of devices. Therefore, the present invention also discloses an apparatus corresponding to the above method, and specific embodiments are given below for detailed description.

[0109] like Figure 2 As shown, one embodiment of the present invention provides an image processing model optimization apparatus, comprising:

[0110] Data acquisition module 202 is used to acquire the erroneous inference data of the image processing model to be optimized, denoted as the optimization data source. The optimization data source includes the structural metadata of the image processing model to be optimized.

[0111] The parameter optimization module 204 is used to perform defect analysis on the image processing model to be optimized based on the optimization data source using a pre-trained defect analysis model, and obtain the model defect localization results and model parameter optimization instructions.

[0112] The instruction compilation module 206 is used to compile model parameter optimization instructions into parameter update opcodes, which are then pushed to the image processing model to be optimized for model optimization.

[0113] The device provided in this application embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0114] The methods and related apparatuses mentioned in the above embodiments are described with reference to the method flowcharts and / or structural diagrams provided in the embodiments of this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0115] The following embodiments illustrate the application of this method to a computer device. It is understood that the computer device can be any device with computing and processing capabilities, including but not limited to servers or personal laptops. In one embodiment, the computer device can be an application server, which can be a server used to run the application under test.

[0116] See Figure 3 This document illustrates a hardware block diagram of an electronic device intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0117] like Figure 3 As shown, the electronic device includes: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0118] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0119] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0120] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0121] The memory stores a program, which the processor can call. The program is used to implement the various processing steps of the aforementioned image processing model optimization scheme.

[0122] This invention also provides a readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements various processing flows of the image processing model optimization scheme provided in any possible implementation of the above embodiments and / or in combination with the embodiments.

[0123] The invention has been described in particular detail above with respect to possible scenarios, and those skilled in the art will recognize that the invention can be practiced through other embodiments. Specific naming of components, capitalization of terms, attributes, data structures, or any other programming or structural aspects are not mandatory or important, and the mechanisms or features of implementing the invention may have different names, forms, or procedures. The system can be implemented through a combination of hardware and software (as described), entirely through hardware elements, or entirely through software elements. The specific division of functions among the various system components described herein is merely exemplary and not mandatory; rather, the functions performed by a single system component can be performed by multiple components, or the functions performed by multiple components can be performed by a single component.

[0124] Those skilled in the art should understand that the various steps of the disclosed methods can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using device-executable program code, which can then be stored in a storage device for execution by the computing device. Alternatively, they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, the embodiments disclosed in this invention are not limited to any specific hardware and software combination.

[0125] The programs (also referred to as programs, software, software applications, or code) executable by these computing devices include machine instructions of a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0126] Certain aspects of this invention include the process steps and instructions described herein in algorithmic form. It should be noted that the process steps and instructions of this invention can be implemented in software, firmware, and / or hardware, and when implemented in software, they can be downloaded, stored on various operating systems and operated from said platforms.

[0127] Those skilled in the art will understand that the structures shown in the figures are merely block diagrams of some structures related to the present application and do not constitute a limitation on the terminal device to which the present application is applied. Specific terminal devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0128] In the description of this specification, the use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "possible design," etc., refers to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0129] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing model optimization method, characterized in that, include: The erroneous inference data of the image processing model to be optimized is denoted as the optimization data source, and the optimization data source includes the structural metadata of the image processing model to be optimized. Using a pre-trained defect analysis model, the image processing model to be optimized is analyzed for defects based on the optimization data source to obtain the model defect localization results and model parameter optimization instructions; The model parameter optimization instructions are compiled into parameter update opcodes, which are then pushed to the image processing model to be optimized for model optimization.

2. The method according to claim 1, characterized in that, The error inference data of the image processing model to be optimized, denoted as the optimization data source, includes: Obtain the image samples that are incorrectly inferred by the image processing model to be optimized and their corresponding image features; Record the current structural metadata of the image processing model to be optimized; The image samples, image features, and structural metadata are combined into an optimized data source.

3. The method according to claim 1, characterized in that, The step of using a pre-trained defect analysis model to perform defect analysis on the image processing model to be optimized based on the optimization data source, and obtaining model defect localization results and model parameter optimization instructions, includes: The image samples, image features, and structural metadata in the optimized data source are encoded to obtain corresponding feature vectors; Cross-modal feature fusion is performed on each feature vector using an attention mechanism to obtain fused features; The fusion features are matched with a predefined defect database to determine the model defects corresponding to the fusion features; The multi-task prediction head outputs model defect localization results and model parameter optimization instructions in parallel.

4. The method according to claim 1, characterized in that, The step of compiling the model parameter optimization instructions into parameter update opcodes includes: The parameter compilation model is used to compile the model parameter optimization instructions into initial values ​​of parameter update opcodes based on regular expression matching rules. Calculate the parameter increment based on the initial value of the update opcode according to the parameters; The parameter increment is constrained and verified based on preset constraints to obtain the verification parameter increment. The update accuracy is tested based on the increment of the verification parameter and the initial value of the parameter update opcode. If the accuracy of the image processing model to be optimized does not decrease after updating the parameters based on the initial value of the parameter update opcode, the initial value of the parameter update opcode is determined to be the final parameter update opcode.

5. The method according to claim 4, characterized in that, The preset constraints include: The norm of the parameter increment does not exceed the preset norm threshold, and the change in the output of the image model to be optimized under the effect of the parameter increment does not exceed the preset output change range.

6. The method according to claim 1, characterized in that, When multiple image processing models to be optimized are in the same system, the method further includes: The corresponding parameter update opcode is used to update the model parameters of each of the image processing models to be optimized, thereby obtaining the optimized image processing model. Obtain the scene fingerprint of each of the optimized image processing models; The update contribution of each optimized image processing model is calculated based on the fingerprint of each scenario. The global parameter update amount is calculated based on the update contribution of each optimized image processing model, and the global parameter update amount is pushed to each optimized image processing model for global update.

7. The method according to claim 6, characterized in that, The calculation of the global parameter update amount based on the update contribution of each optimized image processing model includes: The increments of the verification parameters of each optimized image processing model are weighted and calculated based on the update contribution of each optimized image processing model to obtain the weighted parameter increments. Gaussian noise is introduced into the weighted parameter increment to obtain the global parameter update.

8. An image processing model optimization device, characterized in that, include: The data acquisition module is used to acquire the erroneous inference data of the image processing model to be optimized, denoted as the optimization data source. The optimization data source includes the structural metadata of the image processing model to be optimized. The parameter optimization module is used to perform defect analysis on the image processing model to be optimized based on the optimization data source using a pre-trained defect analysis model, and obtain the model defect location results and model parameter optimization instructions. The instruction compilation module is used to compile the model parameter optimization instructions into parameter update opcodes, which are then pushed to the image processing model to be optimized for model optimization.

9. An electronic device, characterized in that, The device includes a memory storing computer-executable instructions and a processor, which, when executed by the processor, causes the device to perform the image processing model optimization method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, It stores a computer-executable program that, when executed, implements the image processing model optimization method as described in any one of claims 1 to 7.

Images