Dynamic optimization method and device of network model, equipment and storage medium

Abstract

The invention discloses a dynamic optimization method and device for a network model, equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of to-be-processed image data, and obtaining the feature information corresponding to the to-be-processed image data; obtaining performance feedback information of the target network model, and generating a target pruning strategy according to the feature information and the performance feedback information; and performing pruning operation on the target network model based on the target pruning strategy to obtain an optimized target network model.

Description

Technical Field

[0001] The present application relates to the field of model optimization technology, and in particular to a method, apparatus, device and storage medium for dynamic optimization of a network model. Background Art

[0002] With the increasing demand for high-quality image and video processing, many image processing technologies rely on pre-trained network models, which are typically run in high-performance computing environments. However, applying these image processing technologies on resource-constrained devices, such as mobile devices and smart TVs, can result in excessive energy consumption.

[0003] Existing approaches adjust the model architecture by statically pruning the model before deployment, thereby reducing its size and adapting it to resource-constrained devices. However, because the model input changes in real time, the model structure optimized by static pruning is fixed and cannot be dynamically adjusted to the characteristics of real-time input data, resulting in poor image processing performance. Summary of the Invention

[0004] The main purpose of this application is to provide a dynamic optimization method, device, equipment and storage medium for a network model, aiming to solve the technical problem that the existing static pruning method before model deployment is difficult to adapt to real-time changing input data, resulting in poor image processing effect.

[0005] To achieve the above objectives, the present application proposes a dynamic optimization method for a network model, which includes:

[0006] Perform feature extraction on the image data to be processed to obtain feature information corresponding to the image data to be processed;

[0007] Obtain performance feedback information of the target network model and generate a target pruning strategy based on the feature information and performance feedback information;

[0008] The target network model is pruned based on the target pruning strategy to obtain the optimized target network model.

[0009] In addition, to achieve the above objectives, the present application also proposes a dynamic optimization device for a network model, which includes:

[0010] An input analysis module is used to extract features from the image data to be processed and obtain feature information corresponding to the image data to be processed;

[0011] The pruning decision module is used to obtain performance feedback information of the target network model and generate a target pruning strategy based on the feature information and performance feedback information;

[0012] The model optimization module is used to perform pruning operations on the target network model based on the target pruning strategy to obtain the optimized target network model.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a dynamic optimization device for a network model, which includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor, and the computer program is configured to implement the steps of the dynamic optimization method for the network model as described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and stores a computer program on the storage medium. When the computer program is executed by the processor, it implements the steps of the dynamic optimization method of the network model as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 This is a flow chart of the first embodiment of the dynamic optimization method of the network model of the present application;

[0018] Figure 2 This is a flow chart of a second embodiment of the dynamic optimization method for a network model of the present application;

[0019] Figure 3 This is a flow chart of a third embodiment of the method for dynamic optimization of a network model of the present application;

[0020] Figure 4 A schematic diagram of the entire process of the dynamic optimization method of the network model of this application;

[0021] Figure 5 This is a structural block diagram of the first embodiment of the dynamic optimization device for the network model of the present application;

[0022] Figure 6 This is a structural diagram of the dynamic optimization device of the network model of this application.

[0023] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0024] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0025] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0026] The main solution of the embodiment of the present application is: feature extraction of the image data to be processed to obtain feature information corresponding to the image data to be processed; obtaining performance feedback information of the target network model, and generating a target pruning strategy based on the feature information and performance feedback information; pruning the target network model based on the target pruning strategy to obtain an optimized target network model.

[0027] Existing pruning techniques are typically static, meaning they are performed before the network model is deployed on the device. This static pruning approach, when processing dynamic video content, lacks a real-time feedback mechanism to dynamically adjust the model structure. This makes it difficult to adapt to real-time changes in input image data, ultimately resulting in poor image processing performance in the model output.

[0028] The present application provides a dynamic optimization method for a network model. By extracting features from input image data, a target pruning strategy is generated based on the feature information and performance feedback information of the target network model. Then, the target network model is pruned according to the target pruning strategy to achieve dynamic structural adjustment of the network model and obtain an optimized target network model. This ensures the reliability of the image processing results obtained based on the optimized target network model and improves the image processing efficiency.

[0029] It should be noted that this embodiment can be applied to scenarios requiring real-time image processing and optimization, such as image processing on mobile devices, real-time video analysis systems, and visual processing modules in autonomous driving. The execution entity of this embodiment can be a computing service device with data processing, network communication, and program execution capabilities, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of implementing the above functions, such as a smart TV. This embodiment and the following embodiments will be described below using a smart TV as an example.

[0030] Based on this, the embodiment of the present application proposes a dynamic optimization method of a network model, which is applied to the user end, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the dynamic optimization method of the network model of the present application. In this embodiment, the dynamic optimization method of the network model includes steps S10 to S30:

[0031] Step S10: extracting features from the image data to be processed to obtain feature information corresponding to the image data to be processed.

[0032] It should be noted that the image data to be processed may be a single image acquired in real time, or may be a plurality of video frames in a continuous video frame sequence acquired by the smart TV from a pushed video stream.

[0033] In smart TVs, feature extraction algorithms of different dimensions can be used to extract features from the image data to be processed, obtaining features such as image complexity, motion vectors, and brightness changes, so as to improve the adaptability of the pruning strategy for subsequent decision generation and the model processing efficiency.

[0034] It should also be noted that in order to ensure that the numerical scales of the features of the above categories are consistent, the features of the above categories can be normalized to avoid deviations caused by differences in the numerical ranges of the features, and then the features of each category after data normalization can be aggregated into feature vectors to obtain feature information.

[0035] Furthermore, in order to specifically illustrate the feature analysis and extraction process of the image data to be processed, step S10 specifically includes: steps S101 to S103:

[0036] Step S101: Obtaining features of various categories of image data to be processed through a preset feature extraction algorithm.

[0037] It is understood that each category of features may include: image features, motion features, and brightness features. Image features are image complexity, motion features are motion vectors, and brightness features are brightness variation amplitudes. After calculating the edge density value and texture complexity, they can be weighted and summed to obtain a comprehensive image complexity value, which can be used as the image feature corresponding to the image data to be processed.

[0038] It should be understood that the calculation of image complexity can include two parts: edge density value and texture complexity. Edge density is the proportion of edge pixels in an image. Edge density value can be obtained by performing edge detection on the processed image data. Common edge detection processes can be implemented using edge detection algorithms (such as the Canny algorithm and the Sobel algorithm). They can also be implemented by using a deep learning model dedicated to edge detection (such as HED and DeepEdge), or by converting the image to the frequency domain and performing energy statistics of the high-frequency part. This embodiment is not limited to this, and the Canny algorithm is used as an example for illustration.

[0039] For example, when the Canny algorithm is used to obtain edge positions in the image to be processed, the number of edge pixels corresponding to each edge position is counted and then divided by the total number of pixels in the image to obtain an edge density value.

[0040] Texture complexity measures the richness of texture detail in an image. Texture analysis of the processed image data can be performed using methods such as gray-level co-occurrence matrix (GLCM), local binary pattern (LBP), and Gabor filter. This embodiment is not limited to this, and the GaborL filter is used as an example for illustration.

[0041] For example, by convolving the image data with multiple pre-designed Gabor filters designed to capture different orientations and scales in the image, a set of filter response maps can be generated. Statistical features such as mean, variance, energy, and entropy are then extracted from each filter response map. A weighted sum of these statistical features is then taken to obtain a comprehensive texture complexity value.

[0042] It should be noted that motion vector analysis of the image data to be processed can assess the severity of the scene changes corresponding to the image data to be processed by analyzing the pixel motion between consecutive images (video frames). Optical flow-based algorithms (such as the Lucas-Kanade algorithm and the Farneback algorithm) can be used to estimate optical flow by minimizing the displacement error of feature points between adjacent video frames. Block matching-based algorithms (such as the full search algorithm and the fast search algorithm) can also be used to estimate motion vectors by calculating the similarity between image blocks. The motion vector obtained by analysis is then determined as the motion feature corresponding to the image data to be processed.

[0043] It should also be noted that calculating the brightness change between video frames of the image data to be processed can identify scene changes and scene lighting changes corresponding to the image data to be processed by calculating the brightness and color change amplitudes between video frames. Histogram comparison and color space conversion methods can be used to calculate the brightness (and color) change amplitudes. The histogram comparison method can quantify the changes between frames by calculating the difference between two histograms corresponding to consecutive video frames (such as Euclidean distance and chi-square distance); the color space conversion method can convert video frames from the RGB color space to other color spaces that have better brightness and chromaticity separation (such as HSV and LAB), and then calculate the changes in the hue (H), saturation (S), and brightness (V) channels of the video frames to obtain the brightness change amplitude corresponding to the image data to be processed, and determine it as the brightness feature of the image to be processed.

[0044] In a specific implementation, edge detection can be performed on the image data to be processed to obtain edge density; texture analysis can be performed on the image data to be processed to obtain texture complexity; the edge density value and texture complexity can be feature-weighted fused to synthesize the image features of the image data to be processed; motion vector analysis can be performed between video frames on the image data to be processed to obtain the corresponding motion vector, and this can be determined as the motion feature of the image data to be processed; brightness change calculation can be performed between video frames on the image data to be processed to obtain the brightness change amplitude, and this can be determined as the brightness feature of the image data to be processed.

[0045] Step S102: performing a preprocessing operation on each category feature to obtain preprocessed features to be aggregated.

[0046] It should be understood that the preprocessing operation may include: feature standardization operation, feature normalization operation and feature dimension reduction operation.

[0047] In the specific implementation, each category feature is first standardized and normalized to ensure that the numerical scales of different features are consistent. This allows for reasonable comparison and integration when generating pruning strategies, avoiding bias caused by differences in feature numerical ranges, thereby improving the accuracy and effectiveness of the resulting pruning strategies. Feature dimensionality reduction techniques (such as principal component analysis (PCA)) can then be used to reduce the dimensionality of the aforementioned category features, thereby reducing complexity. After preprocessing, the features to be aggregated are obtained.

[0048] Step S103: performing feature aggregation on each feature to be aggregated, generating a first feature vector, and determining it as feature information corresponding to the image data to be processed.

[0049] It should be noted that the feature information may be a feature representation of the image data to be processed, so as to serve as an input feature vector generated by a subsequent pruning strategy.

[0050] In a specific implementation, the above-mentioned features to be aggregated can be aggregated to form a first feature vector that integrates features of various categories, which serves as feature information corresponding to the image data to be processed for subsequent decision-making and generation of pruning strategies.

[0051] Step S20: Obtain performance feedback information of the target network model, and generate a target pruning strategy based on the feature information and the performance feedback information.

[0052] It should be noted that the performance feedback information of the target network model can be generated by performing a performance evaluation on the image processing results output by the target network model at the previous moment. This performance feedback information can be used as a performance maintenance strategy for adjusting the initial pruning strategy, which can dynamically adjust the initial pruning strategy based on the real-time performance of the target network model to adapt to changing model inputs and performance requirements.

[0053] It should be understood that after the smart TV generates the feature information corresponding to the image data to be processed, it can use it as the basic data for generating the pruning strategy: first, the feature information is input into the pre-trained strategy generation model to obtain the initial pruning strategy, and then the initial pruning strategy is dynamically adjusted according to the above performance feedback information, and finally the target pruning strategy is obtained to guide the adjustment of the target network model structure.

[0054] It should be noted that the pre-trained policy generation model, or the preset policy generation model, can be constructed using a machine learning algorithm such as a reinforcement learning algorithm, a decision tree, or a support vector machine, and trained using historical policy generation data. The initial pruning strategy generated based on this model can include, for example, the pruned network layers and pruning ratios. The performance feedback information, or the performance maintenance strategy, can include the pruned network layers that need to be restored and the pruning ratios determined based on the performance evaluation results.

[0055] Step S30: performing a pruning operation on the target network model based on the target pruning strategy to obtain an optimized target network model.

[0056] It should be understood that, first, the pruning object and pruning ratio can be determined in the target network model according to the target pruning strategy. The pruning object can be a specific neuron, connection or even the entire network layer in the target network model, and the pruning ratio can reflect the intensity of pruning, such as the proportion of neurons or connections to be pruned.

[0057] In a specific implementation, the smart device can perform pruning operations in the current network model based on the determined pruning objects and pruning ratios to remove unimportant neurons, connections, or network layers, and structurally adjust the pruned current network model to ensure model integrity, ultimately obtaining an optimized target network model. By removing unimportant parts during the optimization process, the optimized target network model can reduce the number of model parameters, thereby increasing the model's running speed by reducing the amount of computation. Furthermore, since pruning is performed using a reasonable current pruning strategy, the stability of the model's performance can be ensured.

[0058] This embodiment extracts features from input image data, generates a target pruning strategy based on the feature information and performance feedback information of the target network model, and then performs pruning operations on the target network model according to the target pruning strategy to achieve dynamic structural adjustment of the network model and obtain an optimized target network model. This ensures the reliability of the image processing results obtained based on the optimized target network model and improves the image processing efficiency.

[0059] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 2 , Figure 2 This is a flow chart of the second embodiment of the dynamic optimization method of the network model of the present application.

[0060] In this embodiment, in order to specifically illustrate the specific process of deciding to generate a pruning strategy, step S20 specifically includes: steps S201 to S203:

[0061] Step S201: Obtain performance feedback information of the target network model at the current moment.

[0062] It should be understood that the performance feedback information is generated based on the performance evaluation results of the target network model corresponding to the image processing results at the previous moment. The performance feedback information is a performance maintenance strategy used to adjust the initial pruning strategy, including pruning strategy adjustment information and / or pruned object recovery information.

[0063] Step S202: Input the feature information into a preset strategy generation model to obtain a prediction pruning strategy.

[0064] It can be understood that the preset strategy generation model is a model constructed based on a machine learning algorithm such as a reinforcement learning algorithm, a decision tree or a support vector machine, and trained using historical strategy generation data. The model can learn and generate pruning strategies during the training process.

[0065] In a specific implementation, feature information is input into the preset strategy generation model, which can predict the pruning strategy and output a predicted pruning strategy. The predicted pruning strategy can include the pruning method (structural pruning or weight pruning), the pruned object (neurons, connections, or network layers), and the pruning ratio.

[0066] Step S203: Adjust the predicted pruning strategy according to the performance feedback information to obtain a target pruning strategy.

[0067] It should be noted that after the model outputs the predicted pruning strategy, it can also be combined with heuristic rules to quickly generate an initial pruning strategy, and then further optimize the initial pruning strategy. Therefore, step S203 specifically includes: steps S2031 to S2033:

[0068] Step S2031: Scoring several prunable objects in the target network model based on heuristic rules, and updating the predicted pruning strategy according to the scoring result of each prunable object to obtain an initial pruning strategy.

[0069] It is understandable that the heuristic rules can be based on indicators such as the activation frequency and weight size of neurons, connections or network layers in the target network model. First, the prunable objects (neurons, connections or network layers) in the target network model can be scored based on the activation frequency or weight size indicators. Prunable objects with lower activation frequency and / or weight are judged to be less important to the model and can be pruned first, that is, they are prioritized as the target pruning objects in the pruning strategy; conversely, prunable objects with higher activation frequency and / or weight are judged to be more important to the model and can be retained first, that is, they are preferentially removed from the pruning strategy.

[0070] For example, the higher the activation frequency of a neuron, the more important it is in the model. The importance of the neuron can be evaluated by counting its activation frequency, and whether to prune it can be decided based on the importance score. The size of the weight also reflects the importance of the neuron in the model. The smaller the weight, the smaller the contribution of the neuron to the model, and it can be pruned first.

[0071] In a specific implementation, the predicted pruning strategy can be updated based on the heuristic rules to generate an initial pruning strategy, and the pruning objects that need to be pruned and the pruning ratio can be preliminarily determined.

[0072] Step S2032: Initialize the population based on the initial pruning strategy, perform crossover mutation operations on each individual in the population to obtain an updated population, calculate the fitness value of each individual in the updated population according to preset optimization target parameters, and determine the pruning strategy corresponding to the individual with the highest fitness value as the first optimized pruning strategy.

[0073] It should be understood that the preset optimization target parameters may be parameters pre-set according to the optimization target, which may include: computational efficiency, model accuracy, model energy consumption, etc.

[0074] It should be noted that the generation process of the first optimized pruning strategy can be implemented based on a multi-objective optimization algorithm (such as a genetic algorithm or a particle swarm optimization algorithm).

[0075] For example, first, a set of pruning strategies can be randomly generated as the initial population based on the initial pruning strategy, and each individual in the initial population corresponds to a possible pruning strategy; then, the preset optimization target parameters are used to calculate each individual, and the performance of each individual on each optimization target is calculated separately, and quantified by the fitness value; then, crossover and mutation operations are performed on each individual to generate new individuals to enhance the diversity of the population, and then the fitness values ​​of the individuals in the updated population are recalculated; the cycle is repeated until the preset termination condition is met, and the pruning strategy corresponding to the individual with the highest fitness value is determined as the first optimized pruning strategy, ensuring that the first optimized pruning strategy can meet the effect of balancing computing efficiency, accuracy and energy consumption.

[0076] Step S2033: Adjust the first optimized pruning strategy according to the performance feedback information to obtain a target pruning strategy.

[0077] It should be understood that in order to achieve adaptive optimization and continuously improve model performance, the first optimized pruning strategy can be adjusted based on performance feedback information generated based on the performance evaluation results. That is, the pruning objects and pruning ratios determined by the first optimized pruning strategy can be readjusted to obtain the current pruning strategy. This allows the pruning strategy to be dynamically adjusted to adapt to changing input image data and model performance.

[0078] This embodiment uses feature information as the basis for pruning decisions. Based on this information, a machine learning algorithm and heuristic rules are used to generate an initial pruning strategy, determining the objects to be pruned and the pruning ratio. This initial pruning strategy is then optimized using a multi-objective optimization algorithm, balancing computational efficiency and model performance. The strategy is then adjusted based on performance feedback, ultimately outputting a dynamically adaptable current pruning strategy to guide pruning operations. This dynamic pruning technology reduces unnecessary computation and lowers energy consumption for devices (such as smart TVs). This helps extend the lifespan of the device while reducing electricity costs for users.

[0079] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and second embodiments can be referred to the above introduction and will not be described in detail later. Figure 3 , Figure 3 This is a flow chart of the third embodiment of the dynamic optimization method of the network model of the present application.

[0080] In this embodiment, in order to specifically illustrate how to collect user behavior data, step S30 specifically includes: steps S301 to S302:

[0081] Step S301: determining a target pruning object and a target pruning method among all prunable objects according to a target pruning strategy.

[0082] It should be understood that the target pruning strategy may include a target pruning method and a corresponding target pruning object. The target pruning method may be a weight pruning method and / or a structure pruning method.

[0083] The target pruning object can be divided into weight pruning objects (connections) and structure pruning objects (neuron pruning objects and network layer pruning objects) according to the target pruning method.

[0084] Step S302: performing a pruning operation on the target pruning object in the target network model according to the target pruning method, and updating the model parameters of the pruned target network model to obtain the target network model.

[0085] It should be understood that when performing weight pruning, connections with smaller connection weights can be removed in the target network model, that is, the weights of connections with weights less than a threshold are set to zero; when performing neuron pruning, neurons with lower activation frequency or smaller weights can be removed in the target network model, that is, all input and output connection weights of neurons are set to zero; when performing layer pruning, redundant network layers can be directly removed in the target network model, and the leveling of the previous and next network layers can be adjusted.

[0086] It should be noted that after pruning, the model parameters can also be adjusted to adapt to the new network structure. The model parameter adjustment may include: retraining and fine-tuning the model to ensure the performance of the model. In addition, in order to ensure the integrity of the model, the model network structure can also be dynamically adjusted to ensure that the pruned model can still effectively perform subsequent image processing processes. The model network structure adjustment may include: adjusting weights (normalizing or reinitializing the remaining weights), adjusting the layer structure (if the entire network layer is removed, the connection between the front and back layers needs to be adjusted), and reconnecting (if certain connections are removed, the weights of the remaining connections need to be adjusted). Through the above-mentioned model parameter adjustment and model network structure adjustment, model optimization can be achieved to obtain an optimized target network model, so that the image data to be processed can be subjected to subsequent image processing processes through the optimized target network model.

[0087] In addition, in order to prevent the performance of the model from degrading after pruning, a recovery mechanism can be introduced to monitor the model performance in real time when subsequent image processing is performed based on the optimized target network model to ensure that the pruned objects are quickly restored when the model performance degrades or the input changes.

[0088] Furthermore, in order to achieve continuous optimization of the network model, the network model performance can be monitored in real time: the image processing results of the network model at different times are evaluated according to pre-set evaluation indicators, and performance feedback information is generated based on the performance evaluation results to adjust the pruning strategy and model structure to ensure the efficient operation and stable performance of the network model under different input conditions. Therefore, after step S30, the following steps are also included:

[0089] Step S401: input the image data to be processed into the optimized target network model to obtain the image processing result corresponding to the optimized target network model at the current moment.

[0090] It should be understood that the image processing result may be a processed image corresponding to the image to be processed.

[0091] Step S402: Calculate the performance score of the image processing result based on the preset evaluation index.

[0092] It should be understood that the evaluation indicators include: image quality indicator, processing delay indicator and operation energy consumption indicator.

[0093] It is understood that image quality indicators are used to evaluate the image quality of processed images, and can be, for example, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). Processing delay indicators can reflect the real-time performance of the target network model by measuring the time required for the target network model to process a single frame of image. Operational energy consumption indicators can evaluate the energy consumption of the target network model when performing image processing tasks, especially on resource-constrained devices (such as smart TVs), which helps optimize device lifespan.

[0094] In the specific implementation, the performance of the image processing results is evaluated based on the above-mentioned evaluation indicators, and the three types of performance score values, namely, image quality performance score value, processing delay time, and task running time, can be obtained in turn, which can quantify the performance evaluation results of the optimized target network model.

[0095] Step S403: When the performance score value is lower than the preset model performance threshold, an abnormal impact index is determined from the preset evaluation indicators according to the performance evaluation value, and historical pruning experience data is queried through the abnormal impact index to obtain a performance maintenance strategy.

[0096] It should be understood that the model performance threshold corresponding to each evaluation indicator can be set in advance, and then the above three different performance score values ​​can be compared with the model performance threshold of the corresponding category to determine whether the optimized target network model has performance anomalies and abnormal performance items.

[0097] The model performance thresholds of the evaluation indicators may be: PSNR threshold, SSIM threshold, maximum delay time, and maximum energy consumption time.

[0098] For example, when the calculated PSNR value is lower than the signal-to-noise ratio threshold, the SSIM value is lower than the structural similarity threshold, the delay time is higher than the maximum delay time, or the time taken to run the task during the processing delay time is higher than the maximum energy consumption, the optimized target network model has performance abnormalities.

[0099] It is understood that when a performance anomaly is identified, the anomaly impact indicator can be determined from the pre-set evaluation metrics to further identify the specific performance anomaly, such as image quality degradation, increased latency, or increased energy consumption. Subsequently, combined with input feature analysis, the anomaly can be initially located in the network model at the network layer. By querying historical pruning experience data, anomaly resolution strategies for the same or similar situations can be obtained. Finally, through multiple tests, a performance maintenance strategy can be generated.

[0100] Specifically, the performance maintenance strategy is generated as follows: After identifying the specific performance anomaly, the algorithm first uses the image data being processed as input to the target network model to initially locate the abnormal network layer (shallow or deep) within the target network model. Shallow networks are generally more sensitive to low-level features of the input data (such as edges and textures), while deep networks are generally more sensitive to high-level semantic features (such as object shape and category). For example, if the performance evaluation results show high texture complexity and degraded image quality, this may be related to the shallow network layer.

[0101] Next, the system combines the pruning history and the importance scores of each target pruned object to identify the pruned objects that may have caused the anomaly and consider them as possible objects to be restored. For example, if image quality degrades after a pruning operation, the system will focus on examining the target pruned objects involved in that pruning operation and consider them as possible objects to be restored.

[0102] It should also be noted that the basis for determining the objects to be restored can be: during the experimental verification process, the objects to be restored can be gradually restored from several possible objects to be restored (for example, first restoring connections with smaller weights, then restoring neurons with lower activation frequencies). After each restoration, performance is re-evaluated based on the above evaluation indicators, and the corresponding restoration operation and its impact on model performance are recorded. This helps to quickly determine the objects to be restored in different performance anomalies.

[0103] Finally, based on the number and type of objects to be restored, a performance maintenance strategy is determined. This strategy consists of pruning strategy adjustment information and / or pruned object recovery information. Pruning strategy adjustment is a lightweight optimization operation, primarily used in scenarios with slight deviations in performance metrics or slight changes in input characteristics. It can quickly optimize computational efficiency, but the performance improvement is limited. Pruned object recovery is a more complex operation, primarily used in scenarios with severe performance degradation or drastic changes in input characteristics. It can significantly improve model performance, but increases computational complexity and resource consumption.

[0104] The choice between the two depends on the running status and performance requirements of the model. When the performance deviation is small, the pruning strategy is adjusted first for rapid optimization. When the performance degrades severely or the input characteristics change drastically, a recovery operation can be performed to ensure model performance.

[0105] In a specific implementation, after generating a performance maintenance strategy based on the above process, the target network model can use the next performance feedback information received from the target network model as input to the generation of the next pruning strategy for pruning and updating the target network model, thereby reducing the latency and computational burden of subsequent image processing based on the pruned and updated target network model. Alternatively, the performance maintenance strategy can be used to directly trigger a recovery mechanism for the target network model, allowing the restored target network model to reprocess the current image data, ensuring the quality of the processed image corresponding to the current image data.

[0106] This embodiment continuously monitors the model's performance indicators, such as image quality, processing latency, and energy consumption, to assess whether the model is experiencing performance anomalies. When such anomalies occur, performance feedback is generated to indicate adjustments to the pruning strategy or trigger a pruned object recovery mechanism. This enables dynamic adjustment of the pruning strategy and model structure, ensuring efficient model operation and continuous optimization.

[0107] Further, here you can refer to Figure 4 , the whole process of the dynamic optimization method of the network model of this application is explained. Figure 4 This is a schematic diagram of the entire process of the dynamic optimization method of the network model of this application.

[0108] exist Figure 4 The entire process of the dynamic optimization method of the network model of this application can be divided into four stages: input feature analysis stage, pruning decision stage, model adjustment stage, and performance monitoring and feedback stage.

[0109] In the input analysis stage: first, feature extraction is performed on the image data to be processed, then the extracted features of each category are preprocessed to obtain the preprocessed features of each category, and finally the preprocessed features of each category are aggregated to obtain a comprehensive feature vector as the feature information output corresponding to the image data to be processed.

[0110] In the pruning decision stage: first, feature information is used as the basic data for pruning decisions, and the initial pruning strategy is generated by combining machine learning algorithms and heuristic rules; then the initial pruning strategy is optimized through a multi-objective optimization algorithm and performance feedback information to obtain the target pruning strategy.

[0111] During the model adjustment phase, the target network model is pruned according to the target pruning strategy. The pruned target network model is then parameter updated and its structure adjusted to obtain an optimized target network model. Furthermore, when a recovery mechanism is triggered based on performance feedback, the target network model is restored.

[0112] During the performance monitoring and feedback phase: continuously monitor the model's performance indicators such as image quality, processing latency, and energy consumption to evaluate whether the model has performance anomalies; generate performance feedback information when performance anomalies occur to indicate adjustments to the pruning strategy or trigger the pruned object recovery mechanism.

[0113] Based on the above stages, this embodiment realizes the dynamic adjustment of pruning strategy and model structure, ensuring the efficient operation and continuous optimization of the model.

[0114] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the dynamic optimization method of the network model of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0115] In addition, this application also provides a dynamic optimization device for a network model, referring to Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the dynamic optimization device for the network model of this application; Figure 5 As shown, the device includes:

[0116] An input analysis module 501 is used to extract features from the image data to be processed and obtain feature information corresponding to the image data to be processed;

[0117] The pruning decision module 502 is used to obtain performance feedback information of the target network model and generate a target pruning strategy based on the feature information and the performance feedback information;

[0118] The model optimization module 503 is used to perform a pruning operation on the target network model based on the target pruning strategy to obtain an optimized target network model.

[0119] The performance monitoring module 504 is used to input the image data to be processed into the optimized target network model to obtain the image processing result corresponding to the optimized target network model at the current moment; calculate the performance score value of the image processing result based on preset evaluation indicators, the preset evaluation indicators include: image quality indicator, processing delay indicator and operation energy consumption indicator; when the performance score value is lower than the preset model performance threshold, determine the abnormal impact indicator in the preset evaluation indicators according to the performance evaluation value, query historical pruning experience data through the abnormal impact indicator, obtain a performance maintenance strategy, and determine the performance maintenance strategy as the performance feedback information of the target network model at the next moment, and the performance maintenance strategy is pruning strategy adjustment information and / or pruning object recovery information.

[0120] Furthermore, the input analysis module 501 is also used to obtain various category features of the image data to be processed through a preset feature extraction algorithm: each category feature includes: image features, motion features and brightness features; preprocessing operations are performed on each category feature to obtain each preprocessed feature to be aggregated, and the preprocessing operations include: feature standardization operations, feature normalization operations and feature dimensionality reduction operations; feature aggregation is performed on each feature to be aggregated to generate a first feature vector, and it is determined as feature information corresponding to the image data to be processed.

[0121] Furthermore, the input analysis module 501 is also used to perform edge detection on the image data to be processed to obtain edge density; perform texture analysis on the image data to be processed to obtain texture complexity; perform feature-weighted fusion on the edge density value and the texture complexity to synthesize the image features of the image data to be processed; perform motion vector analysis between video frames on the image data to be processed to obtain the corresponding motion vector, and determine it as the motion feature of the image data to be processed; perform brightness change calculation between video frames on the image data to be processed to obtain the brightness change amplitude, and determine it as the brightness feature of the image data to be processed.

[0122] Furthermore, the pruning decision module 502 is also used to obtain performance feedback information of the target network model at the current moment, where the performance feedback information is generated based on the image processing result corresponding to the target network model at the previous moment; input the feature information into the preset strategy generation model to obtain a predicted pruning strategy; and adjust the predicted pruning strategy according to the performance feedback information to obtain a target pruning strategy.

[0123] Furthermore, the pruning decision module 502 is also used to score several prunable objects in the target network model based on heuristic rules, and update the predicted pruning strategy according to the scoring results of each prunable object to obtain an initial pruning strategy; initialize the population based on the initial pruning strategy, and perform crossover mutation operations on each individual in the population to obtain an updated population, calculate the fitness value of each individual in the updated population according to preset optimization target parameters, and determine the pruning strategy corresponding to the individual with the highest fitness value as the first optimized pruning strategy; adjust the first optimized pruning strategy according to performance feedback information to obtain a target pruning strategy.

[0124] Furthermore, the model optimization module 503 is used to determine the target pruning object and the target pruning method among the prunable objects according to the target pruning strategy; perform pruning operations on the target pruning object in the target network model according to the target pruning method, and update the model parameters of the pruned target network model to obtain the target network model.

[0125] This embodiment extracts features from input image data, generates a target pruning strategy based on the feature information and performance feedback information of the target network model, and then performs pruning operations on the target network model according to the target pruning strategy to achieve dynamic structural adjustment of the network model and obtain an optimized target network model. This ensures the reliability of the image processing results obtained based on the optimized target network model and improves the image processing efficiency.

[0126] The present application also provides a dynamic optimization device for a network model, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the dynamic optimization method for the network model in the above-mentioned embodiment one.

[0127] Reference below Figure 6 , which shows a schematic diagram of the structure of a dynamic optimization device for a network model suitable for implementing an embodiment of the present application. The dynamic optimization device for a network model in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The dynamic optimization device of the network model shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0128] like Figure 6As shown, the network model dynamic optimization device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory 1002 or programs loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the network model dynamic optimization device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the network model dynamic optimization device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a network model dynamic optimization device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have alternatively.

[0129] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the dynamic optimization method of the network model in the above-mentioned embodiment.

[0130] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0131] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned method for dynamic optimization of network models, thereby resolving the technical problem of dynamic optimization of network models. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the method for dynamic optimization of network models provided in the aforementioned embodiments, and are not further elaborated here.

[0132] The present application also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned method for dynamic optimization of the network model when executed by a processor.

[0133] The computer program product provided in this application can solve the technical problem of dynamic optimization of network models. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the dynamic optimization method of the network model provided in the above embodiment, and will not be repeated here.

[0134] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional elements in the process, method, article, or system comprising the element.

[0135] The serial numbers of the above-mentioned embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments. Moreover, they are only some embodiments of the present application and do not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the description and drawings of the present application under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A dynamic optimization method for a network model, characterized in that: The method comprises: Performing feature extraction on the image data to be processed to obtain feature information corresponding to the image data to be processed; Obtaining performance feedback information of the target network model, and generating a target pruning strategy based on the feature information and the performance feedback information; A pruning operation is performed on the target network model based on the target pruning strategy to obtain the optimized target network model.

2. The method according to claim 1, wherein The step of extracting features from the image data to be processed to obtain feature information corresponding to the image data to be processed includes: Obtaining various category features of the image data to be processed by a preset feature extraction algorithm; each of the category features includes: image features, motion features, and brightness features; Performing a preprocessing operation on each of the category features to obtain preprocessed features to be aggregated, wherein the preprocessing operation includes: feature standardization operation, feature normalization operation and feature dimensionality reduction operation; Perform feature aggregation on each of the features to be aggregated to generate a first feature vector, and determine it as feature information corresponding to the image data to be processed.

3. The method according to claim 2, wherein The step of obtaining the features of each category of the image data to be processed by using a preset feature extraction algorithm includes: Performing edge detection on the image data to be processed to obtain edge density; Performing texture analysis on the image data to be processed to obtain texture complexity; Performing feature-weighted fusion on the edge density value and the texture complexity to synthesize image features of the image data to be processed; Performing motion vector analysis between video frames on the image data to be processed to obtain corresponding motion vectors, and determining the corresponding motion vectors as motion features of the image data to be processed; The brightness change between video frames of the image data to be processed is calculated to obtain a brightness change amplitude, and the amplitude is determined as the brightness feature of the image data to be processed.

4. The method according to claim 1, wherein The step of obtaining performance feedback information of the target network model and generating a current pruning strategy according to the feature information and the performance feedback information includes: Obtaining performance feedback information of the target network model at a current moment, where the performance feedback information is generated based on an image processing result corresponding to the target network model at a previous moment; Inputting the feature information into a preset strategy generation model to obtain a predicted pruning strategy; The predicted pruning strategy is adjusted according to the performance feedback information to obtain a target pruning strategy.

5. The method according to claim 4, wherein The step of adjusting the predicted pruning strategy according to the performance feedback information to obtain a target pruning strategy includes: Scoring a number of prunable objects in the target network model based on heuristic rules, and updating the predicted pruning strategy according to the scoring results of each of the prunable objects to obtain an initial pruning strategy; Initializing a population based on the initial pruning strategy, performing a crossover mutation operation on each individual in the population to obtain an updated population, calculating the fitness value of each individual in the updated population according to preset optimization target parameters, and determining the pruning strategy corresponding to the individual with the highest fitness value as the first optimized pruning strategy; The first optimized pruning strategy is adjusted according to the performance feedback information to obtain a target pruning strategy.

6. The method according to claim 5, wherein The step of performing a pruning operation on the target network model based on the target pruning strategy to obtain the optimized target network model includes: Determining a target pruning object and a target pruning method among the prunable objects according to the target pruning strategy; A pruning operation is performed on the target pruning object in the target network model according to the target pruning method, and model parameters of the pruned target network model are updated to obtain the optimized target network model.

7. The method according to claim 1, wherein After the step of obtaining the optimized target network model, the method further includes: Inputting the image data to be processed into the optimized target network model to obtain the image processing result corresponding to the optimized target network model at the current moment; Calculating a performance score of the image processing result based on preset evaluation indicators, wherein the preset evaluation indicators include: image quality indicator, processing delay indicator, and operation energy consumption indicator; When the performance score value is lower than a preset model performance threshold, an abnormal impact indicator is determined from the preset evaluation indicators according to the performance evaluation value, historical pruning experience data is queried through the abnormal impact indicator to obtain a performance maintenance strategy, and the performance maintenance strategy is determined as performance feedback information of the target network model at the next moment, and the performance maintenance strategy is pruning strategy adjustment information and / or pruning object recovery information.

8. A dynamic optimization device for a network model, characterized in that: The device comprises: An input analysis module is used to extract features from the image data to be processed and obtain feature information corresponding to the image data to be processed; A pruning decision module is used to obtain performance feedback information of the target network model and generate a target pruning strategy based on the feature information and the performance feedback information; A model optimization module is used to perform a pruning operation on the target network model based on the target pruning strategy to obtain the optimized target network model.

9. A dynamic optimization device for a network model, characterized in that: The device includes a memory, a processor, and a dynamic optimization program for a network model stored in the memory and runnable on the processor. When the dynamic optimization program for the network model is executed by the processor, the steps of the dynamic optimization processing method for the network model as described in any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores a dynamic optimization program for a network model, and when the dynamic optimization program for a network model is executed by a processor, the steps of the dynamic optimization processing method for a network model as described in any one of claims 1 to 7 are implemented.

Images