An image data processing method and system based on a lightweight sparse neural network

Abstract

The application provides an image data processing method and system based on a lightweight sparse neural network, and relates to the technical field of machine learning, comprising: obtaining image data to be processed; sequentially performing random flipping, random cropping data enhancement and tensor standardization preprocessing on the image data to obtain standard tensor data meeting model input requirements; inputting the standard tensor data into a lightweight sparse neural network model; the model extracts local edge and texture features and global semantic features from the image through multiple sparse convolution layers; the physical features extracted are mapped to class confidence through a full connection layer, and finally a classification result is output through a Softmax function. Through the gradient-guided differential mutation strategy and GPU parallel optimization, the application realizes efficient image processing on resource-limited mobile terminals, embedded devices and edge computing devices, reduces the calculation overhead and memory occupation, and at the same time maintains high-precision classification performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to an image data processing method and system based on a lightweight sparse neural network. BACKGROUND

[0002] The statements in this section merely provide background technology related to the present disclosure and do not necessarily constitute prior art.

[0003] In recent years, deep learning-based image processing techniques such as image classification, object detection, and character recognition have been widely applied to intelligent monitoring, autonomous driving, mobile document scanning and recognition (such as license plates, door plates, and bills), etc. These applications usually need to implement real-time or near-real-time image processing on resource-constrained hardware such as mobile terminals, embedded systems, or edge computing devices. However, in order to achieve high precision, the current mainstream convolutional neural networks (such as VGG, ResNet) are often designed as parameter-intensive deep models, which leads to the following technical problems:

[0004] Firstly, the huge model size and computational overhead limit its deployment in resource-constrained environments. For example, when performing real-time license plate recognition in a vehicle-mounted camera or offline document digitization on a mobile phone, the large model will bring unbearable memory occupation and computational delay, seriously affecting the system response speed and user experience.

[0005] Secondly, to address the above problems, existing model compression and acceleration methods have obvious shortcomings in image processing tasks. Although traditional iterative pruning methods (such as the IMP algorithm based on the "lottery ticket hypothesis") can obtain sparse models, their search process converges slowly, is prone to local optimization, and is a serial process with low efficiency, making it difficult to quickly generate efficient sparse sub-networks for different image tasks. In addition, although evolutionary algorithms have global search capabilities and can find better sparse structures, their inherent defects of slow convergence and high computational cost make them difficult to be directly applied to high-dimensional image neural network optimization.

[0006] Therefore, there is a lack of a method in the prior art that can efficiently and automatically search and generate high-performance lightweight sparse neural networks for complex image processing tasks, which is a key bottleneck for the widespread application of advanced image recognition technology on resource-constrained hardware such as mobile terminals, embedded systems, or edge computing devices. SUMMARY

[0007] In order to solve at least one of the technical problems in the above background technology, in particular the high computational overhead, high memory occupation, and low energy efficiency problems faced when deploying deep learning models on edge computing and mobile devices, the present application proposes an image data processing method and system based on a lightweight sparse neural network.

[0008] The first aspect of the present application provides a lightweight sparse neural network-based image data processing method, applied to mobile terminals, embedded devices or edge computing devices, comprising:

[0009] Obtaining image data to be processed;

[0010] Sequentially performing random flipping, random cropping data enhancement and tensor standardization preprocessing on the image data to obtain standard tensor data meeting the model input requirements;

[0011] Inputting the standard tensor data into a lightweight sparse neural network model; the model extracts physical features from the image step by step through multiple sparse convolution layers, including local edge and texture features extracted by the bottom convolution layer, and global semantic features formed by the integration of high-level convolution layers; the local edge and texture features are used to represent the basic structural information in the image, and the global semantic features are used to represent the category attributes of the image;

[0012] Mapping the extracted physical features to category confidence through a fully connected layer, and finally outputting the classification result through a Softmax function.

[0013] Further, the lightweight sparse neural network model is obtained through the following training process:

[0014] Randomly initializing a shared weight dense neural network and multiple continuous masks in the GPU global memory; by starting the CUDA kernel, scheduling the stream multi-processor and CUDA core of the GPU, performing element-wise multiplication on the weight tensor and each mask tensor, generating multiple sparse sub-networks to constitute an initial population; the discrete binary mask is processed into a learnable parameter through continuous relaxation, and is introduced into the objective function for joint optimization with the network weight, wherein the objective function includes a loss term and a sparsity regularization term based on the continuous relaxation mask, and the joint gradient is calculated on the GPU using the automatic differentiation technique;

[0015] The network weight and the continuous relaxation mask are optimized using a gradient-guided differential mutation strategy; the strategy first assigns different differential mutation directions according to the individual fitness, and then re-trains the mutated individuals to accelerate convergence by introducing gradient information;

[0016] Discretizing the optimized continuous mask into a Boolean type tensor as the final sparse connection mask and storing it in the global memory of the GPU, then rolling back the network weight to the historical iteration state, and using the parallel computing capability of the GPU to fine-tune the remaining non-zero weights using the gradient descent algorithm, finally generating a lightweight sparse neural network model that can run efficiently on actual hardware.

[0017] Further, the image data includes any one or several of the following: handwritten digit image, natural scene image, and street view character image.

[0018] The local edge feature corresponds to the contour information of the object in the image, which is expressed as the edge of the number stroke in the handwritten digit image, the object boundary in the natural scene image, and the character contour in the street view character image.

[0019] The texture feature corresponds to the pattern information of the object surface in the image, which is expressed as the texture change of the object surface in the natural scene image, and the stroke thickness and interval of the character in the street view character image.

[0020] The global semantic feature corresponds to the category information of the object in the image, and the local features are integrated to recognize the overall content through high-level semantics.

[0021] Further, the objective function is expressed as:

[0022]

[0023] wherein, represents the loss of the network , is used to control the balance between the loss and the number of parameters, is a Sigmoid function, is a learnable parameter, and the temperature parameter controls the convergence degree of the mask variable to binary 0 or 1.

[0024] Further, the initial population contains a population of individuals, wherein each individual is defined as:

[0025]

[0026] wherein, the mask parameter controls the sparse structure, and the shared model weight.

[0027] Further, the gradient-guided differential mutation strategy includes the following sub-steps:

[0028] Calculate the normalized fitness value of each individual in the current population;

[0029] Select the individual with the optimal fitness value in the current population;

[0030] According to the relationship between the normalized fitness value and the preset threshold, different differential mutation strategies are assigned to the individuals;

[0031] For the intermediate individual generated after differential mutation, gradient information is used to perform one or more gradient descent updates.

[0032] Further, the different differential mutation strategies are assigned as follows: if the normalized fitness value of an individual is lower than a preset threshold, a random differential mutation strategy is adopted:

[0033]

[0034] wherein, is a mutation factor, is a new individual generated after the current mutation operation, are two randomly selected individuals in the population, and ;

[0035] If the normalized fitness value of an individual is higher than or equal to the preset threshold, a guided exploration strategy towards the optimal individual is adopted:

[0036]

[0037] wherein, is the point with the highest fitness value in the population.

[0038] Further, the training process of the sparse subnetwork includes:

[0039] input data set , population size , mask initial , temperature coefficient , penalty coefficient , total training rounds , number of iterations per round , backtracking generation ;

[0040] generate an initial population , wherein , ;

[0041] for to rounds, perform:

[0042] for to iterations, perform:

[0043] sample a small batch of training samples from ;

[0044] perform a gradient-guided differential mutation strategy for each individual ;

[0045] update the parameters , , update temperature parameter ;

[0046] perform soft fallback update mask: reset to , reset to 1;

[0047] determine mask structure , fallback weight ;

[0048] fine-tune sparse subnetwork using gradient descent algorithm;

[0049] return final model: .

[0050] A second aspect of the present application provides a lightweight sparse neural network-based image data processing system, comprising:

[0051] a data acquisition module for acquiring image data to be processed;

[0052] a data preprocessing module for sequentially performing random flipping, random cropping data enhancement and tensor standardization preprocessing on the image data to obtain standard tensor data meeting the model input requirements;

[0053] a model inference module for inputting the standard tensor data to a lightweight sparse neural network model; the model extracts physical features from the image step by step through multiple sparse convolution layers, including local edge and texture features extracted by the bottom convolution layer, and global semantic features formed by the integration of high-level convolution layers; the local edge and texture features are used to represent the basic structural information in the image, and the global semantic features are used to represent the category attributes of the image; the extracted physical features are mapped to category confidence through a fully connected layer, and finally the classification result is output through a Softmax function.

[0054] A third aspect of the present application provides an electronic device comprising a memory, a processor, and a program stored on the memory and running on the processor, wherein the processor implements the steps of the lightweight sparse neural network-based image data processing method according to the first aspect of the present application when executing the program.

[0055] A fourth aspect of the present application provides a computer-readable storage medium having a program stored thereon, wherein the program is executed by a processor to implement the steps of the lightweight sparse neural network-based image data processing method according to the first aspect of the present application.

[0056] The fifth aspect of the present application provides a computer program product comprising software code, wherein the programs in the software code perform the steps in the image data processing method based on a lightweight sparse neural network according to the first aspect of the present application.

[0057] Compared with the prior art, the image data processing method and system based on a lightweight sparse neural network provided by the present application has the following beneficial effects:

[0058] (1) In view of the technical problems of high computational overhead, high memory occupation and low search efficiency in model deployment in a resource-limited hardware environment, the present application performs sparse sub-network search and optimization in parallel on a computing chip through a GPU, randomly initializes a dense neural network with shared weights and multiple continuous masks to generate an initial population composed of multiple sparse sub-networks, and adopts a gradient-guided differential mutation strategy for parallel optimization. This method can obtain multiple deployable lightweight network structures in one training process, significantly reducing the computational load and storage demand of the model in mobile terminals, embedded devices and edge computing scenarios, and effectively improving the inference speed, energy efficiency ratio and real-time response performance of the device side.

[0059] (2) In view of the problems of easy falling into local optimum and slow convergence speed in the training process of a sparse neural network, the present application proposes a gradient-guided differential mutation optimization method. On a GPU platform, the discrete binary mask is continuously relaxed into a learnable parameter and optimized in conjunction with the network weight; at the same time, combined with the fitness-guided differential mutation strategy, gradient information is introduced to retrain the mutated individuals, taking into account global exploration and local convergence, thereby realizing parallel training and automatic structure adjustment of multiple networks. This method can quickly generate sparse structures with high computational efficiency and hardware friendliness in the search process, without the need for additional complex search processes to adapt to different hardware platforms (such as GPU, edge AI chip), significantly improving the deployment flexibility and resource utilization of the system.

[0060] (3) In view of the technical problem that the performance of the model significantly decreases under high compression ratio conditions in resource-limited environments such as mobile terminals and embedded systems, the present application discretizes the continuous mask obtained after optimization into binary form and fine-tunes the weights. By discretizing the continuous mask into binary form to generate the final sparse sub-network and fine-tuning the network weights to the historical iteration state, the model performance is effectively maintained while maintaining a high sparsity (more than 80%), and the test accuracy on the CIFAR-10 and SVHN datasets reaches 90.70% and 95.76%, respectively, generating a lightweight neural network model suitable for hardware deployment, achieving the best balance between model compression and performance in resource-limited environments such as mobile terminals and embedded systems. BRIEF DESCRIPTION OF DRAWINGS

[0061] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0062] Figure 1 The flowchart illustrates the steps of an image data processing method based on a lightweight sparse neural network, as provided in Embodiment 1 of the present invention.

[0063] Figure 2 This is a flowchart illustrating the training process of a lightweight sparse neural network model provided in Embodiment 1 of the present invention.

[0064] Figure 3 The curve showing the change in accuracy of the method during the search process on the MNIST dataset LeNet-5 provided in Embodiment 1 of the present invention is shown.

[0065] Figure 4 The curve showing the change in test accuracy of the LeNet-5 sparse subnetwork on the MNIST dataset during the fine-tuning stage is provided in Embodiment 1 of the present invention.

[0066] Figure 5 The curve showing the change in accuracy of the method during the search process on the CIFAR10 dataset ResNet18 provided in Embodiment 1 of the present invention.

[0067] Figure 6 The curve showing the change in test accuracy of the ResNet18 sparse subnetwork on the CIFAR10 dataset during the fine-tuning stage, as provided in Embodiment 1 of the present invention.

[0068] Figure 7 The curve showing the change in accuracy of the method during the search process on the CIFAR100 dataset ResNet18 provided in Embodiment 1 of the present invention is shown.

[0069] Figure 8 The curve showing the change in test accuracy of the ResNet18 sparse subnetwork on the CIFAR100 dataset during the fine-tuning stage, as provided in Embodiment 1 of the present invention.

[0070] Figure 9 The curve showing the change in accuracy of the method during the search process on the SVHN dataset ResNet18 provided in Embodiment 1 of the present invention is shown.

[0071] Figure 10 This is a comparison chart of the test accuracy of sparse subnetworks of various methods on the ResNet18 dataset of the SVHN dataset, provided in Embodiment 1 of the present invention.

[0072] Figure 11 The curves showing the change in individual population test accuracy during the search process of this method on VGG16 provided in Embodiment 1 of the present invention.

[0073] Figure 12 The curve showing the change in test accuracy of the sparse subnetwork on VGG16 during the fine-tuning stage, provided in Embodiment 1 of the present invention.

[0074] Figure 13 This is the verification accuracy of the optimal subnetwork of ResNet18 under different sparsities provided in Embodiment 1 of the present invention.

[0075] Figure 14 The impact of different backoff periods on the testing accuracy of sparse subnetworks is provided in Embodiment 1 of the present invention.

[0076] Figure 15 This is a schematic diagram of an image data processing system based on a lightweight sparse neural network provided in Embodiment 2 of the present invention. Detailed Implementation

[0077] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0078] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0079] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0080] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0081] Before describing the embodiments of the present invention in detail, the technical terms involved in the present invention are explained and defined as follows:

[0082] 1. Sparse subnetwork: refers to a network whose total number of parameters is significantly reduced after applying a mask to the shared weights.

[0083] 2. Population: In evolutionary algorithms, this term refers to a set of candidate solutions to be optimized. Therefore, the initial population here specifically refers to the set of sparse subnetworks with different structures generated at the beginning of the algorithm using a set of randomly generated continuous masks. These subnetworks will serve as the starting point for the differential evolutionary algorithm optimization.

[0084] Example 1

[0085] like Figure 1 This embodiment provides an image data processing method based on a lightweight sparse neural network, applicable to mobile terminals, embedded devices, or edge computing devices, including:

[0086] Acquire the image data to be processed;

[0087] The image data is sequentially subjected to random flipping, random cropping, data augmentation, and tensor normalization preprocessing to obtain normalized tensor data that meets the model input requirements;

[0088] The canonical tensor data is input into a lightweight sparse neural network model; the model extracts physical features from the image step by step through multiple sparse convolutional layers, including local edge and texture features extracted by the lower convolutional layers, and global semantic features integrated by the higher convolutional layers; the local edge and texture features are used to characterize the basic structural information in the image, and the global semantic features are used to characterize the category attributes of the image;

[0089] The extracted physical features are mapped to class confidence scores through a fully connected layer, and the classification results are finally output through the Softmax function.

[0090] like Figure 2 The lightweight sparse neural network model is obtained through the following training process:

[0091] A dense neural network with shared weights and multiple consecutive masks are randomly initialized in the GPU's global memory. By starting the CUDA kernel, scheduling the GPU's streaming multiprocessors and CUDA cores, element-wise multiplication is performed on the weight tensor and each mask tensor to generate multiple sparse sub-networks to form the initial population. The discrete binary masks are continuously relaxed to become learnable parameters and introduced into the objective function for joint optimization with the network weights. The objective function includes a loss term and a sparsity regularization term based on the continuously relaxed masks. The joint gradient is calculated on the GPU using automatic differentiation techniques.

[0092] The network weights and the continuous relaxation mask are optimized using a gradient-guided differential mutation strategy. The strategy first assigns different differential mutation directions based on the individual fitness, and then introduces gradient information to retrain the mutated individuals to accelerate convergence.

[0093] The optimized continuous mask is discretized into a Boolean tensor and stored in the GPU's global memory as the final sparse connection mask. Then, the network weights are rolled back to the historical iteration state, and the remaining non-zero weights are fine-tuned using the gradient descent algorithm with the parallel computing power of the GPU. Finally, a lightweight sparse neural network model that can run efficiently on actual hardware is generated.

[0094] A lightweight sparse neural network model randomly initializes a dense neural network with shared weights and multiple consecutive masks in the GPU's global memory. By launching the CUDA kernel and scheduling the GPU's streaming multiprocessors and CUDA cores, element-wise multiplication is performed on the weight tensor and each mask tensor to generate multiple sparse subnetworks to form the initial population. A gradient-guided differential mutation strategy is used to optimize the network weights and consecutively relaxed masks. This technical solution solves the problems of sparse neural networks easily getting trapped in local optima and slow convergence during training. It achieves the technical effect of rapidly generating sparse structures with high computational efficiency and hardware friendliness on the GPU platform, significantly improving system deployment flexibility and resource utilization.

[0095] Specifically, the image data includes any one or more of the following: handwritten digit images, natural scene images, and street view character images;

[0096] The local edge features correspond to the contour information of objects in the image. In handwritten digit images, they are represented as the edges of digit strokes; in natural scene images, they are represented as object boundaries; and in street scene character images, they are represented as character outlines.

[0097] The texture features correspond to the pattern information of the object surface in the image. In natural scene images, they are manifested as texture changes on the object surface, and in street scene character images, they are manifested as the thickness and spacing of the character strokes.

[0098] The global semantic features correspond to the category information of objects in the image, and local features are integrated through high-level semantic representation to identify the overall content.

[0099] The lightweight sparse neural network model provided by this invention can be applied to the processing of handwritten digit images, natural scene images, and street scene character images. By connecting physical features with specific scenes, the practicality and interpretability of the lightweight sparse neural network model are enhanced.

[0100] Specifically, the objective function is expressed as:

[0101]

[0102] in, Represents network The loss, Used to control the balance between loss and number of parameters For the Sigmoid function, Temperature parameter is a learnable parameter. Controls the degree of convergence of the mask variable to binary 0 or 1.

[0103] This invention controls the convergence of the mask variable by using learnable parameters and temperature parameters in the objective function, balancing model loss and the number of parameters, effectively maintaining model performance while preserving high sparsity, and achieving the best balance between model compression and performance.

[0104] Specifically, the initial population includes Population of individuals Each individual Defined as:

[0105]

[0106] Among them, mask parameters Controlling sparse structures These are shared model weights.

[0107] By controlling the sparse structure through mask parameters and sharing model weights, a population of individuals with different sparse connection structures is constructed. This solves the problem that a single network structure cannot adapt to diverse deployment requirements, enabling the acquisition of multiple deployable lightweight network structures during a single training process, thus improving the applicability of the method.

[0108] Specifically, the gradient-guided differential mutation strategy includes the following sub-steps:

[0109] Calculate the normalized fitness value for each individual in the current population;

[0110] Select the individual with the best fitness value in the current population;

[0111] Based on the relationship between the normalized fitness value and the preset threshold, different differential mutation strategies are assigned to individuals;

[0112] For the intermediate individuals generated after differential mutation, use their gradient information to perform one or more gradient descent updates.

[0113] Specifically, the different differential mutation strategies are assigned as follows: if an individual's normalized fitness value is lower than a preset threshold, a random differential mutation strategy is adopted.

[0114]

[0115] in, As a variable factor, This refers to the new individual generated after the current mutation operation. Two individuals are randomly selected from the population, and ;

[0116] If an individual's normalized fitness value is higher than or equal to the preset threshold, a guided exploration strategy toward the optimal individual is adopted:

[0117]

[0118] in, This is the point with the highest fitness value in the population.

[0119] A gradient-guided differential mutation strategy is adopted, assigning different differential mutation directions based on individual fitness, and incorporating gradient information to retrain mutated individuals. This solves the problems of slow convergence and high computational cost in evolutionary algorithms, achieving a technical effect of balancing global exploration and local convergence, and accelerating the sparse structure search process.

[0120] Specifically, the training process of the sparse subnetwork includes:

[0121] Input dataset Population size mask initial Temperature coefficient Penalty coefficient Total number of training rounds Number of iterations per round Backward Algebra ;

[0122] Generate the initial population ,in , ;

[0123] for arrive Wheel, execute:

[0124] for arrive In the next iteration, execute:

[0125] from Medium-sampled mini-batch training samples ;

[0126] For each individual Execute a gradient-guided differential mutation strategy;

[0127] Update parameters , Update temperature parameters ;

[0128] Perform a soft rollback update mask: Reset to , Reset to 1;

[0129] Determine the mask structure rollback weight ;

[0130] Gradient descent algorithm is used for sparse subnetworks Make fine adjustments;

[0131] Return to the final model: .

[0132] In one specific embodiment, the present invention was carried out on a server equipped with a 16 vCPU Intel Xeon Platinum 8352V processor and an RTX 4090 (24GB) GPU (24GB VRAM), using the Ubuntu 20.04 operating system and the PyTorch 2.3.0 deep learning framework, along with the CUDA 12.1 acceleration library.

[0133] This invention evaluates the advantages of the GDE-CS algorithm in sparse subnetwork search capabilities on GPUs by comparing its performance with mainstream network sparsification methods such as the IMP algorithm and continuous sparsification CS algorithm on different network architectures, and by comparing the corresponding dense networks as performance references. This invention selects three typical neural network architectures: LeNet-5, ResNet-18, and VGG16, and conducts experiments on their respective datasets.

[0134] The effectiveness of the method (GDE-CS algorithm) provided in this invention was tested on the MNIST dataset and the classic LeNet-5 model. The LeNet-5 model is relatively small (approximately 60,000 parameters) and adopts a 7-layer structure (including 2 convolutional layers, 2 pooling layers, and 3 fully connected layers). This model systematically introduces key concepts such as local receptive field, weight sharing, and downsampling.

[0135] The MNIST dataset contains 70,000 28×28 pixel grayscale images of handwritten digits (0–9), each image accompanied by a corresponding digit label. This dataset is primarily used for handwritten digit recognition in document processing such as bank and express delivery slips, and tax invoices, to automatically recognize handwritten digit input and improve the automation of information entry. It is also frequently used to validate the real-time recognition capabilities of models on low-power devices (such as Jetson Nano, Raspberry Pi, and FPGA modules).

[0136] This experiment sets the population size to 5 individuals (i.e., ... =5), and the GDE-CS algorithm was used to perform a sparsity search on the original dense network, resulting in 5 different sparse subnetworks. Their performance on the test and validation sets and their corresponding sparsity are shown in Table 1.

[0137] Table 1. Performance evaluation results of the sparse population in the LeNet-5 model

[0138]

[0139] As can be observed from the table, all five sub-networks achieved classification accuracy comparable to the original dense model while maintaining high sparsity. Among them, the fifth sub-network, with a sparsity of 76.43%, still achieved a test accuracy of 99.10% and a validation accuracy of 99.21%.

[0140] For performance changes of the GDE-CS algorithm during training on the MNIST dataset and the LeNet-5 model, please refer to [link / reference]. Figure 3 and Figure 4 ,exist Figure 3 The diagram shows the test accuracy curves during the first round of training. It can be seen that all five individuals exhibited a good and rapid convergence trend in the first round of training, indicating that the gradient-guided differential mutation strategy can accelerate the convergence speed of sparse structures and improve the stability and accuracy of the initial training. Figure 4 The results show the change in test accuracy after fine-tuning each individual sparse subnetwork after pruning, indicating that the searched sparse structure has good potential for performance recovery. It can be seen that the GDE-CS algorithm exhibits stable test accuracy and strong generalization ability in both the sparse structure search and fine-tuning stages of the small network architecture LeNet-5.

[0141] To further verify the sparsity effect of GDE-CS, this experiment compared its optimal subnetwork with other current sparsity methods, including the original dense network (Dense), the IMP algorithm, and the CS algorithm. The comparison results are shown in Table 2.

[0142] Table 2. Performance comparison of different sparse algorithms on the LeNet-5 model

[0143]

[0144] As can be observed from the table, GDE-CS achieved the best test loss and test accuracy on the LeNet-5 model and the MNIST dataset, found the lottery network, and slightly improved accuracy compared to mainstream sparse algorithms such as IMP and CS, while achieving a good balance between accuracy and sparsity.

[0145] The results in summary show that the GDE-CS algorithm can effectively maintain or even improve model performance while maintaining high model sparsity, demonstrating superior sparse structure search capabilities.

[0146] The effectiveness of the published method was tested on the CIFAR10 and CIFAR100 datasets and the SVHN dataset on the ResNet-18 model. The ResNet model, through the design of residual blocks, allows information to be passed across layers in the network, formally achieving identity mapping through "shortcut connections," thereby promoting more stable gradient propagation. This allows the network to scale to hundreds or even thousands of layers while maintaining good training performance. A typical ResNet structure consists of multiple stacked residual blocks, with the most common variants including ResNet-18, ResNet-34, and deeper models such as ResNet-50, ResNet-101, and ResNet-152.

[0147] The CIFAR-10 dataset contains diverse images with complex backgrounds, and is often used to verify the performance of convolutional neural networks in recognizing multiple natural scenes, as well as to verify the robustness of the model in recognizing targets such as vehicles, pedestrians, and transportation vehicles. It can also be used to achieve rapid deployment and efficient inference performance evaluation of embedded visual recognition systems on edge computing devices.

[0148] CIFAR-100 is divided into 100 fine-grained categories, with more subtle differences between image categories. It is suitable for evaluating the feature representation ability, transfer learning ability, and multi-task learning effect of deep networks in complex scenes, such as distinguishing and labeling multiple targets in intelligent monitoring, drone vision and other systems; and verifying the recognition ability of the model in "coarse to fine" classification problems.

[0149] Each image in the SVHN (Street View House Numbers) dataset contains one or more street view house numbers (0–9) with precise label information. Compared to MNIST, SVHN images are closer to the real world, containing complex backgrounds, different lighting conditions, and font variations, thus making the task more challenging. The task involves recognizing numerical information such as house numbers, license plates, and road signs in street view or vehicle-mounted cameras; and validating the recognition performance of a lightweight character recognition network in complex natural scenes using mobile terminal OCR (Optical Character Recognition).

[0150] This experiment focuses on searching for sparse subnetworks on the ResNet-18 convolutional neural network architecture. Table 3 shows the performance of multiple sparse subnetworks obtained by the GDE-CS algorithm on the test and validation sets on three datasets.

[0151] Table 3. Performance evaluation results of the ResNet18 model on different datasets with sparse populations.

[0152]

[0153] As shown in the table, the GDE-CS algorithm successfully searched for subnetworks with sparsity ranging from 80% to 87% and exhibited excellent performance on all three datasets, demonstrating strong robustness. On CIFAR-100, despite the increased task complexity, the accuracy of the sparse subnetworks remained comparable to that of dense networks, indicating that the GDE-CS algorithm can effectively search for structurally valid sparse subnetworks even when facing multi-class classification tasks.

[0154] For the training performance of the GDE-CS algorithm on the ResNet-18 model on the CIFAR-10 and CIFAR-100 datasets, please refer to [link / reference]. Figure 5 , Figure 6 , Figure 7 and Figure 8 From 5 and Figure 7 As can be seen, in the first round of training, all five individuals achieved fast and stable convergence on both datasets, verifying the effectiveness of the proposed gradient-guided differential mutation strategy in accelerating structural convergence and improving the stability of initial training. Figure 6 and Figure 8 This reflects the changes in test accuracy of each sparse sub-network during the fine-tuning phase. It can be observed that the pruned models can recover and maintain high performance levels after further training, indicating that the sparse structure generated by the GDE-CS algorithm has good recoverability and adaptability. Overall, the algorithm demonstrates stable optimization results and strong generalization ability on different datasets under the ResNet architecture.

[0155] To verify the adaptability and robustness of the GDE-CS algorithm in different task scenarios, this experiment will further explore its sparse subnetwork search performance on ResNet18 and compare it with IMP, CS and the original dense model. Table 4 shows the model performance comparison of each algorithm while maintaining high sparsity.

[0156] Table 4. Performance comparison of ResNet18 model on different datasets using different sparse algorithms

[0157]

[0158] The results show that GDE-CS achieved good performance on the CIFAR-10 and SVHN datasets with the lowest test losses of 0.3159 and 0.1674, respectively. While maintaining sparsity above 80%, it achieved test accuracies of 90.70% and 95.76%, respectively, slightly better than or on par with dense networks, demonstrating excellent sparse structure search capabilities and strong robustness. On the CIFAR-100 dataset, GDE-CS's test accuracy was slightly lower than other methods, but it still showed a good balance between loss and sparsity, indicating that it still possesses certain compression effects and optimization potential in complex multi-class tasks. (See also...) Figure 9 The GDE-CS algorithm maintains high test accuracy on the SVHN dataset, and in the early stages of training, GDE-CS outperforms other algorithms in test accuracy, demonstrating strong generalization ability and fast convergence speed.

[0159] The effectiveness of the published method was tested on the VGG16 model. VGG employs a multi-layer small convolutional kernel design strategy. This design increases the network depth while maintaining the same receptive field, thereby improving the model's expressive power and further optimizing the performance of the neural network. There are several versions of VGG, the most commonly used being VGG16, which indicates 16 convolutional layers. Other models include VGG11, VGG13, and VGG19. The CIFAR-10 dataset was selected as the evaluation benchmark, and the loss, accuracy, and corresponding sparsity of the sparse population searched by GDE-CS on the test and validation sets were recorded. The results are shown in Table 5.

[0160] Table 5. Performance evaluation results of sparse populations in the VGG16 model

[0161]

[0162] As can be observed from the table, although the sparsity of the obtained sparse subnetworks all exceeded 94%, demonstrating significant compression effects, the test accuracy was slightly lower than that of the dense model, remaining generally between 88% and 89%. This indicates that the model's expressive power is somewhat limited under extremely high sparsity. For performance curves of the GDE-CS algorithm on VGG16 during the search process and sparse subnetwork fine-tuning stage, please refer to [reference needed]. Figure 11 and Figure 12 As can be seen, the performance differences among the sub-networks are small, and the diversity and differences of GDE-CS in population search are not reflected.

[0163] Next, this experiment further verifies the sparse subnetwork search performance of the GDE-CS algorithm on VGG16, and compares and analyzes it with IMP, CS and the original dense model. Table 6 shows the model performance comparison of each algorithm while maintaining high sparsity.

[0164] Table 6. Performance comparison of different sparse algorithms under the VGG16 model

[0165]

[0166] As can be observed from the table, the GDE-CS algorithm achieves the highest sparsity (94.04%) while having the smallest number of parameters, only 2.88M, thus compressing the model size and demonstrating good compression performance. However, its test accuracy is slightly lower than that of the CS and Dense models, indicating a slight performance loss under extremely high sparsity.

[0167] To further investigate the impact of sparsity on model performance, this experiment sets the total number of training epochs to 5 on the ResNet-18 architecture and the CIFAR-10 dataset, and adjusts the initial value of the mask variable. This indirectly controls the final sparsity of the subnetwork during the search process. Subnetworks with different sparsities were ultimately obtained, and their performance results are shown in Table 7.

[0168] Table 7. Sparsity and performance variations of ResNet-18 under different initial mask values

[0169]

[0170] As can be clearly observed from the table, as the initial mask value gradually decreases from 0, the model sparsity gradually increases, from 80.12% to 99.08%. However, at the same time, the model's accuracy on both the test and validation sets shows a significant downward trend. (See also...) Figure 13 When the sparsity is 80%, sparse subnetworks can achieve the same accuracy as dense networks.

[0171] This demonstrates that while excessive sparsity further compresses the model size, it also weakens the model's expressive power and generalization performance. Therefore, in practical applications, a trade-off must be struck between compression ratio and performance to avoid blindly pursuing extremely high sparsity and causing performance collapse. This experiment fully demonstrates the stability of the GDE-CS algorithm under different sparsity control conditions, and verifies the crucial impact of setting a reasonable initial mask value on the final sub-network performance. The flexibility of the sparsity control mechanism also reflects the application potential of GDE-CS in real-world deployment scenarios.

[0172] Finally, the impact of the rewind epoch on the training performance of the sparse subnetwork was investigated. Experiments were conducted on the ResNet-18 model using the GDE-CS algorithm at two sparsity levels: approximately 85% and approximately 93%. The effect of different rewind epoch values ​​on the final model performance was examined. The experimental results are listed in Table 8.

[0173] Table 8. Impact of backoff period on the performance of ResNet-18 high-sparseness subnetworks

[0174]

[0175] The results from the two sets of experiments show that as the backoff period increases, the accuracy of the model on the validation set generally increases and the loss gradually decreases. This indicates that a longer backoff training phase can better preserve the important weight information from the early training, provide a more stable initialization for the sparse subnetwork, and help it recover and improve performance.

[0176] When the sparsity is further increased to approximately 93%, the performance of the subnetwork becomes more sensitive to changes in the backoff period. With a short backoff period (e.g., rewind=2), the model accuracy is significantly lower; while performance improves when rewind is increased to 24 or 30.

[0177] The impact of different backoff periods on the performance of sparse subnetworks is discussed in the following section. Figure 14 While the overall trend is that performance improves with increasing backoff period, slight fluctuations were observed in some experiments, indicating that an excessively long backoff period may lead to initialization that is mismatched with the current training phase, affecting the final convergence result. Therefore, appropriately selecting the backoff period value is particularly crucial for high-sparse models.

[0178] Example 2

[0179] like Figure 15 This embodiment provides an image data processing system based on a lightweight sparse neural network, including:

[0180] The data acquisition module is used to acquire the image data to be processed;

[0181] The data preprocessing module is used to perform random flipping, random cropping, data augmentation, and tensor normalization preprocessing on the image data in sequence to obtain normalized tensor data that meets the model input requirements.

[0182] The model inference module is used to input the canonical tensor data into a lightweight sparse neural network model. This model extracts physical features from the image step by step through multiple sparse convolutional layers, including local edge and texture features extracted by the lower convolutional layers, and global semantic features integrated by the higher convolutional layers. The local edge and texture features are used to represent the basic structural information in the image, and the global semantic features are used to represent the category attributes of the image. The extracted physical features are mapped to category confidence scores through a fully connected layer, and finally the classification result is output through the Softmax function.

[0183] Example 3

[0184] Embodiment 3 of the present invention provides an electronic device.

[0185] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, it implements the steps in the image data processing method based on a lightweight sparse neural network as described in Embodiment 1 of the present invention.

[0186] The detailed steps are the same as those of the image data processing method based on lightweight sparse neural networks provided in Example 1, and will not be repeated here.

[0187] Example 4

[0188] Embodiment 4 of the present invention provides a computer-readable storage medium.

[0189] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the image data processing method based on a lightweight sparse neural network as described in Embodiment 1 of the present invention.

[0190] The detailed steps are the same as those of the image data processing method based on lightweight sparse neural networks provided in Example 1, and will not be repeated here.

[0191] Example 5

[0192] Embodiment 5 of the present invention provides a computer program product.

[0193] A computer program product includes software code, wherein the program in the software code performs the steps of the image data processing method based on a lightweight sparse neural network as described in Embodiment 1 of the present invention.

[0194] The detailed steps are the same as those of the image data processing method based on lightweight sparse neural networks provided in Example 1, and will not be repeated here.

[0195] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0196] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0197] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0198] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0199] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. An image data processing method based on a lightweight sparse neural network, characterized in that, Applied to mobile terminals, embedded devices, or edge computing devices, including: Acquire the image data to be processed; The image data is sequentially subjected to random flipping, random cropping, data augmentation, and tensor normalization preprocessing to obtain normalized tensor data that meets the model input requirements; The canonical tensor data is input into a lightweight sparse neural network model; the model extracts physical features from the image step by step through multiple sparse convolutional layers, including local edge and texture features extracted by the lower convolutional layers, and global semantic features integrated by the higher convolutional layers; the local edge and texture features are used to characterize the basic structural information in the image, and the global semantic features are used to characterize the category attributes of the image; The extracted physical features are mapped to class confidence scores through a fully connected layer, and the classification results are finally output through the Softmax function. The lightweight sparse neural network model is obtained through the following training process: A dense neural network with shared weights and multiple consecutive masks are randomly initialized in the GPU's global memory. By starting the CUDA kernel, the GPU's streaming multiprocessors and CUDA cores are scheduled to perform element-wise multiplication on the shared weights and each mask tensor to generate multiple sparse subnetworks to form the initial population. The discrete binary masks are continuously relaxed to become learnable parameters and introduced into the objective function for joint optimization with the network weights. The objective function includes a loss term and a sparsity regularization term based on the continuously relaxed masks. The joint gradient is calculated on the GPU using automatic differentiation techniques. The network weights and the continuous relaxation mask are optimized using a gradient-guided differential mutation strategy. The strategy first assigns different differential mutation directions based on the individual fitness, and then introduces gradient information to retrain the mutated individuals to accelerate convergence. The optimized continuous mask is discretized into a Boolean tensor and stored in the global memory of the GPU as the final sparse connection mask. Then, the network weights are rolled back to the historical iteration state, and the remaining non-zero weights are fine-tuned using the gradient descent algorithm with the parallel computing power of the GPU. Finally, a lightweight sparse neural network model that can run efficiently on actual hardware is generated. The gradient-guided differential mutation strategy includes the following sub-steps: Calculate the normalized fitness value for each individual in the current population; Select the individual with the best fitness value in the current population; Based on the relationship between the normalized fitness value and the preset threshold, different differential mutation strategies are assigned to individuals; For intermediate individuals generated after differential mutation, use their gradient information to perform one or more gradient descent updates. Specifically, the different differential mutation strategies are assigned as follows: if an individual's normalized fitness value is lower than a preset threshold, a random differential mutation strategy is adopted. in, As a variable factor, This refers to the new individual generated after the current mutation operation. Two individuals are randomly selected from the population, and ; If an individual's normalized fitness value is higher than or equal to the preset threshold, a guided exploration strategy toward the optimal individual is adopted: in, This is the point with the highest fitness value in the population; The training process of the sparse subnetwork includes: Input dataset Population size mask initial Temperature coefficient Penalty coefficient Total number of training rounds Number of iterations per round Backward Algebra ; Generate the initial population ,in , ; for arrive Wheel, execute: for arrive In the next iteration, execute: from Medium-sampled mini-batch training samples ; For each individual Execute a gradient-guided differential mutation strategy; Update parameters , Update temperature parameters ; Perform a soft rollback update mask: Reset to , Reset to 1; Determine the mask structure rollback weight ; Gradient descent algorithm is used for sparse subnetworks Make fine adjustments; Return to the final model: .

2. The method as described in claim 1, characterized in that, The image data includes any one or more of the following: handwritten digit images, natural scene images, and street view character images; The local edge features correspond to the contour information of objects in the image. In handwritten digit images, they are represented as the edges of digit strokes; in natural scene images, they are represented as object boundaries; and in street scene character images, they are represented as character outlines. The texture features correspond to the pattern information of the object surface in the image. In natural scene images, they are manifested as texture changes on the object surface, and in street scene character images, they are manifested as the thickness and spacing of the character strokes. The global semantic features correspond to the category information of objects in the image, and local features are integrated through high-level semantic representation to identify the overall content.

3. The method as described in claim 1, characterized in that, The objective function is expressed as: in, Represents network The loss, Used to control the balance between loss and number of parameters For the Sigmoid function, Temperature parameter is a learnable parameter. Controls the degree of convergence of the mask variable to binary 0 or 1.

4. The method as described in claim 1, characterized in that, The initial population includes Population of individuals Each individual Defined as: Among them, mask parameters Controlling sparse structures These are shared model weights.

5. An image data processing system based on a lightweight sparse neural network, characterized in that, include: The data acquisition module is used to acquire the image data to be processed; The data preprocessing module is used to perform random flipping, random cropping, data augmentation, and tensor normalization preprocessing on the image data in sequence to obtain normalized tensor data that meets the model input requirements. The model inference module is used to input the canonical tensor data into a lightweight sparse neural network model. The model extracts physical features from the image step by step through multiple sparse convolutional layers, including local edge and texture features extracted by the lower convolutional layers, and global semantic features integrated by the higher convolutional layers. The local edge and texture features are used to characterize the basic structural information in the image, and the global semantic features are used to characterize the category attributes of the image. The extracted physical features are mapped to category confidence scores through a fully connected layer, and the classification results are finally output through the Softmax function. The lightweight sparse neural network model is obtained through the following training process: A dense neural network with shared weights and multiple consecutive masks are randomly initialized in the GPU's global memory. By starting the CUDA kernel, the GPU's streaming multiprocessors and CUDA cores are scheduled to perform element-wise multiplication on the shared weights and each mask tensor to generate multiple sparse subnetworks to form the initial population. The discrete binary masks are continuously relaxed to become learnable parameters and introduced into the objective function for joint optimization with the network weights. The objective function includes a loss term and a sparsity regularization term based on the continuously relaxed masks. The joint gradient is calculated on the GPU using automatic differentiation techniques. The network weights and the continuous relaxation mask are optimized using a gradient-guided differential mutation strategy. The strategy first assigns different differential mutation directions based on the individual fitness, and then introduces gradient information to retrain the mutated individuals to accelerate convergence. The optimized continuous mask is discretized into a Boolean tensor and stored in the global memory of the GPU as the final sparse connection mask. Then, the network weights are rolled back to the historical iteration state, and the remaining non-zero weights are fine-tuned using the gradient descent algorithm with the parallel computing power of the GPU. Finally, a lightweight sparse neural network model that can run efficiently on actual hardware is generated. The gradient-guided differential mutation strategy includes the following sub-steps: Calculate the normalized fitness value for each individual in the current population; Select the individual with the best fitness value in the current population; Based on the relationship between the normalized fitness value and the preset threshold, different differential mutation strategies are assigned to individuals; For intermediate individuals generated after differential mutation, use their gradient information to perform one or more gradient descent updates. Specifically, the different differential mutation strategies are assigned as follows: if an individual's normalized fitness value is lower than a preset threshold, a random differential mutation strategy is adopted. in, As a variable factor, This refers to the new individual generated after the current mutation operation. Two individuals are randomly selected from the population, and ; If an individual's normalized fitness value is higher than or equal to the preset threshold, a guided exploration strategy toward the optimal individual is adopted: in, This is the point with the highest fitness value in the population; The training process of the sparse subnetwork includes: Input dataset Population size mask initial Temperature coefficient Penalty coefficient Total number of training rounds Number of iterations per round Backward Algebra ; Generate the initial population ,in , ; for arrive Wheel, execute: for arrive In the next iteration, execute: from Medium-sampled mini-batch training samples ; For each individual Execute a gradient-guided differential mutation strategy; Update parameters , Update temperature parameters ; Perform a soft rollback update mask: Reset to , Reset to 1; Determine the mask structure rollback weight ; Gradient descent algorithm is used for sparse subnetworks Make fine adjustments; Return to the final model: .

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the image data processing method based on a lightweight sparse neural network as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the image data processing method based on a lightweight sparse neural network as described in any one of claims 1 to 4.

8. A computer program product, comprising software code, characterized in that, The program in the software code performs the steps of the image data processing method based on a lightweight sparse neural network as described in any one of claims 1 to 4.

Images