Image processing method, system, device and readable storage medium

Abstract

The present application discloses an image processing method, comprising: grouping the weight matrix of a preset long short-term memory network model according to inherent structured sparsity to obtain corresponding weight groups; calculating the Pearson correlation coefficient of each weight group with other weight groups respectively, using the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly selecting the corresponding weight group for compression according to the sampling probability according to a preset compression ratio to obtain a compressed long short-term memory network model; and performing image processing on an input image using the compressed long short-term memory network model. The present application can perform neuron pruning according to a user-specified compression ratio, and directly obtain new weights of the compressed network weight matrix by minimizing the reconstruction error output by the weight matrix without relying on training, thereby achieving compression of the recurrent neural network. The present application also provides an image processing system, device, and readable storage medium, which have the above-mentioned beneficial effects.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, system, device and readable storage medium. Background Art

[0002] The benefits of overparameterization have significantly improved the accuracy of neural networks, enabling the widespread application of deep learning in machine vision fields such as multi-object tracking and image segmentation. This has also led to a demand for deployment on various embedded devices and mobile platforms. However, these platforms have limited computing and storage resources, making them unable to store and run neural networks with large parameter counts. Neural network compression technology can effectively reduce the number of parameters and the computational effort during inference, addressing the deployment challenges of deep learning in resource-constrained environments. Pruning algorithms for convolutional neural networks (CNNs) have been widely adopted in industry, demonstrating diverse development trends, including structured and unstructured pruning algorithms, inference-time pruning, training-based pruning, and static and dynamic pruning. Unlike CNN pruning methods, pruning methods for recurrent neural networks (RNNs) have been understudied. Due to the differences in computational logic between RNNs and CNNs, most CNN pruning methods cannot be directly applied to RNNs, making the research of RNN pruning algorithms more challenging.

[0003] Therefore, how to achieve compression of recurrent neural networks is a technical problem that those skilled in the art currently need to solve. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, system, device and readable storage medium for realizing compression of recurrent neural networks.

[0005] To solve the above technical problems, the present application provides an image processing method, which includes:

[0006] Grouping the weight matrices of the preset long short-term memory network model according to inherent structured sparsity to obtain corresponding weight groups;

[0007] Calculate the Pearson correlation coefficient of each weight group with other weight groups respectively, use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly select the corresponding weight group according to the sampling probability according to the preset compression rate for compression, to obtain a compressed long short-term memory network model;

[0008] The compressed long short-term memory network model is used to perform image processing on the input image.

[0009] Optionally, before performing image processing on the input image using the compressed long short-term memory network model, the method further includes:

[0010] Calculating an output feature map of the weight matrix of the preset long short-term memory network model based on the weight group;

[0011] Calculating a compressed output feature map of the weight matrix of the compressed long short-term memory network model based on the weight group;

[0012] Determining the optimal weight matrix of the compressed long short-term memory network model based on the output feature map and the compressed output feature map by the least squares method;

[0013] The optimal weight matrix is ​​used to optimize the parameters of the compressed long short-term memory network model.

[0014] Optionally, calculating an output feature map of a weight matrix of the preset long short-term memory network model according to the weight group includes:

[0015] Calculate the input value of the activation function of the forget gate, input gate and output gate of the first layer of the preset long short-term memory network model at time step t and the output value at the previous time step t-1 And determine the corresponding output weight matrix and the input weight matrix

[0016] According to the formula and Calculating an output feature map of the weight matrix of the preset long short-term memory network model;

[0017] Among them, W hl is the output weight matrix of the first layer of the preset long short-term memory network model, are the output weights of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, W xl is the input weight matrix of the first layer of the preset long short-term memory network model, are the input weights of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, FM hl is the output feature map of the output weight matrix of the first layer of the preset long short-term memory network model, y fh 、y ih 、y uh 、y oh are the output feature maps of the output weight matrices of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, is the output value of the preset long short-term memory network model at the previous time step t-1, FM xl is the output feature map of the input weight matrix of the first layer of the preset long short-term memory network model, y fx 、y ix 、y ux 、y ox are the output feature maps of the input weight matrices of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, The input value of the lth layer of the preset long short-term memory network model at time step t.

[0018] Optionally, calculating a compressed output feature map of the weight matrix of the compressed long short-term memory network model according to the weight group includes:

[0019] Get the output value of the first layer of the preset long short-term memory network model at the previous time step t-1 And the output value As the input value of the compressed long short-term memory network model;

[0020] Will As the compressed output feature map of the output weight matrix of the compressed long short-term memory network model, A compressed output feature map as the input weight matrix of the compressed long short-term memory network model;

[0021] in, W hl The compressed output weight matrix, for The transposed matrix of W xl The compressed input weight matrix, for The transposed matrix of is the output value of the lth layer of the preset long short-term memory network model at the previous time step t-1, and T is the total number of time steps of the input data x.

[0022] Optionally, determining the optimal weight matrix of the compressed long short-term memory network model based on the output feature map and the compressed output feature map by a least squares method includes:

[0023] Record the numbers of the compressed weight groups in the preset long short-term memory network model as a set θ;

[0024] Extract the output feature map local data of the output weight matrix of the preset long short-term memory network model at the previous time step t-1 based on the set θ And the output feature map local data of the input weight matrix

[0025] According to the formula Determine the optimal output weight matrix of the lth layer of the compressed long short-term memory network model

[0026] According to the formula Determine the optimal input weight matrix of the lth layer of the compressed long short-term memory network model

[0027] in, is the F norm, is the local data of the output feature map of the output weight matrix, It is the local data of the output feature map of the input weight matrix.

[0028] Optionally, it also includes:

[0029] Extract the input weight matrix W of the l+1th layer of the preset long short-term memory network model based on the set θ x(l +1) The weight of the corresponding row

[0030] According to the formula Determine the optimal input weight matrix of the l+1 layer of the compressed long short-term memory network model

[0031] Among them, FM x(l+1) is the output feature map of the input weight matrix of the l+1th layer of the preset long short-term memory network model, W x(l+1) is the input weight matrix of the l+1th layer of the preset long short-term memory network model, W x(l+1) The compressed input weight matrix, for The transposed matrix of It is the input value of the l+1th layer of the preset long short-term memory network model at time step t.

[0032] Optionally, after optimizing the parameters of the compressed long short-term memory network model using the optimal weight matrix, the method further includes:

[0033] When all layers of the compressed long short-term memory network model have completed parameter optimization, the compressed long short-term memory network model is retrained a preset number of times.

[0034] To solve the above technical problems, the present application also provides an image processing system, which includes:

[0035] A grouping module, configured to group the weight matrices of the preset long short-term memory network model according to inherent structured sparsity to obtain corresponding weight groups;

[0036] A compression module is used to calculate the Pearson correlation coefficient of each weight group with other weight groups respectively, use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly select the corresponding weight group according to the sampling probability according to a preset compression rate for compression, so as to obtain a compressed long short-term memory network model;

[0037] The image processing module is used to perform image processing on the input image using the compressed long short-term memory network model.

[0038] To solve the above technical problems, the present application further provides an image processing device, which includes:

[0039] Memory for storing computer programs;

[0040] A processor is configured to implement the steps of any one of the above-mentioned image processing methods when executing the computer program.

[0041] In order to solve the above technical problems, the present application also provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image processing methods are implemented.

[0042] The image processing method provided in the present application includes: grouping the weight matrix of a preset long short-term memory network model according to inherent structured sparsity to obtain corresponding weight groups; calculating the Pearson correlation coefficient of each weight group with other weight groups respectively, using the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly selecting the corresponding weight group according to the sampling probability according to a preset compression rate for compression to obtain a compressed long short-term memory network model; and using the compressed long short-term memory network model to perform image processing on the input image.

[0043] The technical solution provided by this application randomly selects corresponding weight groups for compression based on a preset compression ratio using a sampling probability determined by the Pearson correlation coefficient. This allows the application to perform neuron pruning according to a user-specified compression ratio. This approach directly minimizes the reconstruction error output by the weight matrix, without relying on training, to obtain new weights for the compressed network weight matrix, thereby achieving compression of recurrent neural networks. This application also provides an image processing system, device, and readable storage medium, all of which have the aforementioned beneficial effects and will not be further elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0045] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0046] Figure 2 A flowchart of another image processing method provided in an embodiment of the present application;

[0047] Figure 3 A structural diagram of an image processing system provided in an embodiment of the present application;

[0048] Figure 4 This is a structural diagram of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The core of this application is to provide an image processing method, system, device and readable storage medium for realizing the compression of recurrent neural networks.

[0050] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0051] Existing RNN pruning algorithms include amplitude-based weight pruning or structured pruning based on least absolute shrinkage and selection operator (LASSO) regression. The latter uses LASSO regression regularization to achieve structured pruning of different granularities, such as block-structured weight pruning based on grouped LASSO regularization, weight pruning based on intrinsic structured sparsity (ISS), or neuron pruning based on LASSO regularization. Among them, ISS-based pruning constructs a specific weight grouping. Assuming that a certain layer of Long Short-Term Memory (LSTM) network model has K hidden states, when the kth hidden state is identified as a useless state and needs to be pruned, the kth cell state and the kth output gate that generate the kth hidden state can be deleted together. That is, the ISS grouping weights composed of the columns associated with the k-th unit state in the weight matrix of the current time step, the columns corresponding to the k-th hidden state in the current time step, and the rows corresponding to the k-th hidden state in the previous time step can all be removed. However, this method does not compress and prune the rows associated with the input in the weight matrix, and cannot prune the neurons of the LSTM. Neuron pruning based on LASSO regularization addresses this defect by introducing two gating variables for the input and hidden state and constructing corresponding LASSO regularization constraints. The values ​​of the gating variables are learned during training as masks for the compressed weight matrix, thereby achieving neuron-level pruning. However, structured pruning methods based on LASSO regression all belong to pruning during training, which rely on learning compression masks during the training process and cannot achieve flexible compression based on a given compression rate. Therefore, the present application provides an image processing method for solving the above problems.

[0052] Please refer to Figure 1 , Figure 1 A flowchart of an image processing method provided in an embodiment of the present application.

[0053] It specifically includes the following steps:

[0054] S101: Grouping the weight matrix of a preset long short-term memory network model according to inherent structured sparsity to obtain corresponding weight groups.

[0055] In this step, the purpose of grouping the weight matrix is ​​to randomly select the corresponding weight group for compression through sampling probability according to the preset compression rate, so as to achieve pruning during inference of the long short-term memory network model. Then, the new weights of the compressed network weight matrix can be obtained directly by minimizing the reconstruction error of the weight matrix output without relying on training, thereby realizing the compression of the recurrent neural network.

[0056] S102: Calculate the Pearson correlation coefficient of each weight group with other weight groups respectively, use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly select the corresponding weight group for compression according to the sampling probability based on the preset compression rate to obtain a compressed long short-term memory network model.

[0057] The Pearson Correlation Coefficient mentioned here is used to measure whether two data sets are on the same line, to measure the linear relationship between interval variables, and to reflect the degree of linear correlation between two variables X and Y. The Pearson Correlation Coefficient value ranges from -1 to 1. The larger the absolute value, the stronger the correlation.

[0058] Taking the target weight group as an example, the Pearson correlation coefficient of the target weight group and other weight groups is calculated respectively, and the Pearson correlation coefficient is used as the sampling probability of the target weight group being sampled. This application randomly selects the corresponding weight group for compression based on the sampling probability according to the preset compression rate to obtain a compressed long short-term memory network model. For example, the sampled weight group can be set to 0 to achieve compression of the long short-term memory network model.

[0059] S103: Perform image processing on the input image using the compressed long short-term memory network model.

[0060] Based on the above embodiment, in a specific embodiment, before the compressed long short-term memory network model is used to process the input image, the following steps may be performed: Figure 2 The steps shown below are used to optimize the parameters of the compressed long short-term memory network model. Figure 2 , Figure 2 A flowchart of another image processing method provided in an embodiment of the present application.

[0061] It specifically includes the following steps:

[0062] S201: Calculate an output feature map of a weight matrix of a preset long short-term memory network model based on the weight group.

[0063] In a specific embodiment, the output feature map of the weight matrix of the preset long short-term memory network model is calculated based on the weight group, which can be specifically implemented by performing the following steps:

[0064] Calculate the input value of the activation function of the forget gate, input gate and output gate of the lth layer of the preset long short-term memory network model at time step t and the output value at the previous time step t-1 And determine the corresponding output weight matrix and the input weight matrix

[0065] According to the formula and Calculate the output feature map of the weight matrix of the preset long short-term memory network model;

[0066] Among them, W hl is the output weight matrix of the first layer of the preset long short-term memory network model, are the output weights of the forget gate, input gate, update gate, and output gate of the preset long short-term memory network model, W xl is the input weight matrix of the first layer of the preset long short-term memory network model, They are the input weights of the forget gate, input gate, update gate, and output gate of the preset long short-term memory network model, FM hl is the output feature map of the output weight matrix of the lth layer of the preset long short-term memory network model, y fh 、y ih 、y uh 、y oh They are the output feature maps of the output weight matrices of the forget gate, input gate, update gate, and output gate of the preset long short-term memory network model, is the output value of the preset long short-term memory network model at the previous time step t-1, FM xl is the output feature map of the input weight matrix of the lth layer of the preset long short-term memory network model, y fx 、y ix 、y ux 、y ox They are the output feature maps of the input weight matrices of the forget gate, input gate, update gate, and output gate of the preset long short-term memory network model, is the input value of the lth layer of the preset long short-term memory network model at time step t.

[0067] S202: Calculate a compressed output feature map of the weight matrix of the compressed long short-term memory network model based on the weight group.

[0068] In a specific embodiment, the compressed output feature map of the weight matrix of the compressed long short-term memory network model is calculated based on the weight group, which can be specifically implemented by performing the following steps:

[0069] Get the output value of the lth layer of the preset long short-term memory network model at the previous time step t-1 And the output value As the input value of the compressed long short-term memory network model;

[0070] Will As the compressed output feature map of the output weight matrix of the compressed long short-term memory network model, The compressed output feature map of the input weight matrix of the compressed long short-term memory network model;

[0071] in, W hl The compressed output weight matrix, for The transposed matrix of W xl The compressed input weight matrix, for The transposed matrix of is the output value of the lth layer of the preset long short-term memory network model at the previous time step t-1, and T is the total number of time steps of the input data x.

[0072] S203: Determine the optimal weight matrix of the compressed long short-term memory network model based on the output feature map and the compressed output feature map through the least squares method.

[0073] In a specific embodiment, the optimal weight matrix of the compressed long short-term memory network model is determined based on the output feature map and the compressed output feature map by the least squares method, which can be specifically achieved by performing the following steps:

[0074] The numbers of the compressed weight groups in the preset long short-term memory network model are recorded as a set θ;

[0075] Extract the local data of the output feature map of the output weight matrix of the preset long short-term memory network model at the previous time step t-1 based on the set θ And the output feature map local data of the input weight matrix

[0076] According to the formula Determine the optimal output weight matrix of the lth layer of the compressed long short-term memory network model

[0077] According to the formula Determine the optimal input weight matrix of the lth layer of the compressed long short-term memory network model

[0078] in, is the F norm, is the local data of the output feature map of the output weight matrix, It is the local data of the output feature map of the input weight matrix.

[0079] Based on the above embodiment, the optimal input weight matrix of the l+1th layer can also be determined by performing the following steps:

[0080] Extract the input weight matrix W of the l+1 layer of the preset long short-term memory network model based on the set θ x(l+1) The weight of the corresponding row

[0081] According to the formula Determine the optimal input weight matrix for the l+1th layer of the compressed LSTM network model

[0082] Among them, FM x(l+1) is the output feature map of the input weight matrix of the l+1 layer of the preset long short-term memory network model, W x(l+1) is the input weight matrix of the l+1th layer of the preset long short-term memory network model, W x(l+1) The compressed input weight matrix, for The transposed matrix of is the input value of the l+1th layer of the preset long short-term memory network model at time step t.

[0083] S204: Optimizing parameters of the compressed long short-term memory network model using the optimal weight matrix.

[0084] In a specific embodiment, in order to further restore the network accuracy of the compressed LSTM network model, after optimizing the parameters of the compressed LSTM network model using the optimal weight matrix, the following steps may be performed:

[0085] When all layers of the compressed long short-term memory network model have completed parameter optimization, the compressed long short-term memory network model is retrained a preset number of times.

[0086] This application compresses the neurons in each hidden layer of a preset long short-term memory network model by compressing the output weights and input weights of the neurons in steps, wherein the random compression process is executed layer by layer.

[0087] Based on the above technical solution, the image processing method provided by the present application randomly selects the corresponding weight group for compression according to the sampling probability determined by the Pearson correlation coefficient based on the preset compression rate, so that the present application can perform neuron pruning according to the compression ratio specified by the user, and directly obtain the new weights of the compressed network weight matrix by minimizing the reconstruction error output by the weight matrix without relying on training, thereby realizing the compression of the recurrent neural network.

[0088] Please refer to Figure 3 , Figure 3 A structural diagram of an image processing system provided in an embodiment of the present application.

[0089] The system may include:

[0090] A grouping module 100 is used to group the weight matrices of the preset long short-term memory network model according to the inherent structured sparsity to obtain corresponding weight groups;

[0091] The compression module 200 is used to calculate the Pearson correlation coefficient between each weight group and other weight groups, use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly select the corresponding weight group according to the sampling probability according to the preset compression rate for compression, thereby obtaining a compressed long short-term memory network model;

[0092] The image processing module 300 is used to perform image processing on the input image using the compressed long short-term memory network model.

[0093] Based on the above embodiment, in a specific embodiment, the system may further include:

[0094] A first calculation module is used to calculate an output feature map of a weight matrix of a preset long short-term memory network model according to the weight group;

[0095] A second calculation module is used to calculate a compressed output feature map of the weight matrix of the compressed long short-term memory network model according to the weight group;

[0096] A determination module, configured to determine an optimal weight matrix of a compressed long short-term memory network model based on the output feature map and the compressed output feature map by using a least squares method;

[0097] The optimization module is used to optimize the parameters of the compressed long short-term memory network model using the optimal weight matrix.

[0098] Based on the above embodiment, in a specific embodiment, the first calculation module may include:

[0099] Calculate the input value of the activation function of the forget gate, input gate and output gate of the lth layer of the preset long short-term memory network model at time step t and the output value at the previous time step t-1 And determine the corresponding output weight matrix and the input weight matrix

[0100] According to the formula and Calculate the output feature map of the weight matrix of the preset long short-term memory network model;

[0101] Among them, W hl is the output weight matrix of the first layer of the preset long short-term memory network model, are the output weights of the forget gate, input gate, update gate, and output gate of the preset long short-term memory network model, W xl is the input weight matrix of the first layer of the preset long short-term memory network model, They are the input weights of the forget gate, input gate, update gate, and output gate of the preset long short-term memory network model, FM hl is the output feature map of the output weight matrix of the lth layer of the preset long short-term memory network model, y fh 、y ih 、y uh 、y oh They are the output feature maps of the output weight matrices of the forget gate, input gate, update gate, and output gate of the preset long short-term memory network model, is the output value of the preset long short-term memory network model at the previous time step t-1, FM xl is the output feature map of the input weight matrix of the lth layer of the preset long short-term memory network model, y fx 、y ix 、y ux 、y ox They are the output feature maps of the input weight matrices of the forget gate, input gate, update gate, and output gate of the preset long short-term memory network model, is the input value of the lth layer of the preset long short-term memory network model at time step t.

[0102] Based on the above embodiment, in a specific embodiment, the second calculation module may include:

[0103] Get submodule, used to get the output value of the lth layer of the preset long short-term memory network model at the previous time step t-1 And the output value As the input value of the compressed long short-term memory network model;

[0104] The first determining submodule is used to As the compressed output feature map of the output weight matrix of the compressed long short-term memory network model, The compressed output feature map of the input weight matrix of the compressed long short-term memory network model;

[0105] in, W hl The compressed output weight matrix, for The transposed matrix of Wxl The compressed input weight matrix, for The transposed matrix of is the output value of the lth layer of the preset long short-term memory network model at the previous time step t-1, and T is the total number of time steps of the input data x.

[0106] Based on the above embodiment, in a specific embodiment, the determination module may include:

[0107] A recording submodule, used to record the numbers of the compressed weight groups in the preset long short-term memory network model as a set θ;

[0108] The first extraction submodule is used to extract the output feature map local data of the output weight matrix of the preset long short-term memory network model at the previous time step t-1 based on the set θ And the output feature map local data of the input weight matrix

[0109] The second determination submodule is used to determine the Determine the optimal output weight matrix of the lth layer of the compressed long short-term memory network model

[0110] The third determination submodule is used to determine the Determine the optimal input weight matrix of the lth layer of the compressed long short-term memory network model

[0111] in, is the F norm, is the local data of the output feature map of the output weight matrix, It is the local data of the output feature map of the input weight matrix.

[0112] Based on the above embodiment, in a specific embodiment, the determining module may further include:

[0113] The second extraction submodule is used to extract the input weight matrix W of the l+1 layer of the preset long short-term memory network model based on the set θ x(l+1) The weight of the corresponding row

[0114] The fourth determination submodule is used to determine the Determine the optimal input weight matrix for the l+1th layer of the compressed LSTM network model

[0115] Among them, FM x(l+1) is the output feature map of the input weight matrix of the l+1 layer of the preset long short-term memory network model, W x(l+1)is the input weight matrix of the l+1th layer of the preset long short-term memory network model, W x(l+1) The compressed input weight matrix, for The transposed matrix of is the input value of the l+1th layer of the preset long short-term memory network model at time step t.

[0116] Based on the above embodiment, in a specific embodiment, the system may further include:

[0117] The retraining module is used to retrain the compressed long short-term memory network model for a preset number of times when all layers of the compressed long short-term memory network model have completed parameter optimization.

[0118] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, and will not be repeated here.

[0119] Please refer to Figure 4 , Figure 4 This is a structural diagram of an image processing device provided in an embodiment of the present application.

[0120] The image processing device 400 may vary significantly depending on its configuration or performance, and may include one or more processors (central processing units, CPUs) 422 and memory 432, and one or more storage media 430 (e.g., one or more mass storage devices) storing application programs 442 or data 444. The memory 432 and storage media 430 may be either transient or persistent storage. The program stored in the storage media 430 may include one or more modules (not shown), each of which may include a series of instructions for operating on the device. Furthermore, the processor 422 may be configured to communicate with the storage media 430 to execute the series of instructions stored in the storage media 430 on the image processing device 400.

[0121] The image processing device 400 may also include one or more power supplies 424, one or more wired or wireless network interfaces 450, one or more input and output interfaces 458, and / or one or more operating systems 441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0122] above Figures 1 to 2 The steps in the described image processing method are performed by the image processing device based on the Figure 4 The structure shown is implemented.

[0123] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0125] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0126] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0127] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a function calling device, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0128] The above is a detailed introduction to an image processing method, system, device and readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

[0129] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. An image processing method, characterized in that: include: The weight matrix of the preset long short-term memory network model is grouped according to the inherent structured sparsity to obtain the corresponding weight group; Calculating the Pearson correlation coefficient of each weight group with other weight groups respectively, using the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly selecting the corresponding weight group according to the sampling probability for compression based on a preset compression rate to obtain a compressed long short-term memory network model; when compressing the preset long short-term memory network model, compressing the neurons in each hidden layer by step-by-step compressing the output weights and input weights of the neurons, wherein the process of randomly selecting the weight group for compression is performed layer by layer; The compressed long short-term memory network model is used to perform image processing on the input image.

2. The method according to claim 1, characterized in that Before performing image processing on the input image using the compressed long short-term memory network model, the method further includes: Calculating an output feature map of the weight matrix of the preset long short-term memory network model based on the weight group; Calculating a compressed output feature map of the weight matrix of the compressed long short-term memory network model based on the weight group; Determining the optimal weight matrix of the compressed long short-term memory network model based on the output feature map and the compressed output feature map by the least squares method; The optimal weight matrix is ​​used to optimize the parameters of the compressed long short-term memory network model.

3. The method according to claim 2, characterized in that Calculating an output feature map of the weight matrix of the preset long short-term memory network model according to the weight group includes: Calculate the first The input values ​​of the activation functions of the forget gate, input gate, and output gate of the layer at time step t, as well as the output values ​​at the previous time step t-1 , and determine the corresponding output weight matrix and the input weight matrix ; According to the formula and Calculating an output feature map of the weight matrix of the preset long short-term memory network model; in, The first The output weight matrix of the layer, are the output weights of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, The first The input weight matrix of the layer, are the input weights of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, The first The output feature map of the output weight matrix of the layer, 、 、 、 are the output feature maps of the output weight matrices of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, is the output value of the preset long short-term memory network model at the previous time step t-1, The first The output feature map of the layer's input weight matrix, 、 、 、 are the output feature maps of the input weight matrices of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, The first The input value of the layer at time step t.

4. The method according to claim 3, characterized in that Calculating a compressed output feature map of the weight matrix of the compressed long short-term memory network model based on the weight group includes: Get the first The output value of the layer at the previous time step t-1 , and the output value As the input value of the compressed long short-term memory network model; Will As the compressed output feature map of the output weight matrix of the compressed long short-term memory network model, A compressed output feature map as the input weight matrix of the compressed long short-term memory network model; in, for The compressed output weight matrix, for The transposed matrix of for The compressed input weight matrix, for The transposed matrix of The first The output value of the layer at the previous time step t-1, T is the input data The total number of time steps.

5. The method according to claim 4, characterized in that Determining the optimal weight matrix of the compressed long short-term memory network model based on the output feature map and the compressed output feature map by the least squares method, including: The numbers of the compressed weight groups in the preset long short-term memory network model are recorded as a set ; According to the collection Extracting local data of the output feature map of the output weight matrix and local data of the output feature map of the input weight matrix of the preset long short-term memory network model at the previous time step t-1; According to the formula Determine the first The optimal output weight matrix of the layer ; According to the formula Determine the first The optimal input weight matrix of the layer ; in, is the F norm, is the local data of the output feature map of the output weight matrix, It is the local data of the output feature map of the input weight matrix.

6. The method according to claim 5, characterized in that Also includes: According to the collection Extract the first The input weight matrix of the layer The weight of the corresponding row ; According to the formula Determine the first The optimal input weight matrix of the layer ; in, The first The output feature map of the layer's input weight matrix, The first The input weight matrix of the layer, for The compressed input weight matrix, for The transposed matrix of The first The input value of the layer at time step t.

7. The method according to claim 2, characterized in that After optimizing the parameters of the compressed long short-term memory network model using the optimal weight matrix, the method further includes: When all layers of the compressed long short-term memory network model have completed parameter optimization, the compressed long short-term memory network model is retrained a preset number of times.

8. An image processing system, characterized in that: include: A grouping module is used to group the weight matrix of the preset long short-term memory network model according to the inherent structured sparsity to obtain corresponding weight groups; A compression module is configured to calculate the Pearson correlation coefficient of each weight group with other weight groups, use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly select the corresponding weight group according to the sampling probability based on a preset compression rate for compression, thereby obtaining a compressed long short-term memory network model; when compressing the preset long short-term memory network model, the neurons in each hidden layer are compressed by step-by-step compressing the output weights and input weights of the neurons, wherein the process of randomly selecting the weight group for compression is performed layer by layer; The image processing module is used to perform image processing on the input image using the compressed long short-term memory network model.

9. An image processing device, characterized in that include: memory for storing computer programs; A processor, configured to implement the steps of the image processing method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image processing method according to any one of claims 1 to 7.

Images