Image processing method, electronic device, and computer-readable storage medium
By dividing image data into frequency domain component data and extracting spectral features, the problems of information loss and time consumption caused by image scaling are solved, and efficient image processing by neural networks is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2022-12-31
- Publication Date
- 2026-05-19
AI Technical Summary
In existing technologies, image scaling leads to information loss or increased processing time, affecting the detection performance and real-time performance of neural networks. Furthermore, image formats such as JPEG2000 affect processing speed during decoding.
Image data is divided into several frequency domain components, and spectral features are extracted through an image processing network. The feature extraction layer uses the output of the previous feature extraction layer and the current output to extract features, making full use of the differences in feature information between different spectra.
It improves the speed and accuracy of neural network image processing, reduces image processing time, and optimizes image processing results.
Smart Images
Figure CN116051956B_ABST
Abstract
Description
Technical Field
[0001] This application relates to deep learning, and more particularly to an image processing method, an electronic device, and a computer-readable storage medium. Background Technology
[0002] In the field of deep learning, neural networks often use RGB images in the spatial domain as direct input, and the image size is usually fixed. Scaling the image down to a preset size generally results in the loss of important information, leading to a decrease in detection performance. Enlarging the image to a preset size increases the processing time depending on the input image size; smaller images take more time, and larger images take less time. This method cannot meet real-time requirements. Furthermore, input images are typically stored in formats such as JPEG and JPEG2000, and decoding these images to the spatial domain also affects the image processing speed of the neural network. Summary of the Invention
[0003] The main objective of this application is to provide an image processing method, electronic device, and computer-readable storage medium that can improve the image processing speed of neural networks.
[0004] To address the aforementioned technical problems, the first technical solution adopted in this application is to provide an image processing method. This method includes acquiring image data and obtaining several frequency domain component data corresponding to the image data; sequentially inputting the several frequency domain component data into an image processing network to obtain an image processing result corresponding to the image data; wherein the image processing network includes an input layer and multiple sequentially connected feature extraction layers, the input layer of each image processing network being used to extract the spectral features of the currently input frequency domain component data; each feature extraction layer performing feature extraction based on the output of the previous feature extraction layer, wherein the output of the previous feature extraction layer includes the current output of the previous feature extraction layer and the previous output, the previous output being obtained based on the previously input frequency domain component data.
[0005] To address the aforementioned technical problems, the second technical solution adopted in this application is to provide an electronic device. This electronic device includes a memory and a processor. The memory stores program data, which can be executed by the processor to implement the method described in the first technical solution.
[0006] To address the aforementioned technical problems, the third technical solution adopted in this application is to provide a computer-readable storage device. This computer-readable storage device stores program data and can be executed by a processor to implement the method described in the first technical solution.
[0007] The beneficial effects of this application are as follows: Unlike existing technologies, this application improves the image processing speed of the neural network by acquiring image data in advance, dividing the image data into several frequency domain components, and then inputting them into the image processing network. After inputting the several frequency domain components into the image processing network, the spectral features corresponding to the frequency domain components are extracted by the input layer. The feature extraction layer performs feature extraction based on the current output and the previous output of the previous feature extraction layer. The previous output is based on the frequency domain components of the previous input, and the current output is based on the frequency domain components of the current input. This allows the output of the feature extraction layer to be used forward, making full use of the differences in feature information between different spectra, resulting in more accurate image processing results. Attached Figure Description
[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0009] Figure 1 This is a flowchart illustrating the first embodiment of the image processing method of this application;
[0010] Figure 2 It is a basic structural diagram of a multi-level image processing network;
[0011] Figure 3 This is a process unfolding diagram of a multi-level image processing network;
[0012] Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;
[0013] Figure 5 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation
[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0015] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0016] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0017] Reference Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image processing method of this application.
[0018] S11: Acquire image data and obtain several frequency domain component data corresponding to the image data.
[0019] The process involves acquiring image data, which can be any image data. There are no restrictions on the format or type of the image data, nor on the processing method used to obtain the frequency domain component data; the only requirement is to ultimately obtain several frequency domain component data from the image data.
[0020] S12: Input several frequency domain component data into the image processing network in sequence to obtain the image processing results of the corresponding image data.
[0021] Image processing networks consist of an input layer and multiple sequentially connected feature extraction layers. The input layer extracts the spectral features of the frequency domain components of the current input data. Each feature extraction layer extracts features based on the output of the previous layer. The output of the previous layer includes its current output and the previous output, which is derived from the frequency domain components of the previous input. Image processing can be used for target prediction, target localization, target classification, action recognition, etc.
[0022] In this embodiment, the present application improves the image processing speed of the neural network by acquiring image data in advance, dividing the image data into several frequency domain components, and then inputting the image data into the image processing network. After the several frequency domain components are input into the image processing network, the spectral features corresponding to the frequency domain components are extracted by the input layer. The feature extraction layer performs feature extraction based on the current output and the previous output of the previous feature extraction layer. The previous output is based on the frequency domain components of the previous input, and the current output is based on the frequency domain components of the current input. This allows the output obtained by the feature extraction layer to be used forward, making full use of the differences in feature information between different spectra, and making the image processing results more accurate.
[0023] In one embodiment, when several frequency domain component data are input into an image processing network to obtain the image processing result of the corresponding image data, the several frequency domain component data are input into the image processing network in order of frequency from low to high to obtain the image processing result of the corresponding image data.
[0024] By inputting frequency domain component data into the image processing network in ascending order of frequency, the image processing network can make full use of the differences in spectral features from low to high frequencies, resulting in more accurate image processing results.
[0025] In one embodiment, when acquiring image data and obtaining several frequency domain component data corresponding to the image data, the acquired image data is divided into frequency domains according to the decision tree results to obtain several corresponding frequency domain component data. The decision tree results are determined based on historical image processing results and historical frequency domain component data. The decision tree determines the frequency of the current optimal input frequency domain component data based on historical image processing results and historical frequency domain component data. By deploying the decision tree, the neural network can select the most suitable frequency branch based on historical data, reducing the dimensionality of image data, improving the processing speed of the neural network for image data, and ensuring that the accuracy of the image processing results does not differ too much.
[0026] When the current input frequency domain component data is the first input frequency domain component data, the output of the previous feature extraction layer only includes the current output of the previous feature extraction layer. When the current input frequency domain component data is the first input frequency domain component data in a processing flow, the feature extraction layer in the image processing network has not yet processed the frequency domain component data, and there is no previous output. Therefore, when the current input frequency domain component data is the first input frequency domain component data in a processing flow, the output of the next feature extraction layer only includes the current output of the previous feature extraction layer.
[0027] In one embodiment, each feature extraction layer in the image processing network includes at least one of a convolutional layer, an activation layer, a normalization layer, and a pooling layer. A convolutional layer in a convolutional neural network consists of several convolutional units, and the parameters of each convolutional unit are optimized through backpropagation. The purpose of convolution is to extract different features from the input. The first convolutional layer may only extract low-level features such as edges, lines, and corners, while more layers in the network can iteratively extract more complex features from these low-level features.
[0028] The most important component of the activation layer is the activation function. Its main function is to introduce nonlinear factors, mapping features to a high-dimensional nonlinear region for interpretation, thus solving problems that linear models cannot address. Activation functions can include the sigmoid function and the ReLU function.
[0029] The specific function of normalization is to summarize and unify the statistical distribution of samples. Normalization between 0 and 1 represents a statistical probability distribution, while normalization between -1 and +1 represents a statistical coordinate distribution. Normalization implies uniformity, unification, and consistency. Whether for modeling or computation, the basic unit of measurement must be consistent. Neural networks are trained (probability calculation) and predicted based on the statistical probabilities of samples in events, and the sigmoid function takes values between 0 and 1. The output of the last node in the network also follows this pattern, so the output of the samples often needs to be normalized. Normalization unifies the statistical probability distribution between 0 and 1. When all input signals of samples are positive, the weights connected to the neurons in the first hidden layer can only increase or decrease simultaneously, resulting in a very slow learning speed. In addition, there are often outlier samples in the data. The presence of outlier samples increases the network training time and may cause the network to fail to converge. To avoid this situation and to facilitate subsequent data processing and accelerate network learning, the input signal can be normalized so that the mean of the input signal for all samples is close to 0 or very small compared to its root mean square error.
[0030] Pooling is another important concept in convolutional neural networks (CNNs), and it's essentially a form of downsampling. There are various non-linear pooling functions, with max pooling being the most common. It divides the input image into several rectangular regions and outputs the maximum value for each sub-region. Intuitively, this mechanism is effective because, after a feature is discovered, its precise location is far less important than its relative position to other features. Pooling layers continuously reduce the spatial size of the data, thus decreasing the number of parameters and computational cost, which to some extent controls overfitting. Typically, pooling layers are periodically inserted between convolutional layers in a CNN. Pooling layers usually act on each input feature individually and reduce its size.
[0031] The image processing network described above also includes a prediction layer, which is used to predict the features output by the feature extraction layer to obtain the image processing result. The prediction layer may include a classification head, a detection head, or a segmentation head.
[0032] When training an image processing network using the aforementioned image processing method, the image data is adjusted to be of the same category. All frequency domain components of the image data are obtained and input into the image processing network for processing. The network parameters are adjusted based on the image processing results and their corresponding image data, enabling the network to achieve more accurate image processing results. After training, a decision tree method is used to select the optimal frequency combination for each frequency domain component. This optimal frequency combination is then used to process subsequent image data of the same category. This reduces the dimensionality of the image data, improves the processing speed of the neural network, and ensures that the accuracy of the image processing results remains consistent.
[0033] The image processing method of this application will be described in more detail below with a specific embodiment.
[0034] Step 1: Obtain the frequency domain image.
[0035] This application does not limit the method of acquiring the frequency domain format image to be processed, nor does it limit the format (such as JPEG, JPEG2000, etc.) or size of the acquired image to be detected. If there is no frequency domain image, it can also be generated based on the original format image such as bmp.
[0036] Step 2: Construction of a multi-level image processing network.
[0037] The basic structure of a multi-level image processing network includes an input F, a feature extraction layer N, and a prediction layer H, such as... Figure 2 As shown, Figure 2This is a basic structural diagram of a multi-level image processing network. The input layer extracts spectral features of a specific frequency in the image. The feature extraction layer uses a classic neural network structure, containing n sub-networks. These sub-modules can perform feature extraction and dimensionality reduction, and may include convolutional layers, activation layers, normalization layers, pooling layers, etc. The prediction layer can employ classification heads, detection heads, segmentation heads, etc., depending on the task, without being limited to a specific task.
[0038] Step 3: Optimization of the multi-level image processing network.
[0039] Multi-level image processing networks can perform multi-level processing based on the amount of spectrum in the input image. If the spectral feature dimension of the image is m, and the spectral feature is denoted as F... i Where i∈[1,m-1]. The feature extraction layer contains n sub-networks, denoted as B. j , j∈[1,n-1].
[0040] like Figure 3 The diagram shown is an unfolded representation of a multi-level image processing network. All input frequency domain components F x Forward feature extraction is performed on all components. For frequency domain components that are not the lowest frequency, forward feature extraction is performed while also relying on the forward features of the previous frequency domain component, i.e.:
[0041] E i =N(F i +N(F i-1 )), i∈[1,m-1]
[0042] When the input feature is the lowest frequency domain component:
[0043] E0 = N(F0)
[0044] For subnetwork B in the feature extraction network j Its input and output form can be represented by the following formula:
[0045]
[0046] Step 4: Deployment of a multi-level image processing network.
[0047] Deployment of the trained multi-level image processing network. For compressed images, such as those in JPEG2000 format, features are transmitted sequentially from low to high frequency during network transmission. The multi-level image processing network outputs features sequentially based on their frequency for processing, similar to the optimization process. As follows:
[0048] H0 = H(N(F0))
[0049] H i=H(N(F) i +N(F i-1 )))
[0050] Among them, N(F) i-1 () is a feature map of the forward process of the previous frequency.
[0051] In actual deployment, a method similar to decision trees can also be used to obtain the optimal frequency combination.
[0052] like Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of this application.
[0053] The electronic device includes a processor 110 and a memory 120.
[0054] Processor 110 controls the operation of electronic devices. Processor 110 may also be referred to as a CPU (Central Processing Unit). Processor 110 may be an integrated circuit chip with signal sequence processing capabilities. Processor 110 may also be a general-purpose processor, a digital signal sequence processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.
[0055] The memory 120 stores the instructions and program data required for the processor 110 to operate.
[0056] The processor 110 is used to execute instructions to implement the methods provided by any embodiment and possible combination of the image processing methods described above in this application.
[0057] like Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application.
[0058] One embodiment of the readable storage medium of this application includes a memory 210 that stores program data, which, when executed, implements the method provided by any embodiment and possible combination of the image processing method of this application.
[0059] The memory 210 may include a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or other media that can store program instructions. Alternatively, it may be a server that stores the program instructions, which can send the stored program instructions to other devices for execution or execute the stored program instructions itself.
[0060] In summary, this application improves the image processing speed of the neural network by pre-acquiring image data, dividing it into several frequency domain components, and then inputting them into the image processing network. After inputting the frequency domain components into the image processing network, the input layer extracts the spectral features corresponding to each frequency domain component. The feature extraction layer then performs feature extraction based on the current output and the previous output of the previous feature extraction layer. The previous output is based on the frequency domain components input previously, and the current output is based on the frequency domain components input currently. This allows the output obtained by the feature extraction layer to be used forward, fully utilizing the differences in feature information between different spectra, resulting in more accurate image processing results.
[0061] In the multi-level image processing network of this application, the spectral features of each frequency domain component data are processed under the same network structure. Image data is input from low frequency to high frequency, realizing multi-level image processing. Feature information of different frequencies shares the same network structure and parameters, which can reduce the size of the image processing network and facilitate its deployment. The input of each current feature extraction network unidirectionally depends on the output of the previous spectrum in the previous extraction network, making the input frequency domain component data decomposable. When processing image data, the optimal frequency combination can be selected according to the decision tree, thereby further reducing the image processing time of the image processing network.
[0062] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
[0063] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0064] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0065] If the integrated units in the other embodiments described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0066] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. An image processing method, characterized in that, The method includes: Acquire image data and obtain several frequency domain component data corresponding to the image data; The frequency domain component data are input into the image processing network in order of frequency from low to high to obtain the image processing result corresponding to the image data; The image processing network includes an input layer and multiple sequentially connected feature extraction layers. The input layer is used to extract the spectral features of the frequency domain component data of the current input. Each feature extraction layer performs feature extraction based on the output of the previous feature extraction layer. The output of the previous feature extraction layer includes the current output of the previous feature extraction layer and the previous output, which is obtained based on the frequency domain component data of the previous input.
2. The method according to claim 1, characterized in that, The acquisition of image data and the obtaining of several frequency domain component data corresponding to the image data include: Image data is acquired, and the image data is divided into frequency domains according to the decision tree results to obtain several corresponding frequency domain component data. The decision tree results are determined based on historical image processing results and historical frequency domain component data.
3. The method according to claim 1, characterized in that, When the current input frequency domain component data is the first input frequency domain component data, the output of the previous feature extraction layer only includes the current output of the previous feature extraction layer.
4. The method according to claim 1, characterized in that, Each feature extraction layer includes at least one of an activation layer, a normalization layer, a pooling layer, and a convolutional layer.
5. The method according to claim 1, characterized in that, The image processing network further includes a prediction layer, which is used to predict the features output by the feature extraction layer to obtain the image processing result.
6. The method according to claim 5, characterized in that, The prediction layer includes a classification head, a detection head, or a segmentation head.
7. The method according to claim 1, characterized in that, The network parameters of the image processing network are adjusted using the image processing results.
8. An electronic device, characterized in that, It includes a memory and a processor, the memory being used to store program data, the program data being executable by the processor to implement the method as described in any one of claims 1-7.
9. A computer-readable storage medium, characterized in that, It stores program data that can be executed by a processor to implement the method as described in any one of claims 1-7.