Inplausible lightweight deep neural network for resource-constrained equipment

Abstract

The invention discloses an interpretable lightweight deep neural network for resource-constrained equipment, and relates to the field of artificial intelligence and machine learning. Aiming at the problems of poor interpretability, high calculation complexity and large resource consumption when an existing deep neural network processes a complex data task, iterative calculation in a kernel least mean square error (KLMS) algorithm is converted into forward propagation in the deep neural network. Each layer of network in the KLMS-Net corresponds to an iteration step of an original KLMS, a kernel-like implicit mapping is learned in a data-driven manner through a deep neural network module, and a kernel skill which is difficult to analyze in the original KLMS is replaced. According to the method, good interpretability is kept, performance superior to that of a traditional deep neural network can be provided while low calculation complexity is kept, and the decision-making process of the method has high transparency and credibility. Compared with a traditional method, the network model provided by the invention has excellent performance, good interpretability and higher resource utilization efficiency.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and machine learning, and in particular to a lightweight deep neural network structure and a training method thereof suitable for resource-constrained devices (such as mobile devices and Internet of Things devices). Background Art

[0002] With the development of artificial intelligence and machine learning technologies, deep neural networks have achieved remarkable results in various fields, such as image recognition, natural language processing, speech recognition, etc. However, traditional deep neural networks still face some important challenges in practical applications, mainly including the following aspects:

[0003] 1) Explainability issues: Although deep neural networks have powerful modeling capabilities, their internal structure and decision-making process are often difficult to explain. This "black box" nature makes it difficult for users to understand the decision logic of the model, thereby reducing its credibility and transparency in safety-critical and demanding applications. For example, in medical diagnosis, unexplainable decisions may lead to misdiagnosis; in financial decision-making, opaque models may increase systemic risks; in autonomous driving, models that lack explanations may lead to safety hazards. Therefore, the interpretability of the model is crucial to ensure the reliability and safety of the system.

[0004] 2) Computational complexity and resource consumption: Traditional deep neural networks usually contain a large number of parameters and complex calculations, resulting in a large amount of computing resources and storage space required for their training and inference processes. This poses a huge challenge to devices with limited computing resources, such as mobile devices and IoT devices. For example, running complex neural network models on mobile devices may cause the battery to drain quickly, affecting the user experience. In addition, the high energy consumption and long latency of large-scale neural network models in actual deployment also limit their use in real-time applications, which is particularly evident in fields such as autonomous driving and real-time video processing.

[0005] 3) Model generalization ability: Deep neural networks often perform well on training data, but their generalization ability may be limited when processing new data that they have never seen. This is because the model may overfit the training data during training and ignore the underlying mathematical and physical laws. For example, a medical diagnosis model that is well trained in a laboratory environment may perform poorly in an actual clinical environment, affecting its reliability and practicality. Summary of the invention

[0006] Aiming at the problems of poor interpretability, complex calculation and weak generalization ability of existing neural network models on resource-constrained devices, the present invention proposes a new interpretable lightweight deep neural network (KLMS-Net) for resource-constrained devices.

[0007] The technical solution of the present invention is: an interpretable lightweight deep neural network for resource-constrained devices, the network is composed of a multi-layer network, the input signal of the i-th layer network is the current incremental sample x(i) and the historical sample x(1:i-1), and the historical sample of the first layer network is the initialized sample; the incremental sample x(i) and the historical sample x(1:i-1) are both sent to the deep neural network module for processing; the deep neural network module is linearly combined with the historical error e(1:i-1) to obtain a new output value y(i); then, the output value y(i) of the network module is compared with the expected value T(i) to obtain the error value e(i), the current error value e(i) is calculated, and the error value e(i) is passed to the next layer network;

[0008] The deep neural network module includes three subnetworks with the same architecture, namely: subnetwork 1, subnetwork 2, and subnetwork 3. The incremental sample x(i) is input into subnetwork 1, and the historical sample x(1:i-1) is input into subnetwork 2. The output of subnetwork 1 and subnetwork 2 is input into subnetwork 3 after being subtracted. The output of subnetwork 3 is the output of the deep neural network module. Subnetwork 1 and subnetwork 2 convolve the corresponding input into a high-dimensional space, and subnetwork 3 deconvolutes the corresponding input into a low-dimensional space.

[0009] Furthermore, the sub-network 1 and sub-network 2 have the same structure, both including two convolutional layers and corresponding activation layers. Suppose the input is x i ∈R B×C×L , where B represents the batch size, C represents the number of input channels, and L is the length of the input sequence; the convolution kernel of the first convolutional layer is W1∈R Cout×C×K , the offset term is b1∈R Cout ; Among them, R Cout×C×K represents a real number set of dimension Cout×C×K, where C is the number of input channels, K is the convolution kernel size, and C out is the number of output channels; the output y1∈R of the first convolutional layer B×Cout×L The expression is:

[0010]

[0011] where b indexes the samples in the batch, and C out and C represent the output and input channels respectively, and l represents the position in the output sequence.

[0012] The activation function uses LeakyReLU, and the expression is:

[0013]

[0014] Then the expression of the output a1 of the first activation layer is:

[0015] a1[b,c out , l]=max(0.2·y1[b,c out , l], y1[b, c out , l])

[0016] The steps for the second convolutional layer and activation layer are the same as the first convolutional layer and activation layer;

[0017] After the incremental samples and historical samples are convolved into high-dimensional space by sub-network 1 and sub-network 2 respectively, the Euclidean distance between the samples is calculated and sent to sub-network 3; the structure of sub-network 3 is the same as that of sub-networks 1 and 2, but the role of the convolution layer is opposite to that in sub-networks 1 and 2. Its two convolution layers gradually deconvolute the Euclidean distance in the high-dimensional space processed by sub-networks 1 and 2 to the original space of the input signal.

[0018] Furthermore, the network dynamics processes of subnetwork 1, subnetwork 2 and subnetwork 3 are respectively θ1 (·), f θ2 (·) and f θ3 (·), then the expression of SRF(i) of the i-th layer network is:

[0019] SRF(i)=f θ3 (f θ1 (x(i))-f θ2 (x(1:i-1)))

[0020] Among them, x(i) and x(1:i-1) represent the input samples of the i-th layer network and the input samples of the first i-1 layers network respectively;

[0021] The output f(i) of the i-th layer network is expressed as a linear combination of SRF(i) and the historical error of the previous i-1 layers of the network, and its expression is:

[0022]

[0023] Where e(j) = d(j) - f(j), is the output error of the j-th layer network; d(j) is the expected output of the j-th layer network; θ i is the adaptive learning step size;

[0024] The network output f(i) of the i-th layer is subtracted from the current expected value to obtain the calculation error of the i-th layer network, that is, e(i)=d(i)-f(i), and this value is passed to the i+1-th layer network; in this way, the entire interpretable lightweight deep neural network is trained end-to-end, and the deep neural network module can adaptively adjust the form and parameters of SRF based on the characteristics of each data, thereby enhancing the expression ability of KLMS-Net;

[0025] The training process involves two processes: back propagation and parameter update. The loss function L is to minimize the mean square error, and the update rule is the Adam rule, which is expressed as follows:

[0026]

[0027] Among them, θ old represents the neuron weight value before updating, θ new Represents the updated neuron weight value; parameter α represents a predetermined learning rate; ∈ A fixed value is set to avoid division by zero; and are the bias corrections for the first-order and second-order matrix estimates, respectively, and are expressed as follows:

[0028]

[0029] in, Represents the decay rate, which is usually close to 1. The first-order matrix m in the formula t and the second-order matrix v t The expression is as follows:

[0030]

[0031] represents the change in neuron weights, and L represents the loss function.

[0032] A mobile image enhancement device using the explainable lightweight deep neural network is applied to the scene of image enhancement in real-time video calls;

[0033] In real-time video calls, in order to improve the problem of video quality degradation caused by network bandwidth limitations, compression algorithms or device performance, mobile devices using the above-mentioned explainable lightweight deep neural network can perform real-time image enhancement. Since the network structure is lightweight, it is suitable for running on resource-constrained mobile devices. It can process each video frame in real time during the video call to ensure that the call quality is not affected, while saving computing resources and battery consumption. Ensure that users can still enjoy clearer and smoother video call quality under low bandwidth or undesirable network conditions.

[0034] The mobile image enhancement device first divides each real-time video frame into image blocks of fixed size (for example, 16x16 pixels) for block-by-block processing; at the same time, the corresponding image blocks of the previous frame are used as historical samples to help understand background stability and motion trends; during the processing, subnetwork 1 convolves the image blocks of the current frame into a high-dimensional feature space to extract details and texture information; subnetwork 2 convolves the image blocks of the historical frame into a high-dimensional feature space to capture background information and motion patterns; by calculating the Euclidean distance between the incremental sample and the historical sample in the high-dimensional space, important detail changes are identified and retained; subnetwork 3 restores the features in the high-dimensional space to the low-dimensional image space through deconvolution operations, and outputs the enhanced image blocks to maintain clarity and details;

[0035] Then, the output enhanced image block is compared with the expected clear image block, and the error sum and its gradient are calculated; the network parameters are dynamically adjusted according to the error gradient to optimize the sample relationship function SRF learned by KLMS-Net to adapt to different scenes and lighting conditions; through frame reconstruction technology, all enhanced image blocks are spliced ​​into a complete enhanced video frame and updated to the video stream in real time to ensure the smoothness and clarity of the call screen. By dynamically adjusting the SRF learned from the image blocks, the network can adapt to different lighting, motion and scene changes, improving the user experience of video calls.

[0036] Compared with the classic deep neural network model, the present invention retains the original mathematical structure of KLMS and realizes the complementary advantages of deep neural network and KLMS. The network model completes the training with mathematical architecture, strengthens the generalization of deep neural network, compresses the scale of solution space, inherits the advantages of KLMS online incremental learning, and fundamentally alleviates the demand for data volume and computing resource consumption of deep neural network model training. At the same time, more efficient learning and training weakens the demand for network model scale, and the accuracy of the model of the present invention of the same size far exceeds the classic deep neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a network architecture diagram of the present invention.

[0038] Figure 2 Schematic diagram of the deep neural network module of the present invention.

[0039] Figure 3 Schematic diagram of comparative experimental results of Mackey-Glass chaotic system.

[0040] Figure 4 Schematic diagram of comparative experimental results of Lorenz attractor chaotic system.

[0041] Figure 5Schematic diagram of comparative experimental results of sunspot activity sequence. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] The present invention converts the iterative calculation in the kernel least mean square error algorithm (KLMS) into forward propagation in a deep neural network, and each layer of the deep neural network corresponds to an iteration of KLMS. These layers are jointly optimized during the training process, and the powerful approximation ability of the deep neural network is used to automatically learn iterative rules that are better and faster than the manually designed optimization algorithm; each layer of operation is equivalent to the transient state of the original system in time, and the variables transferred between layers are equivalent to the evolution between states in the dynamic system, forming a small discrete approximation of the overall state evolution equation.

[0044] The specific architecture of the deep neural network of the present invention is shown in Figure 1 , the input signal of the i-th layer network is the current incremental sample x(i) and the historical sample x(1:i-1), which are sent to the deep neural network module; the output value y(i) of the deep neural network module is subtracted from the expected value T(i) to obtain the error value e(i), which is sent to the next layer network;

[0045] This deep neural network module attempts to capture relationships from data and provides a customized, data-driven function to evaluate the relationship between samples, called the "sample relationship function" (SRF). The form of the SRF is then adaptively adjusted through end-to-end training. The specific architecture is shown in Figure 2 , which consists of a cascaded convolutional deep neural network, specifically including three sub-network modules; each sub-network has the same architecture, consisting of two convolutional layers and two activation layers; sub-network 1 and sub-network 2 convolve the incremental sample x(i) and the historical sample x(1:i-1) into a high-dimensional space and make a difference; sub-network 3 deconvolves the difference into a low-dimensional space, namely SRF(i) and outputs it.

[0046] Example 1: Network architecture of KLMS-Net

[0047] like Figure 1 As shown in the figure, the present invention proposes a novel lightweight deep neural network architecture (KLMS-Net) for complex data processing tasks on resource-constrained devices. KLMS-Net treats the kernel mapping operation of KLMS as a black box module and uses the deep neural network module to learn implicit mappings in a data-driven manner. The deep neural network module includes three cascaded sub-networks, each of which contains two convolutional layers and two activation layers. The specific architecture is shown in Figure 2 .

[0048] 1. Convolution operations of subnetwork 1 and subnetwork 2:

[0049] Incremental sample x i and historical sample x i-1 Input into sub-network 1 and sub-network 2 respectively, and after passing through two convolutional layers and activation layers, they are mapped to high-dimensional space. The output of the first convolutional layer is:

[0050]

[0051] The activation function uses LeakyReLU, and the output expression is:

[0052] a1[b,c out ,l]=max(0.2·y1[b,c out ,l],y1[b,c out ,l])

[0053] The second convolutional layer and activation layer repeat the previous calculations and will not be described in detail.

[0054] 2. Deconvolution operation of subnetwork 3:

[0055] Subnetwork 3 subtracts the output of subnetwork 1 and subnetwork 2, deconvolves the result into a low-dimensional space, and outputs the sample relationship function (SRF). The specific calculation of the SRF (i) of the i-th layer network is as follows:

[0056] SRF(i)=f θ3 (f θ1 (x(i))-f θ2 (x(1:i-1)))

[0057] The output f(i) of the i-th layer network can be expressed as follows:

[0058]

[0059] Example 2: KLMS-Net training process

[0060] KLMS-Net adaptively adjusts the form and parameters of SRF through an end-to-end training process. The training process includes the following steps:

[0061] Step 1 (Data Preparation): Collect and prepare training data and perform preprocessing.

[0062] Step 2 (network initialization): Initialize the parameters of each layer of KLMS-Net, including convolution kernel, offset term and adaptive learning step size.

[0063] Step 3 (forward propagation): Input the training data into KLMS-Net, pass through the convolutional layers and activation layers of each sub-network, and calculate the output f(i).

[0064] Step 4 (loss calculation): Calculate the loss between the predicted output and the true value, using the mean square error (MSE) as the loss function, which is expressed as follows:

[0065]

[0066] Step 5 (back propagation and parameter update): The gradient of the loss function relative to the parameters of each layer is calculated through the back propagation algorithm, and the parameters are updated using the Adam optimization algorithm. The expression is as follows:

[0067]

[0068] Example 3: Application of KLMS-Net

[0069] Experiments were conducted on multiple datasets, including Mackey-Glass chaotic system, Lorenz attractor chaotic system, and sunspot activity sequence prediction tasks, and the performance of KLMS-Net was compared with traditional deep neural networks and classic adaptive filtering algorithms. The comparison algorithms specifically include the least mean square algorithm (LMS), the kernel least mean square algorithm (KLMS), the recursive least squares algorithm (RLS), the kernel recursive least squares algorithm (KRLS), and the end-to-end deep neural network. The evaluation indicators are mean square error, root mean square error, convergence rate after 50 iterations, and convergence rate after 100 iterations.

[0070] The specific experimental results are shown in Tables 1 to 3, and the convergence images are shown in the attached Figures 3 to 5 The results show that the method proposed by the present invention has the highest output accuracy and the fastest convergence speed compared with the classical adaptive filtering algorithm and the classical deep neural network.

[0071] In view of the above situation, in order to overcome the defects of the prior art, the innovation of the present invention is:

[0072] 1. The iterative process of the KLMS algorithm is transformed into the forward propagation of a deep neural network, achieving innovative integration of the algorithm structure;

[0073] 2. Learning implicit kernel mapping through deep neural network module improves the expressiveness and generalization of the model;

[0074] 3. The mathematical structure of KLMS is retained, while the advantages of deep learning are utilized to achieve the complementary advantages of the two methods. It has both efficient computing power and good interpretability, and can provide better performance than traditional deep neural networks while maintaining low computational complexity, and its decision-making process has high transparency and credibility.

[0075] Through the description of the above specific implementation methods, the implementation process and application scenarios of KLMS-Net are described in detail, so that technicians can implement the deep neural network architecture according to the content of the present invention. The above embodiments are only used to illustrate the technical solution of the present invention and do not limit the scope of protection of the present invention. Technicians in the relevant field can make various deformations and modifications according to the content of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

[0076] Table 1 Comparison of the algorithms in the Mackey-Glass chaos system experiment

[0077]

[0078] Table 2 Comparison of various algorithms in the Lorenz chaotic system experiment

[0079]

[0080] Table 3 Comparison of various algorithms in the sunspot sequence experiment

[0081]

Claims

1. An interpretable lightweight deep neural network for resource-constrained devices, the network consists of a multi-layer network, the input signal of the i-th layer network is the current incremental sample x(i) and the historical sample x(1:i-1), and the historical sample of the first layer network is the initialized sample; the incremental sample x(i) and the historical sample x(1:i-1) are both sent to the deep neural network module for processing; the deep neural network module is linearly combined with the historical error e(1:i-1) to obtain a new output value y(i); then, the output value y(i) of the network module is compared with the error value e(i) obtained by subtracting the expected value T(i), the current error value e(i) is calculated, and the error value e(i) is passed to the next layer network; The deep neural network module includes three subnetworks with the same architecture, namely: subnetwork 1, subnetwork 2, and subnetwork 3. The incremental sample x(i) is input into subnetwork 1, and the historical sample x(1:i-1) is input into subnetwork 2. The output of subnetwork 1 and subnetwork 2 is input into subnetwork 3 after being subtracted. The output of subnetwork 3 is the output of the deep neural network module. Subnetwork 1 and subnetwork 2 convolve the corresponding input into a high-dimensional space, and subnetwork 3 deconvolutes the corresponding input into a low-dimensional space.

2. An interpretable lightweight deep neural network for resource-constrained devices as claimed in claim 1, characterized in that: The sub-network 1 and sub-network 2 have the same structure, both of which include two convolutional layers and corresponding activation layers. Suppose the input is x i∈ R B ×C×L , where B represents the batch size, C represents the number of input channels, and L is the length of the input sequence; the convolution kernel of the first convolutional layer is W 1∈ R Cout×C×K , the offset term is b1∈R Cout , where R Cout×C×K represents a real number set of dimension Cout×C×K, where C is the number of input channels, K is the convolution kernel size, and C out is the number of output channels; the output y of the first convolutional layer 1∈ R B×Cout×L The expression is: where b indexes the samples in the batch, and C out and C represent the output and input channels respectively, and l represents the position in the output sequence. The activation function uses LeakyReLU, and the expression is: Then the expression of the output a1 of the first activation layer is: a1[b,c out ,l] ＝ max(0.2·y1[b,c out ,l],y1[b,c out ,l]) The steps for the second convolutional layer and activation layer are the same as the first convolutional layer and activation layer; After the incremental samples and historical samples are convolved into high-dimensional space by sub-network 1 and sub-network 2 respectively, the Euclidean distance between the samples is calculated and sent to sub-network 3; the structure of sub-network 3 is the same as that of sub-networks 1 and 2, but the role of the convolution layer is opposite to that in sub-networks 1 and 2. Its two convolution layers gradually deconvolute the Euclidean distance in the high-dimensional space processed by sub-networks 1 and 2 to the original space of the input signal.

3. An interpretable lightweight deep neural network for resource-constrained devices as claimed in claim 1, characterized in that: The network dynamics processes of subnetwork 1, subnetwork 2 and subnetwork 3 are respectively f θ1 (·), f θ2 (·) and f θ3 (·), then the expression of SRF(i) of the i-th layer network is: SRF(i)=f θ3 (f θ1 (x(i))-f θ2 (x(1:i-1))) Among them, x(i) and x(1:i-1) represent the input samples of the i-th layer network and the input samples of the first i-1 layers network respectively; The output f(i) of the i-th layer network is expressed as a linear combination of SRF(i) and the historical error of the previous i-1 layers of the network, and its expression is: Where e(j) = d(j) - f(j), is the output error of the j-th layer network; d(j) is the expected output of the j-th layer network; θ i is the adaptive learning step size; The network output f(i) of the i-th layer is subtracted from the current expected value to obtain the calculation error of the i-th layer network, that is, e(i) = d(i) - f(i), and this value is passed to the i+1-th layer network; The training process involves two processes: back propagation and parameter update. The loss function L is to minimize the mean square error, and the update rule is the Adam rule, which is expressed as follows: Among them, θ old represents the neuron weight value before updating, θ new Represents the updated neuron weight value; parameter α represents the predetermined learning rate; ∈ is a fixed value set to avoid division by zero; and are the bias corrections for the first-order and second-order matrix estimates, respectively, and are expressed as follows: in, Represents the decay rate, which is usually close to 1. The first-order matrix m in the formula t and the second-order matrix v t The expression is as follows: represents the change in neuron weights, and L represents the loss function.

4. A mobile image enhancement device using the interpretable lightweight deep neural network described in claim 1, applied to the scene of image enhancement in real-time video calls; The mobile image enhancement device first divides each real-time video frame into image blocks of fixed size (for example, 16x16 pixels) for block-by-block processing; at the same time, the corresponding image blocks of the previous frame are used as historical samples to help understand background stability and motion trends; during the processing, subnetwork 1 convolves the image blocks of the current frame into a high-dimensional feature space to extract details and texture information; subnetwork 2 convolves the image blocks of the historical frame into a high-dimensional feature space to capture background information and motion patterns; by calculating the Euclidean distance between the incremental sample and the historical sample in the high-dimensional space, important detail changes are identified and retained; subnetwork 3 restores the features in the high-dimensional space to the low-dimensional image space through deconvolution operations, and outputs the enhanced image blocks to maintain clarity and details; Then, the output enhanced image block is compared with the expected clear image block, and the total error and its gradient are calculated; the network parameters are dynamically adjusted according to the error gradient, and the sample relationship function SRF learned by KLMS-Net is optimized to adapt to different scenes and lighting conditions; through frame reconstruction technology, all enhanced image blocks are spliced ​​into a complete enhanced video frame and updated to the video stream in real time to ensure the smoothness and clarity of the call picture.

Images