An image classification method based on constructing local optimal solution of deep ReLU neural network based on sample space partitioning

Abstract

An image classification method for constructing a local optimal solution of a deep ReLU neural network based on sample space partitioning belongs to the field of deep learning. Currently, neural networks are widely used in the field of image classification, but their convergence to a local optimal solution or even a global optimal solution requires iteration through a gradient descent algorithm, which consumes a lot of computing resources and time. Starting from this problem, the present invention divides a specific image classification data set into subsets, solves the optimal linear fitting function for each subset, adds auxiliary linear segments between adjacent subsets, and constructs a continuous piecewise linear function through the max-min method. A deep ReLU neural network is constructed based on the basic properties of the ReLU activation function, so that the network output is the target continuous piecewise linear function. It is verified by theory and experiment that the current network parameters correspond to a local optimal solution. The present invention greatly improves the efficiency of the deep ReLU neural network in finding the local optimal solution by dividing the sample space of the data set.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and more specifically to a deep learning approach for image classification based on sample space partitioning to construct a local optimal solution of a deep ReLU neural network. Background Art

[0002] In recent years, artificial intelligence (AI) has been widely applied across various industries, particularly in image classification, where it has reached or even surpassed average human performance. This is largely due to the rapid development of deep learning technology. With the advent of the big data era, large datasets of samples can be collected for specific image classification problems. Deep learning technology, on the other hand, achieves accurate predictions for unknown samples by constructing various deep neural networks to fit these datasets. Essentially, this approach involves adding nonlinear activation functions and randomly initializing network parameters, resulting in a high-dimensional nonlinear function as the output of the neural network. Continuous training and optimization using the backpropagation algorithm ultimately ensures that the output function closely matches the sample points in the dataset. This allows accurate predictions for new problems of the same type using the trained deep neural network.

[0003] Deep learning technology has significant advantages in the field of image classification. Compared to traditional artificial intelligence techniques, it can handle more complex problems and achieve higher accuracy. However, due to the large number of parameters in deep neural networks and the extensive data required for training and optimization, the network training process places high demands on computer computing power and is time-consuming, with some deep models taking up to months to train. To address this limitation of deep learning, numerous solutions have been proposed. For example, Google has launched the Tensor Processing Unit (TPU), a dedicated chip for neural network computing, which improves computer computing power from a hardware perspective. Research has also been conducted on optimizing gradient descent algorithms to enable faster training and fitting of deep learning models. Others have investigated pruning methods for deep learning models to reduce model complexity and thus shorten training time. While these methods have significantly improved the efficiency of neural network training, they still require significant computing resources and time to find the local or global optimal solution. Therefore, how to skip the neural network training process and find the local or even global optimal solution during network parameter initialization has become an important issue worthy of in-depth consideration.

[0004] Inspired by the above problems, the present invention proposes an image classification method based on sample space partitioning to construct a local optimal solution of a deep ReLU neural network. Summary of the Invention

[0005] The problem to be solved by the present invention is that the deep learning model needs to go through the process of model training to converge to the local optimum, and the parameter values ​​are continuously fine-tuned by the gradient descent algorithm, which consumes a lot of computing resources and time. In response to the above problems, the present invention studies the deep ReLU neural network. According to the properties of the ReLU activation function, the output of the deep ReLU neural network is a continuous piecewise linear function, which is equivalent to dividing the data set into several non-overlapping subsets, each subset is fitted by a corresponding linear segment, and the network training process is the process of continuous fine-tuning of each linear segment. Based on this idea, a reverse analysis is performed. If the data set is artificially divided into several non-overlapping subsets, the optimal linear fit is performed on each subset, and then these linear segments are used to form a continuous piecewise linear function, and then a deep ReLU network is constructed so that its output is this function. Since each linear segment is the optimal linear fit of the current subset, and adding a perturbation to the network weights will not change the way the subsets are divided, the deep ReLU neural network constructed in this way is a local minimum point.

[0006] The present invention is mainly divided into three parts:

[0007] (1) Divide the data set into N (1<N≤M) disjoint subsets, where M is the number of sample points in the data set, and calculate the optimal linear fitting function corresponding to each subset.

[0008] (2) The optimal fitting function of each subset is used to construct a continuous piecewise linear function, and by adding auxiliary linear segments in the area between the two subsets, it is ensured that after the continuous piecewise linear function is constructed, the sample point area that each linear segment is responsible for fitting remains unchanged.

[0009] (3) Design the network structure and construct a deep ReLU neural network. According to the properties of the ReLU activation function, the output of the network is made into the above-mentioned continuous piecewise linear function to complete the image classification task.

[0010] The specific technical solution of the method proposed in the present invention includes 5 steps:

[0011] 1. Preprocess the image dataset. If the samples in the dataset are image features, they are in the form of one-dimensional vectors and can be used directly. If the samples are images themselves, the images must be reduced in dimensionality. Taking the Cifar10 image dataset as an example, each image is a color RGB image of 32*32 dimensions, with ten categories. Reduce the image pixel matrix and stretch it into a one-dimensional vector, while retaining the category labels for later use.

[0012] 2. Dataset subset division. According to the distribution of the image dataset, the image dataset D is artificially divided into N non-overlapping subsets, namely: Among them D iRepresents the i-th subset after division, and the corresponding sample area is Save the subset division method for subsequent use.

[0013] 3. Perform an optimal linear fit on the sample points in each subset. Determine the loss function and find the linear hyperplane that best fits all sample points in the subset, i.e., minimizes the overall loss. For image classification problems, the cross-entropy loss function is generally used. In this case, a two-layer neural network can be constructed, with only an input layer and an output layer. No activation function is used, and the number of neurons in the output layer corresponds to the number of output dimensions of the optimal linear hyperplane. The network parameters are randomly initialized, and the optimal linear hyperplane corresponding to the subset is found using a gradient descent algorithm. Because the network has very few parameters, the operation is very fast. The optimal linear hyperplane for each subset and its corresponding subset range are saved.

[0014] 4. Connect the optimal linear slices corresponding to each subset to form a continuous piecewise linear function, and verify that the continuous piecewise linear function obtained at this time is a local optimal solution. For the optimal linear fitting function {f j , j∈[1, N]}, the corresponding continuous piecewise linear function L can be obtained by max-min method * .Right now:

[0015]

[0016] Among them, calculate Ψ i When f j Must satisfy the subset D i The corresponding sample area The value at any point is greater than or equal to t i , f i For subset D i The corresponding optimal linear fitting function. K i Indicates The linear function f that satisfies the above conditions j The number of Ψ i Indicates In the function f, all functions that satisfy the conditions are j A continuous piecewise linear function constructed by solving the minimum value. L * is the final target continuous piecewise linear function.

[0017] Since the sample points in the data set are discretely distributed, there must be areas without sample points between adjacent subsets, which are called blank areas. When the intersection of adjacent linear slices is not in the blank area between the two subsets, the sample points corresponding to the linear segments in the continuous piecewise linear function will change, and then the overall loss of the data set D will change. To solve this problem, this method constructs auxiliary linear slices in the blank area to connect the optimal linear slices of adjacent subsets. Since there are no sample points in the blank area, the auxiliary linear slices will not affect the overall empirical loss. At this time, it can be guaranteed that after the continuous piecewise linear function is constructed, the overall loss of the data set D is the same as that of each subset D. i The sum of the losses calculated by the optimal linear slices in is equal to:

[0018] Consider adding an arbitrary direction perturbation ΔW to the parameter W of a continuous piecewise linear function f. When the perturbation is small enough, each linear segment f in f(W+ΔW) i (W+ΔW) is still responsible for the sample area corresponding to the previous one Because f i (W) is the data subset D i The best linear fit corresponding to The empirical risk R of the region i (W+ΔW)>R i (W). The overall empirical risk of dataset D From this we can obtain: R(W+ΔW)>R(W), which means that the continuous linear piecewise function f(W) currently obtained is a local optimal solution.

[0019] 5. Determine the neural network structure and parameters. According to the properties of the ReLU activation function, it can be nested to represent the continuous piecewise linear function of the max-min form in 4. The formula for solving the maximum and minimum values ​​of two constants using the ReLU activation function is as follows:

[0020] max(a, b)=σ(σ(a)+σ(ba)), a, b≥0

[0021] min(a, b)=σ(σ(b)-σ(ba)), a, b≥0

[0022] Where a and b are two arbitrary constants, satisfying a, b ≥ 0, and σ represents the ReLU activation function.

[0023] When using the ReLU neural network module to represent the max and min functions, the network architecture is the same, both are single hidden layer neural networks, with two neurons in the input layer, two neurons in the hidden layer, and one neuron in the output layer. The only difference is the network parameters.

[0024] For the max function, the two neurons in the input layer represent two constants a and b respectively, the parameters corresponding to the two neurons in the hidden layer are (1, 0) and (-1, 1), and the neurons are activated by ReLU. The outputs of the two neurons correspond to σ(a) and σ(ba) in the formula respectively; the neuron parameters of the output layer are (1, 1), and the neurons are activated by ReLU. The output corresponds to σ(σ(a)+σ(ba)) in the formula, that is, the output of this neural network module is max(a, b).

[0025] For the min function, its input layer is consistent with the max function. The parameters corresponding to the two neurons in the hidden layer are (-1, 1) and (0, 1), and the neurons are activated by ReLU. The outputs of the two neurons correspond to σ(ba) and σ(b) in the formula respectively; the neuron parameters of the output layer are (-1, 1), and the neurons are activated by ReLU. The output corresponds to σ(σ(b)-σ(ba)) in the formula, that is, the output of this neural network module is min(a, b).

[0026] The above formula can also be used to find max(f1, f2) or min(f1, f2) of a continuous piecewise linear function formed by two functions f1 and f2 within a specific interval. However, before solving, add a sufficiently large positive number Q to each function to ensure that the values ​​of f1+Q and f2+Q within the interval are both ≥ 0. Subtract Q from the result after the calculation to obtain max(f1, f2) or min(f1, f2).

[0027] Specifically, for the constructed deep ReLU neural network, the output of the first hidden layer is each linear segment f in the continuous piecewise linear function f(W) i (W). That is, the number of neurons in the first hidden layer is equal to the subset D divided by the data set D i The number of (including the number of blank areas where the auxiliary linear segments are located), the parameters of each neuron correspond to D i The optimal linear fitting function f i (W). The remaining hidden layers in the network are constructed according to the ReLU module of max(f1, f2) and min(f1, f2), which converts the linear function f output by each neuron in the first hidden layer into i The continuous piecewise linear function is constructed in the max-min form, and the final output of the network is the continuous piecewise linear function f(W) constructed in 4. At this time, the parameters of the deep ReLU neural network correspond to a local optimal solution in the parameter space.

[0028] Beneficial effects

[0029] This paper proposes an image classification method that uses dataset partitioning to obtain a local optimal solution for a deep Reinforced Luminaire (ReLU) neural network without training. This method, based on the fundamental properties of continuous piecewise linear functions, is not constrained by datasets and has a wide range of applicability. It uses mathematical methods to construct a local optimal solution for the neural network. This method significantly addresses the computational resource and time constraints associated with iterative optimization of neural networks to a local optimal solution using methods such as gradient descent. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 , a complete flow chart of the method of the present invention;

[0031] Figure 2 , min and max functions of the ReLU network architecture;

[0032] Figure 3 ,In this experiment, the structure diagram of the deep ReLU neural network under any output dimension;

[0033] Figure 4 ,Network loss curve under random perturbation direction in this experiment; DETAILED DESCRIPTION

[0034] The experiment of an image classification method based on sample space partitioning to construct a local optimal solution of a deep ReLU neural network was run on Windows 10 system and implemented using Python 3.8.3 and PyTorch 1.7.0 framework.

[0035] The specific experimental steps are as follows:

[0036] Step 1: Divide the data set into N (1<N≤M) disjoint subsets, where M is the number of sample points in the data set.

[0037] This experiment uses the iris dataset. Correct results are obtained for any value of N within this range. In this experiment, N is the number of image categories in the dataset. Because this dataset has three different categories, the number of subsets N is set to 3. The dataset is divided into three mutually exclusive subsets based on the values ​​of the first dimension of the data samples. The number of sample points in each subset is shown below:

[0038]

[0039] Among them, subset 2 and subset 4 are blank areas between two adjacent subsets, and auxiliary linear slices are subsequently constructed in this area.

[0040] Step 2: Solve the optimal linear fitting function corresponding to each subset.

[0041] A two-layer neural network was constructed to solve the optimal linear fit function for each subset. The network consisted of only an input layer and an output layer. Because the images in this dataset fall into three categories, the function to be solved required three output dimensions. The output layer was configured with three neurons, each outputting the result for one of these dimensions. Since this experiment involved image classification, the cross-entropy loss was chosen as the loss function. For each subset, the neural network did not use an activation function, and the network parameters were randomly initialized. A gradient descent algorithm was used to iterate the solution until the network loss stopped decreasing. At this point, the network parameters corresponded to the optimal linear fit function for the current subset. The optimal linear fit function for each subset was stored accordingly.

[0042] Step 3: Construct auxiliary linear patches in the blank areas between adjacent subsets

[0043] For every two adjacent data subsets, an auxiliary linear patch is constructed in the blank area between them, and the optimal linear fitting functions of the adjacent subsets are connected to ensure that the sample area responsible for each linear segment remains unchanged after the continuous piecewise linear function is constructed.

[0044] The specific construction method of the auxiliary linear piece is as follows:

[0045] (1) Let each subset D i The sample space where all sample points are located is The corresponding optimal linear hyperplane is Z i , the mathematical expression is: Z i :t=w i x+b i ,in express Any sample point in the region. i represents the weight of function t, b i Represents the bias of function t. Then the optimal linear hyperplane corresponding to two adjacent subsets is: Z i , Z i+1 .

[0046] (2) Project the data set onto the hyperplane at t = 0 and determine the hyperline L that completely divides the sample points of the two adjacent subsets. Since the subsets are divided according to the values ​​of the first dimension x1 of the data, the expression of the hyperline L can be set as:

[0047]

[0048] Among them, u is a constant, which can take any value in the interval of x1 dimension in the blank area between the two subsets without affecting the experimental results. In this experiment, the value of u is the midpoint of the interval. For example, D i Range in x1 dimension: [2.6, 3.7], D i+1The range of x1 is [3.9, 4.6], so the range of u is (3.7, 3.9). The value of u is determined to be the midpoint of the interval, that is, u = 3.8. The end of step 3 gives the specific value of u in this experiment.

[0049] (3) In the hyperplane Z i Find a straight line L on i , so that its projection on the hyperplane of f = 0 is L. Line L i The expression is:

[0050]

[0051] (4) Obtain the super straight line L i The plane beam equation is:

[0052] λ(u-x1)+μ(w i x+b i -f)=0, where λ and μ are arbitrary real numbers.

[0053] This equation represents the line L i In particular, let λ = 1, then the above equation means that except for the plane f = w i x+b i L i The plane beam equation.

[0054] (5) Find the hyperplane represented by the plane beam equation and Z i+1 Intersecting super lines L i+1 Equation, straight line L i+1 The expression is:

[0055]

[0056] (6) L i+1 Projected onto the hyperplane at t=0, the expression after projection is:

[0057] u-x1+μ(w i x+b i -w i+1 xb i+1 )=0

[0058] (7) The projected hyperline in constraint (6) can divide the sample points in the two subsets, and the value range of the variable μ is solved:

[0059] (u-x1)+μ(w i x+b i -w i+1 xb i+1 )<0,x∈D i

[0060] (u-x1)+μ(w i x+b i -w i+1 xb i+1 )>0,x∈D i+1

[0061] (8) Any value of μ in the range of values ​​obtained in (7) will not affect the experimental results. Randomly select a constant v in the range of values, let μ = v, and substitute it into the plane beam equation to obtain two adjacent subsets D i and D i+1 Auxiliary hyperplane equation for the middle sample region:

[0062] (u-x1)+v(w i x+b i -y)=0

[0063] In this experiment, the value ranges of u and μ and the value ranges of v in the two auxiliary linear segments are as follows:

[0064] Auxiliary linear segment 1 Auxiliary linear segment 2 The value range of u (5.5，5.6) (6.1，6.2) The value of u 5.55 6.15 The value range of μ (-9.90，17.51) (-4.01，2.82) The value of v 3.81 -0.60

[0065] Step 4: Construct a continuous piecewise linear function using the max-min method.

[0066] The continuous piecewise linear function can be compactly represented by the expression of each linear segment through the max-min method. After all the auxiliary linear segments are obtained through step 3, they are combined with the optimal linear fit of each subset through the max-min method to form the target continuous piecewise linear function L * .

[0067] In this process, it is necessary to solve the min function for some linear slices in each interval to form a continuous piecewise linear function. The corresponding optimal linear fitting function or auxiliary linear function is f i , the linear slice f participating in the min function calculation in this interval j Need to meet The value of any point in is greater than or equal to f i . Due to f i With f j They are all linear functions, so we only need to calculate the value of the linear function at the endpoints of the interval. If f i ≥f i If both are true, it means that at any position in the interval, there is f j ≥f i , then record f j , participate in solving the min function of the current interval.

[0068] Step 5: Construct a deep ReLU neural network and determine the network parameters based on the continuous piecewise linear function obtained in step 4 to complete the image classification task.

[0069] The iris dataset is divided into three categories, so the network output is set to 3D. Since the max-min representation of continuous piecewise linear functions only applies to one-dimensional output, it is necessary to construct subnetworks for each of the three dimensions to obtain a one-dimensional continuous piecewise linear function. The architecture of the three subnetworks is exactly the same, with only the parameters of the first hidden layer being different. The subnetwork structure for each dimension is set as follows:

[0070] Based on the optimal linear fit functions corresponding to the three subsets solved in step 2 and the two auxiliary linear slices solved in step 3, a total of five linear slices ultimately constitute the continuous piecewise linear function. Therefore, the first hidden layer is configured with five neurons, each outputting one of the five linear slices. The neurons in the remaining hidden layers are used to represent the max and min functions, and the final output function is obtained using the max-min representation of the continuous piecewise linear function.

[0071] Finally, the three subnetworks are combined into a large deep ReLU neural network to obtain a continuous piecewise linear function in three dimensions. Because the calculation of the continuous piecewise linear function in each dimension is independent of each other, the output of the final network is exactly the same as the continuous piecewise linear function obtained in step 3.

[0072] The specific network structure used in this experiment is as follows:

[0073] For each sub-network, there are 8 hidden layers. The first hidden layer has 5 neurons, which output the 5 optimal linear fitting functions solved in steps 2 and 3 under the current dimension respectively. The optimal linear fitting function corresponding to the i-th (0<i≤5) interval is denoted as f i , all linear functions in the interval are denoted as {f j , j∈[1,5]};

[0074] The 2nd to 5th hidden layers of the network are used to calculate the min function corresponding to each interval. The second hidden layer has 7 neurons and the third hidden layer has 4 neurons. These two hidden layers represent 3 min functions according to the neural network architecture corresponding to the min function, which are used to solve the min function that satisfies f in the 1st to 4th intervals. j ≥f i The minimum linear function min(f j). The min functions corresponding to the first and second intervals are the same, both min(f1, f2), the min function corresponding to the third interval is min(f1, f3), and the min function corresponding to the fourth interval is min(f1, f4). Since the min function corresponding to the fifth interval is min(f1, f4, f5), it is necessary to output min(f1, f4) in the third hidden layer and then construct a neural network module of the min function in the fourth and fifth hidden layers to solve min(min(f1, f4), f5) to obtain it. Therefore, the fourth and fifth hidden layers each have 4 neurons, which include a min function network module for calculating the min function min(f1, f4, f5) corresponding to the fifth interval, and directly output the min functions min(f1, f2), min(f1, f3), and min(f1, f4) corresponding to the 1st to 4th intervals obtained previously.

[0075] The min function corresponding to the i-th interval is denoted as Ψ i , then the four neurons in the fifth hidden layer output Ψ1, Ψ3, Ψ4, and Ψ5 respectively, where Ψ1 represents the min function common to the two intervals 1 and 2. The sixth hidden layer to the output layer is used to calculate max(Ψ1, Ψ3, Ψ4, Ψ5), and finally obtain the target continuous piecewise linear function. The sixth hidden layer has 4 neurons, and the seventh hidden layer has 2 neurons. These two layers represent two max functions according to the neural network architecture corresponding to the max function, and obtain max(Ψ1, Ψ3) and max(Ψ4, Ψ5) respectively. The eighth hidden layer has 2 neurons, and the output layer has 1 neuron. These two layers represent a max function by constructing a network architecture of the max function, calculating max(max(Ψ1, Ψ3), max(Ψ4, Ψ5)) to obtain the continuous piecewise linear function max(Ψ1, Ψ3, Ψ4, Ψ5). The subnetwork is now complete.

[0076] Finally, the three sub-networks are combined into a large deep ReLU neural network by simply connecting the three sub-networks to the same input. The network architecture and parameters remain unchanged, and the final target continuous piecewise linear function can be obtained.

[0077] Step 6: Verify whether the deep ReLU neural network constructed in step 5 is a local optimal solution

[0078] Theoretical verification: Based on the max-min representation of the continuous piecewise linear function and the ReLU network representation of the max and min functions, it can be guaranteed that the output function of the deep ReLU neural network constructed in step 5 is consistent with the continuous piecewise linear function obtained in step 4. Let the corresponding parameter of the network at this time be W. Since the continuous piecewise linear function is composed of linear slices of each subset, the empirical risk of the current neural network is where R i Denotes the empirical risk of the i-th subset calculated by the optimal linear fit. Randomly add perturbations ΔW to the parameters of the network. When the perturbations are small enough, the function output by the neural network changes very little, and the sample area responsible for each linear segment remains unchanged. Let R′ denote the empirical risk after adding perturbations. Since R i The corresponding linear slice is already the optimal linear fit of the corresponding sample space, so R′ i >R i , This shows that adding perturbations to the network parameters will definitely increase the empirical risk, which theoretically proves that the constructed deep ReLU neural network is a local optimal solution.

[0079] Experimental verification: Each time, 30 perturbation dimensions are randomly selected, and the perturbation value of each dimension is randomly selected between (0, 0.02) to obtain the perturbation direction. The empirical risk of the network is calculated in the interval [-0.5.0.5] with a step size of 0.01, and the empirical risk change curve of the current perturbation direction is obtained.

[0080] The experiment was repeated 100 times, and all empirical risk curves showed an upward-opening parabola shape, and the empirical risk was minimum when the perturbation was 0. Since the perturbation direction was randomly selected, it can be experimentally demonstrated that the current network parameter settings are a local optimal solution.

[0081] From the above experiments, it can be seen that the method proposed in this patent is based on the max-min representation of continuous piecewise linear functions and the basic properties of deep ReLU neural networks. It can directly determine the structure and parameter values ​​of the network without training optimization, obtain local optimal solutions, and has been verified through theory and experiments. In addition, due to the diversity of data set partitioning methods, a large number of local optimal solutions can be obtained through this method. As long as the number of partitioned subsets is the same, the deep ReLU network architecture obtained is also the same. This also illustrates the universal existence of local minima in the parameter space of deep ReLU neural networks.

Claims

1. An image classification method based on sample space partitioning to construct a local optimal solution of a deep ReLU neural network, characterized in that: It includes the following steps: Step 1: Prepare an image classification dataset and divide the dataset into N disjoint subsets, where M is the number of sample points in the dataset, 1 <N≤M; Step 2: Construct a single hidden layer neural network and solve the optimal linear fitting function for each subset; Step 3: Construct an auxiliary linear function in the blank area between two adjacent subsets, and connect the optimal linear fitting functions of the two adjacent subsets to ensure that after forming a continuous piecewise linear function, each linear function is still responsible for the sample points in the corresponding subset range; Step 4: Use the max-min representation method to construct a continuous piecewise linear function from the optimal linear fitting function of each subset and the auxiliary linear function; Step 5: Build a deep ReLU neural network to output the continuous piecewise linear function in step 4 to complete the image classification task; Step 6: Verify whether the deep ReLU neural network parameters constructed in step 5 are a local optimal solution under the current network architecture.

2. According to the image classification method for constructing a local optimal solution of a deep ReLU neural network based on sample space partitioning according to claim 1, the construction of the auxiliary linear function implemented in step 3 is characterized by: Two adjacent subsets D i ,D i+1 , and the corresponding sample spaces are Since the sample points in the data set are all discretely distributed, and There must be a blank area P with no sample points between i ; Construct an auxiliary linear function in this area, because P i There are no sample points in , so the corresponding empirical risk The auxiliary linear function is a linear function that is only used to connect two adjacent subsets. Its specific construction method is as follows: 1) Let each subset D i The optimal linear hyperplane expression is: Z i :f=w i x+b i ,in express Any sample point in the region; w i represents the weight of function f, b i Represents the bias of function f; the optimal linear hyperplane corresponding to two adjacent subsets is: Z i 、Z i+1 ; 2) Project the data set onto the hyperplane with f = 0 and determine the hyperline L that completely divides the sample points of the two subsets. Since the subsets are divided according to the values ​​of the first dimension x1 of the data, the expression of the hyperline L can be set as: Where u is a constant, in the blank area P i Any value within the x1 dimension will not affect the experimental results; 3) In the hyperplane Z i Find a straight line L on i , so that its projection on the hyperplane of f = 0 is L; the straight line L i The expression is: 4) Get the super line L i The plane beam equation is: λ(u-x1)+μ(w i x+b i -f)=0, where λ,μ are any real numbers; This equation represents the line L i The set of all planes; let λ = 1, then the above plane bundle equation means except plane f = w i x+b i L i Plane beam equations of ; 5) Find the hyperplane represented by the plane beam equation and Z i+1 Intersecting super lines L i+1 The equation, L i+1 The expression is: 6) The super straight line L i+1 Projected onto the hyperplane of f=0, the expression after projection is: u-x1+μ(w i x+b i -w i+1 xb i+1 )=0 7) The projected hyperline in constraint 6) can divide the sample points in the two subsets and solve the value range of the variable μ: (u-x1)+μ(w i x+b i -w i+1 xb i+1 )<0,x∈D i (u-x1)+μ(w i x+b i -w i+1 xb i+1 )>0,x∈D i+1 8) Any value of μ in the range of values ​​obtained in 7) will not affect the experimental results; any constant v is selected in the range of values, let μ = v, and substitute it into the plane beam equation to obtain two adjacent subsets D i and D i+1 Auxiliary hyperplane equation of the middle sample area: (u-x1)+v(w i x+b i -f)=0.

3. The image classification method according to claim 1, wherein: The formula for solving the maximum and minimum values ​​of two constants through the ReLU activation function is as follows: max(a,b)=σ(σ(a)+σ(ba)),a,b≥0 min(a,b)=σ(σ(b)-σ(ba)),a,b≥0 Where a and b are two arbitrary constants, satisfying a and b ≥ 0, and σ represents the ReLU activation function; When using the ReLU neural network module to represent the max and min functions, the network architecture is the same, both are single hidden layer neural networks, with two neurons in the input layer, two neurons in the hidden layer, and one neuron in the output layer. The only difference is the network parameters. For the max function, the two neurons in the input layer represent two constants a and b respectively. The parameters corresponding to the two neurons in the hidden layer are (1,0) and (-1,1), and the neurons are activated by ReLU. The outputs of the two neurons correspond to σ(a) and σ(ba) in the formula respectively. The neuron parameters of the output layer are (1,1), and the neurons are activated by ReLU. The output corresponds to σ(σ(a)+σ(ba)) in the formula. That is, the output of this neural network module is max(a,b). For the min function, its input layer is consistent with the max function. The parameters corresponding to the two neurons in the hidden layer are (-1, 1) and (0, 1), and the neurons are activated by ReLU. The outputs of the two neurons correspond to σ(ba) and σ(b) in the formula respectively; the neuron parameters of the output layer are (-1, 1), and the neurons are activated by ReLU. The output corresponds to σ(σ(b)-σ(ba)) in the formula, that is, the output of this neural network module is min(a, b); When solving max(f1,f2) or min(f1,f2) of the continuous piecewise linear function formed by two functions f1 and f2 in a specific interval, the above formulas for maximum and minimum values ​​can also be used. However, a positive number Q must be added to each function before solving to ensure that the values ​​of f1+Q and f2+Q within the interval are both greater than or equal to 0. After the calculation is completed, Q is subtracted from the result to obtain max(f1,f2) or min(f1,f2).

4. According to the image classification method for constructing a local optimal solution of a deep ReLU neural network based on sample space partitioning according to claim 1, the method for constructing the deep ReLU network implemented in step 5 is characterized by: For all P linear functions {f j ,j∈[1,P]}, the corresponding continuous piecewise linear function L can be obtained by max-min method * ;Right now Among them, calculate Ψ i When f j It must be satisfied that in the i-th sample area The value at any point is greater than or equal to f i , f i for The corresponding optimal linear fitting function or auxiliary linear function in K i Indicates The linear function f that satisfies the corresponding conditions j The number of i Indicates In the function f, all functions that satisfy the conditions are j By solving the continuous piecewise linear function consisting of the minimum value; L * is the final target continuous piecewise linear function; When building a deep ReLU neural network, first output the optimal linear fitting function and all auxiliary linear functions of each subset in step 2 and step 3 in the first hidden layer. Then, according to the number of divided intervals, build a module corresponding to the min function through the ReLU network, and calculate the min function corresponding to each interval. The min function corresponding to the i-th interval is denoted as Ψ i ; After obtaining the min functions corresponding to all intervals, the module corresponding to the max function is constructed through the ReLU network, and the maximum value of all max functions is solved. The final output of the network is the continuous piecewise linear function obtained in step 4; The above max-min representation method of the continuous piecewise linear function is only applicable to one-dimensional output; for the continuous piecewise linear function with n-dimensional output, since each dimension is independent of each other, a continuous piecewise linear function can be constructed for each dimension; these continuous piecewise linear functions are all represented by the above-mentioned deep ReLU neural network, and finally n such sub-networks are combined into a larger deep ReLU neural network, simply by connecting the n sub-networks to the same input; the network architecture and parameters remain unchanged, and the final continuous piecewise linear function with n-dimensional output can be obtained.

Images