A method for determining decision factors of convolutional neural network and electronic device

Abstract

A method and electronic device for determining decision factors of a convolutional neural network, relating to the field of communication technology, determining the decision factors of a convolutional neural network, and facilitating users to understand the decision process of the convolutional neural network. The method comprises: acquiring an input image, and identifying a convolutional neural network model of the input image; determining a second weight matrix corresponding to each convolutional layer according to a convolution kernel contained in each convolutional layer in the convolutional neural network model; determining a third weight matrix corresponding to each pooling layer according to a pooling window matrix contained in each pooling layer; determining a fourth weight matrix corresponding to each fully connected layer according to an activation function contained in each fully connected layer; determining a first weight matrix according to these weight matrices; and determining one or more pixels in the input image corresponding to the positions of each element in the first weight matrix according to the weight values ​​of each element in the first weight matrix as decision factors of the convolutional neural network model.

Description

Technical Field

[0001] The present application relates to the technical field of neural network, and in particular to a method and electronic device for determining decision factors of a convolutional neural network. Background Art

[0002] Convolutional neural networks (CNN) are a type of feedforward neural networks that include convolutional calculations and have a deep structure. They are one of the representative algorithms of deep learning. With the development of deep learning and artificial intelligence, convolutional neural networks have developed rapidly and have been widely used in computer vision, natural language processing and other fields.

[0003] The most well-known disadvantage of convolutional neural networks is their "black box" nature, which means that users do not know how and why the convolutional neural network comes up with certain outputs. For example, when a user inputs an image of a cat into a convolutional neural network and the convolutional neural network predicts that it is a car, it is difficult to understand why it makes this prediction. In other words, convolutional neural networks lack transparency and explainability, which will seriously affect the degree of trust users have in them, and thus limit the application and development of convolutional neural networks in real-world tasks (especially risk-sensitive tasks such as driverless driving, healthcare, financial tasks, etc.). Therefore, how to accurately explain the decision-making process and classification results of convolutional neural networks is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a method and electronic device for determining the decision factors of a convolutional neural network, which can determine the decision factors of the convolutional neural network, that is, which pixels of the input data play an important role in the decision conclusion, making it convenient for users to intuitively understand the decision process of the convolutional neural network.

[0005] In order to achieve the above objectives, this application can adopt the following solutions:

[0006] In a first aspect, a method for determining a decision factor of a convolutional neural network is provided, comprising:

[0007] An input image is obtained, and a convolutional neural network model for identifying the input image is provided, wherein the convolutional neural network model includes at least one convolutional layer, at least one pooling layer, and at least one fully connected layer; a second weight matrix corresponding to each convolutional layer is determined according to one or more convolution kernels contained in each convolutional layer in at least one convolutional layer; a third weight matrix corresponding to each pooling layer is determined according to a pooling window matrix contained in each pooling layer in at least one pooling layer; a fourth weight matrix corresponding to each fully connected layer is determined according to an activation function contained in each fully connected layer in at least one fully connected layer; a first weight matrix is ​​determined according to the second weight matrix, the third weight matrix, and the fourth weight matrix, wherein the first weight matrix has the same size as the pixel size of the input image; one or more pixels in the input image corresponding to the positions of each element in the first weight matrix are determined according to the weight values ​​of each element in the first weight matrix as decision factors of the convolutional neural network model.

[0008] It can be seen that the present application converts the decision-making process of a complex convolutional neural network into a linear expression, wherein the weight value of the weight matrix of the linear expression can reflect the degree of influence of the input data on the decision-making process, making it convenient for users to intuitively understand the decision-making process of the convolutional neural network.

[0009] In one possible implementation, a second weight matrix corresponding to each convolution layer is determined according to one or more convolution kernels contained in each convolution layer in at least one convolution layer, including: determining the width of the second weight matrix corresponding to each convolution layer according to the input data of each convolution layer; determining the height of the second weight matrix corresponding to each convolution layer according to the receptive field and convolution step of one or more convolution kernels of each convolution layer, and the size of the input data of each convolution layer.

[0010] In one possible implementation, the input data of the first convolutional layer in at least one convolutional layer is a multi-channel matrix of m1×n1×r1, where m1×n1 is the size of the channel matrix, r1 is the number of channel matrices, and includes k convolution kernels, the receptive field of the k convolution kernels is p1×q1, the convolution step size is s1, and the width of the second weight matrix corresponding to the first convolutional layer is Wc width =m1×n1×r1; the height of the second weight matrix corresponding to the first convolutional layer is

[0011] In one possible implementation, the second weight matrix corresponding to each convolution layer is determined according to one or more convolution kernels contained in each convolution layer in at least one convolution layer, specifically: the value of each element in the second weight matrix corresponding to each convolution layer is initialized to zero; according to the weight values ​​of one or more convolution kernels contained in each convolution layer, the order of the channel matrix in the input data, the arrangement order of the elements in the channel matrix and the convolution calculation order, the value of each element in the second weight matrix corresponding to each convolution layer is set.

[0012] In one possible implementation, the third weight matrix corresponding to each pooling layer is determined according to the size of the pooling window contained in each pooling layer in at least one pooling layer, including: determining the width of the third weight matrix corresponding to each pooling layer according to the input data of each pooling layer; determining the height of the third weight matrix corresponding to each pooling layer according to the pooling window matrix and the pooling step size of each pooling layer, and the input data of each pooling layer.

[0013] In one possible implementation, the input data of the first pooling layer in at least one pooling layer is a multi-channel matrix of m2×n2×r2, m1×n1 is the size of the channel matrix, r1 is the number of channel matrices, and the size of the pooling window is p2×q2, the pooling step size is s2, and the width of the third weight matrix corresponding to the first pooling layer is Wp width =m2×n2×r2; the height of the third weight matrix corresponding to the first pooling layer is

[0014] In one possible implementation, the third weight matrix corresponding to each pooling layer is determined according to the pooling window included in each pooling layer in at least one pooling layer, specifically: the pooling result corresponding to the input data of each pooling layer is calculated using the pooling window matrix included in each pooling layer and the pooling method adopted by each pooling layer; according to the pooling result corresponding to the input data of each pooling layer, the values ​​of each element in the third weight matrix corresponding to each pooling layer are set according to the order of the channel matrix, the arrangement order of the elements in the channel matrix and the pooling calculation order.

[0015] In one possible implementation, the first fully connected layer in at least one fully connected layer includes a linear activation function and a fifth weight matrix, the fifth weight matrix is ​​the weight matrix of the first fully connected layer, and the fourth weight matrix corresponding to each fully connected layer is determined according to the activation function included in each fully connected layer in at least one fully connected layer, including: representing the linear activation function included in the first fully connected layer as a first piecewise linear function, and the multiple activation slopes included in the first piecewise linear function form a first activation slope matrix; multiplying the first activation slope matrix by the fifth weight matrix to obtain the fourth weight matrix corresponding to the first fully connected layer.

[0016] In one possible implementation, the second fully connected layer in at least one fully connected layer includes a nonlinear activation function and a sixth weight matrix, and the sixth weight matrix is ​​the weight matrix of the second fully connected layer; according to the activation function included in each fully connected layer in at least one fully connected layer, the fourth weight matrix corresponding to each fully connected layer is determined, including: according to the input data of the second fully connected layer, the nonlinear activation function included, and the sixth weight matrix, the output data of the second fully connected layer is calculated; according to the input data and output data of the second fully connected layer, a linear fully connected neural network is trained; the linear activation function in the trained linear fully connected neural network is represented as a second piecewise linear function; the multiple activation slopes included in the second piecewise linear function form a second activation slope matrix, and the trained fully connected neural network includes a seventh weight matrix; the second activation slope matrix is ​​multiplied by the seventh weight matrix to obtain the fourth weight matrix corresponding to the second fully connected layer.

[0017] In a second aspect, a method for determining decision factors of a convolutional neural network is provided, comprising: obtaining an input image, and identifying a convolutional neural network model of the input image, the convolutional neural network model comprising at least one convolutional layer and at least one fully connected layer; determining a second weight matrix corresponding to each convolutional layer according to one or more convolution kernels contained in each convolutional layer in at least one convolutional layer; determining a third weight matrix corresponding to each fully connected layer according to an activation function contained in each fully connected layer in at least one fully connected layer; determining a first weight matrix according to the second weight matrix and the third weight matrix, the first weight matrix having the same size as a pixel size of the input image; determining one or more pixels in the input image corresponding to the positions of each element in the first weight matrix according to the weight values ​​of each element in the first weight matrix as decision factors of the convolutional neural network model.

[0018] In one possible implementation, a second weight matrix corresponding to each convolution layer is determined according to one or more convolution kernels contained in each convolution layer in at least one convolution layer, including: determining the width of the second weight matrix corresponding to each convolution layer according to the input data of each convolution layer; determining the height of the second weight matrix corresponding to each convolution layer according to the receptive field and convolution step of one or more convolution kernels of each convolution layer, and the size of the input data of each convolution layer.

[0019] In one possible implementation, the input data of the first convolutional layer in at least one convolutional layer is a multi-channel matrix of m1×n1×r1, where m1×n1 is the size of the channel matrix, r1 is the number of channel matrices, and includes k convolution kernels, the receptive field of the k convolution kernels is p1×q1, the convolution step size is s1, and the width of the second weight matrix corresponding to the first convolutional layer is Wc width =m1×n1×r1; the height of the second weight matrix corresponding to the first convolutional layer is

[0020] In one possible implementation, the second weight matrix corresponding to each convolution layer is determined according to one or more convolution kernels contained in each convolution layer in at least one convolution layer, specifically: the value of each element in the second weight matrix corresponding to each convolution layer is initialized to zero; according to the weight values ​​of one or more convolution kernels contained in each convolution layer, the order of the channel matrix in the input data, the arrangement order of the elements in the channel matrix and the convolution calculation order, the value of each element in the second weight matrix corresponding to each convolution layer is set.

[0021] In one possible implementation, the first fully connected layer in at least one fully connected layer includes a linear activation function and a fourth weight matrix, the fourth weight matrix is ​​the weight matrix of the first fully connected layer, and the third weight matrix corresponding to each fully connected layer is determined according to the activation function included in each fully connected layer in at least one fully connected layer, including: representing the linear activation function included in the first fully connected layer as a first piecewise linear function, and the multiple activation slopes included in the first piecewise linear function form a first activation slope matrix; multiplying the first activation slope matrix by the fourth weight matrix to obtain the third weight matrix corresponding to the first fully connected layer.

[0022] In one possible implementation, the second fully connected layer in at least one fully connected layer includes a nonlinear activation function and a fifth weight matrix, and the fifth weight matrix is ​​the weight matrix of the second fully connected layer; according to the activation function included in each fully connected layer in at least one fully connected layer, the third weight matrix corresponding to each fully connected layer is determined, including: according to the input data of the second fully connected layer, the nonlinear activation function included, and the fifth weight matrix, the output data of the second fully connected layer is calculated; according to the input data and output data of the second fully connected layer, a linear fully connected neural network is trained; the linear activation function in the trained linear fully connected neural network is expressed as a second piecewise linear function; the multiple activation slopes included in the second piecewise linear function form a second activation slope matrix, and the trained fully connected neural network includes a sixth weight matrix; the second activation slope matrix is ​​multiplied by the sixth weight matrix to obtain the third weight matrix corresponding to the second fully connected layer.

[0023] In a third aspect, a device for determining a decision factor of a convolutional neural network is provided, comprising: an acquisition unit for acquiring an input image and identifying a convolutional neural network model of the input image, the convolutional neural network model comprising at least one convolutional layer, at least one pooling layer, and at least one fully connected layer; a determination unit for determining a second weight matrix corresponding to each convolutional layer according to one or more convolution kernels contained in each convolutional layer in at least one convolutional layer; determining a third weight matrix corresponding to each pooling layer according to a pooling window matrix contained in each pooling layer in at least one pooling layer; determining a fourth weight matrix corresponding to each fully connected layer according to an activation function contained in each fully connected layer in at least one fully connected layer; the determination unit is further used to determine a first weight matrix according to the second weight matrix, the third weight matrix and the fourth weight matrix, the first weight matrix having the same size as the pixel size of the input image; the determination unit is further used to determine one or more pixels in the input image corresponding to the positions of each element in the first weight matrix according to the weight values ​​of each element in the first weight matrix, as decision factors of the convolutional neural network model.

[0024] In one possible implementation, a second weight matrix corresponding to each convolution layer is determined according to one or more convolution kernels contained in each convolution layer in at least one convolution layer, including: determining the width of the second weight matrix corresponding to each convolution layer according to the input data of each convolution layer; determining the height of the second weight matrix corresponding to each convolution layer according to the receptive field and convolution step of one or more convolution kernels of each convolution layer, and the size of the input data of each convolution layer.

[0025] In one possible implementation, the input data of the first convolutional layer in at least one convolutional layer is a multi-channel matrix of m1×n1×r1, where m1×n1 is the size of the channel matrix, r1 is the number of channel matrices, and includes k convolution kernels, the receptive field of the k convolution kernels is p1×q1, the convolution step size is s1, and the width of the second weight matrix corresponding to the first convolutional layer is Wc width =m1×n1×r1; the height of the second weight matrix corresponding to the first convolutional layer is

[0026] In one possible implementation, the second weight matrix corresponding to each convolution layer is determined according to one or more convolution kernels contained in each convolution layer in at least one convolution layer, specifically: the value of each element in the second weight matrix corresponding to each convolution layer is initialized to zero; according to the weight values ​​of one or more convolution kernels contained in each convolution layer, the order of the channel matrix in the input data, the arrangement order of the elements in the channel matrix and the convolution calculation order, the value of each element in the second weight matrix corresponding to each convolution layer is set.

[0027] In one possible implementation, the third weight matrix corresponding to each pooling layer is determined according to the size of the pooling window contained in each pooling layer in at least one pooling layer, including: determining the width of the third weight matrix corresponding to each pooling layer according to the input data of each pooling layer; determining the height of the third weight matrix corresponding to each pooling layer according to the pooling window matrix and the pooling step size of each pooling layer, and the input data of each pooling layer.

[0028] In one possible implementation, the input data of the first pooling layer in at least one pooling layer is a multi-channel matrix of m2×n2×r2, m1×n1 is the size of the channel matrix, r1 is the number of channel matrices, and the size of the pooling window is p2×q2, the pooling step size is s2, and the width of the third weight matrix corresponding to the first pooling layer is Wp width =m2×n2×r2; the height of the third weight matrix corresponding to the first pooling layer is

[0029] In one possible implementation, the third weight matrix corresponding to each pooling layer is determined according to the pooling window included in each pooling layer in at least one pooling layer, specifically: the pooling result corresponding to the input data of each pooling layer is calculated using the pooling window matrix included in each pooling layer and the pooling method adopted by each pooling layer; according to the pooling result corresponding to the input data of each pooling layer, the values ​​of each element in the third weight matrix corresponding to each pooling layer are set according to the order of the channel matrix, the arrangement order of the elements in the channel matrix and the pooling calculation order.

[0030] In one possible implementation, the first fully connected layer in at least one fully connected layer includes a linear activation function and a fifth weight matrix, the fifth weight matrix is ​​the weight matrix of the first fully connected layer, and the fourth weight matrix corresponding to each fully connected layer is determined according to the activation function included in each fully connected layer in at least one fully connected layer, including: representing the linear activation function included in the first fully connected layer as a first piecewise linear function, and the multiple activation slopes included in the first piecewise linear function form a first activation slope matrix; multiplying the first activation slope matrix by the fifth weight matrix to obtain the fourth weight matrix corresponding to the first fully connected layer.

[0031] In one possible implementation, the second fully connected layer in at least one fully connected layer includes a nonlinear activation function and a sixth weight matrix, and the sixth weight matrix is ​​the weight matrix of the second fully connected layer; according to the activation function included in each fully connected layer in at least one fully connected layer, the fourth weight matrix corresponding to each fully connected layer is determined, including: according to the input data of the second fully connected layer, the nonlinear activation function included, and the sixth weight matrix, the output data of the second fully connected layer is calculated; according to the input data and output data of the second fully connected layer, a linear fully connected neural network is trained; the linear activation function in the trained linear fully connected neural network is represented as a second piecewise linear function; the multiple activation slopes included in the second piecewise linear function form a second activation slope matrix, and the trained fully connected neural network includes a seventh weight matrix; the second activation slope matrix is ​​multiplied by the seventh weight matrix to obtain the fourth weight matrix corresponding to the second fully connected layer.

[0032] In the fourth aspect, a device for determining a decision factor of a convolutional neural network includes: an acquisition unit for acquiring an input image and identifying a convolutional neural network model of the input image, the convolutional neural network model including at least one convolutional layer and at least one fully connected layer; a determination unit for determining a second weight matrix corresponding to each convolutional layer according to one or more convolution kernels contained in each convolutional layer in at least one convolutional layer; determining a third weight matrix corresponding to each fully connected layer according to an activation function contained in each fully connected layer in at least one fully connected layer; the determination unit is also used to determine a first weight matrix according to the second weight matrix and the third weight matrix, the first weight matrix having the same size as a pixel size of the input image; the determination unit is also used to determine one or more pixels in the input image corresponding to the positions of each element in the first weight matrix according to the weight values ​​of each element in the first weight matrix, as decision factors of the convolutional neural network model.

[0033] In one possible implementation, a second weight matrix corresponding to each convolution layer is determined according to one or more convolution kernels contained in each convolution layer in at least one convolution layer, including: determining the width of the second weight matrix corresponding to each convolution layer according to the input data of each convolution layer; determining the height of the second weight matrix corresponding to each convolution layer according to the receptive field and convolution step of one or more convolution kernels of each convolution layer, and the size of the input data of each convolution layer.

[0034] In one possible implementation, the input data of the first convolutional layer in at least one convolutional layer is a multi-channel matrix of m1×n1×r1, where m1×n1 is the size of the channel matrix, r1 is the number of channel matrices, and includes k convolution kernels, the receptive field of the k convolution kernels is p1×q1, the convolution step size is s1, and the width of the second weight matrix corresponding to the first convolutional layer is Wc width=m1×n1×r1; the height of the second weight matrix corresponding to the first convolutional layer is

[0035] In one possible implementation, the second weight matrix corresponding to each convolution layer is determined according to one or more convolution kernels contained in each convolution layer in at least one convolution layer, specifically: the value of each element in the second weight matrix corresponding to each convolution layer is initialized to zero; according to the weight values ​​of one or more convolution kernels contained in each convolution layer, the order of the channel matrix in the input data, the arrangement order of the elements in the channel matrix and the convolution calculation order, the value of each element in the second weight matrix corresponding to each convolution layer is set.

[0036] In one possible implementation, the first fully connected layer in at least one fully connected layer includes a linear activation function and a fourth weight matrix, the fourth weight matrix is ​​the weight matrix of the first fully connected layer, and the third weight matrix corresponding to each fully connected layer is determined according to the activation function included in each fully connected layer in at least one fully connected layer, including: representing the linear activation function included in the first fully connected layer as a first piecewise linear function, and the multiple activation slopes included in the first piecewise linear function form a first activation slope matrix; multiplying the first activation slope matrix by the fourth weight matrix to obtain the third weight matrix corresponding to the first fully connected layer.

[0037] In one possible implementation, the second fully connected layer in at least one fully connected layer includes a nonlinear activation function and a fifth weight matrix, and the fifth weight matrix is ​​the weight matrix of the second fully connected layer; according to the activation function included in each fully connected layer in at least one fully connected layer, the third weight matrix corresponding to each fully connected layer is determined, including: according to the input data of the second fully connected layer, the nonlinear activation function included, and the fifth weight matrix, the output data of the second fully connected layer is calculated; according to the input data and output data of the second fully connected layer, a linear fully connected neural network is trained; the linear activation function in the trained linear fully connected neural network is expressed as a second piecewise linear function; the multiple activation slopes included in the second piecewise linear function form a second activation slope matrix, and the trained fully connected neural network includes a sixth weight matrix; the second activation slope matrix is ​​multiplied by the sixth weight matrix to obtain the third weight matrix corresponding to the second fully connected layer.

[0038] In a fifth aspect, an electronic device comprises: a processor, a memory and a touch screen, wherein the memory and the touch screen are coupled to the processor, the memory is used to store computer program code, and the computer program code comprises computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes the method as described in the first aspect and the second aspect and any possible implementation thereof.

[0039] In the sixth aspect, a computer storage medium comprises computer instructions. When the computer instructions are executed on a terminal, the terminal executes the method as described in the first aspect and the second aspect and any possible implementation manner thereof.

[0040] A seventh aspect is a computer program product. When the computer program product is run on a computer, it enables the computer to execute the method as described in the first aspect and the second aspect and any possible implementation manner thereof.

[0041] In an eighth aspect, a chip comprises a processor. When the processor executes instructions, the processor executes the method as described in the first aspect and the second aspect and any possible implementation manner thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of the structure of a convolutional neural network model provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a convolution calculation method provided in an embodiment of the present application;

[0045] Figure 4A A method for constructing a convolutional layer equivalent weight matrix provided in an embodiment of the present application is shown in FIG. Figure 1 ;

[0046] Figure 4B A method for constructing a convolutional layer equivalent weight matrix provided in an embodiment of the present application is shown in FIG. Figure 2 ;

[0047] Figure 5A A method for constructing a convolutional layer equivalent weight matrix provided in an embodiment of the present application is shown in FIG. Figure 3 ;

[0048] Figure 5B Schematic diagram 4 of a method for constructing a convolutional layer equivalent weight matrix provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of a pooling process provided in an embodiment of the present application;

[0050] Figure 7 A method for constructing an equivalent weight matrix of a pooling layer provided in an embodiment of the present application is shown in FIG. Figure 1 ;

[0051] Fig. 8A A schematic diagram of a process for determining a decision factor of a convolutional neural network provided in an embodiment of the present application Figure 1 ;

[0052] Figure 8B A schematic diagram of a process for determining a decision factor of a convolutional neural network provided in an embodiment of the present application Figure 2 ;

[0053] Fig. 9 A schematic diagram of the structure of a device provided in an embodiment of the present application;

[0054] Fig.10 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" is two or more. In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0056] In order to explain the decision-making process and classification results of the convolutional neural network, the embodiment of the present application deeply studies the internal working mechanism of the convolutional neural network, combines the convolution process and the pooling process, and the forward propagation mechanism of the neural network, and proposes a method for determining the decision factors of the convolutional neural network. In this method, for each input instance, and a specific convolutional neural network model, the decision-making process of the convolutional neural network is converted into an equivalent linear expression. Based on the weight matrix in the linear expression, a decision factor graph is generated. It can be understood that the numerical value of each element in the weight matrix can be interpreted as the degree of influence of the input corresponding to each element on the decision-making process. That is, the larger the value of the element, the greater the influence of the input corresponding to the element on the decision-making process, and the greater the contribution to the classification result obtained. The input corresponding to the element with a value greater than a certain threshold in the weight matrix can be considered as a factor that plays a key role in the decision-making process, which can be called a key factor.

[0057] In other words, users can understand the influence of each input on the decision-making process of the convolutional neural network and the contribution to the classification results through the decision factor diagram. This will help improve the transparency of the convolutional neural network and establish trust with users.

[0058] In addition, the reliability and security of the convolutional neural network model can also be analyzed through the decision factor graph to reduce the potential risks of the convolutional neural network model in actual deployment.

[0059] The method provided in the embodiment of the present application can be applied to a convolution device of a convolutional neural network, which may be part or all of an electronic device using a convolutional neural network, thereby realizing specific functions.

[0060] The above-mentioned specific functions can be, for example, computer vision applications such as image classification, object recognition, action recognition, pose estimation, neural style transfer, etc., or natural language processing (NLP) applications.

[0061] For example, convolutional neural network algorithms can be used in the camera function of mobile phones. When taking pictures, the mobile phone can run the corresponding convolutional neural network algorithm based on the image captured by the viewfinder, identify the current photographed object as a person, scene, object, etc., and set different photographic parameters for different photographed objects, such as adding different filters. Alternatively, the mobile phone can also run the corresponding convolutional neural network algorithm to identify the specific food of the current photographed object, estimate the size and weight of the food, and give different calorie information for different foods.

[0062] Convolutional neural network algorithms can also be used in the automatic driving function of cars. For example, the car's automatic navigation system can run the corresponding convolutional neural network algorithm to identify external weather conditions and automatically turn on the wiper function when it is determined to be rainy, snowy, foggy and other weather conditions. Another example: the car's automatic navigation system can also run the corresponding convolutional neural network to identify road traffic markings and realize functions such as lane recognition and assisted driving control.

[0063] Convolutional neural network algorithms can also be used in the auxiliary diagnosis function of medical equipment. For example, medical equipment can run the corresponding convolutional neural network algorithm to analyze the patient's medical imaging data and achieve functions such as rapid lesion location.

[0064] Exemplarily, the electronic device in the present application may be a mobile phone, a tablet computer, a personal computer (PC), a personal digital assistant (PDA), a smart watch, a netbook, a wearable electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a vehicle-mounted device, a smart car, a smart speaker, a robot, a medical device, etc. The present application does not impose any special restrictions on the specific form of the electronic device.

[0065] Figure 1 A schematic structural diagram of an electronic device 100 is shown.

[0066] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0067] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0068] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0069] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0070] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0071] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0072] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0073] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0074] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0075] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0076] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0077] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0078] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0079] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0080] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0081] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0082] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0083] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0084] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0085] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0086] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0087] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0088] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0089] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0090] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0091] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0092] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0093] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0094] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0095] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0096] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0097] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0098] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0099] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0100] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0101] The technical solutions involved in the following embodiments can all be implemented in the electronic device 100 having the above hardware architecture.

[0102] In order to facilitate understanding of the technical solution provided in the embodiments of the present application, the structure of the convolutional neural network is first described.

[0103] Generally speaking, a convolutional neural network consists of an input layer, a hidden layer, and an output layer. Figure 2 As shown in FIG. 1 , a schematic diagram of the structure of a convolutional neural network is shown.

[0104] Among them, the input layer of the convolutional neural network can process multi-dimensional data. Taking image processing as an example, the input layer can receive the pixel values ​​(three-dimensional array) of the image, that is, the two-dimensional pixel points on the plane and the values ​​of the RGB channels.

[0105] The hidden layer of a convolutional neural network includes one or more convolutional layers, one or more pooling layers, and one or more fully-connected layers. Usually, one or more convolutional layers are followed by a pooling layer. The structure and working principle of each layer in the hidden layer will be described in detail below, so we will not explain them here. In some examples, the hidden layer of a convolutional neural network may not include a pooling layer.

[0106] The output layer of the convolutional neural network has the same structure and working principle as the output of the traditional feedforward neural network. For example, for the convolutional neural network for image classification, the output layer uses a logistic function or a normalized exponential function (softmax function) to output classification labels, such as people, scenery, objects, etc. For the convolutional neural network for object recognition, the output layer can be designed to output the center coordinates, size, and classification of the object.

[0107] For example, Figure 2 As shown, it is a structural schematic diagram of a convolutional neural network model. The convolutional neural network model in this figure includes an input layer, m convolutional layers, m pooling layers, L fully connected layers and an output layer. Specifically, the output of the input layer is the input of the first convolutional layer (i.e., convolutional layer 1). Subsequent convolutional layers and pooling layers appear alternately, and the output of the previous layer is the input of this layer, and the output of this layer is the input of the next time. The output of the mth pooling layer (i.e., pooling layer m) is the input of the first fully connected layer (i.e., fully connected layer 1). The output of the Lth fully connected layer (i.e., fully connected layer L) is the input of the output layer. The output of the output layer is the output result of the convolutional neural network model. It should be noted that the embodiment of the present application does not limit the number of convolutional layers, the number of pooling layers, and the number of fully connected layers contained in the hidden layer in the convolutional neural network model, as well as the ordering between the convolutional layers, pooling layers, and fully connected layers. For example, the next layer of the input layer can also be a fully connected layer. For another example: a pooling layer can be connected after multiple convolutional layers. For another example: A convolutional neural network model may not include a pooling layer.

[0108] The technical solution provided in the embodiment of the present application is based on the specific structure of the convolutional neural network model, combined with the input of the input layer, and converts the calculation process (i.e., the decision-making process) of the convolutional neural network layer by layer into an equivalent linear expression, and explains the decision-making process through the equivalent linear expression. Therefore, the equivalent linear expression can also be called an explanation model.

[0109] The following describes the conversion of each layer in detail in combination with the specific structure and working mechanism of each layer.

[0110] First of all, it should be noted that a convolutional neural network model contains multiple convolutional layers, multiple pooling layers, and multiple fully connected layers, and the convolutional layers and pooling layers are connected alternately. The output of the previous layer is the input of this layer, and the output of this layer is the input of the next layer. Therefore, when introducing the explanation method of each layer below, the input and output data between each layer will not be associated. When explaining the entire convolutional neural network model at the end, the input and output data between each layer will be associated.

[0111] 1. Equivalent conversion of convolutional layer

[0112] The function of the convolution layer is to extract features from the input data. The convolution layer contains multiple convolution kernels (filters), each of which corresponds to multiple weight coefficients and a bias vector, similar to a neuron in a feedforward neural network. Each neuron in the convolution layer is connected to multiple neurons in a similar area in the previous layer. The size of the area depends on the size of the convolution kernel, also known as the "receptive field". When the convolution kernel is working, it will regularly scan the input data, perform matrix element multiplication on the input data within the receptive field, and superimpose the bias.

[0113] For example, the input data of a certain convolution layer is a multidimensional array, denoted as x, with a size of m1×n1×r1. For example, when the input data is image data, m1 and n1 are the height and width of the image, and r1 is the number of channels. For example, the number of channels of an RGB image is r1=3. This convolution layer contains k convolution kernels, and each convolution kernel also corresponds to a convolution stride. The convolution stride defines the distance between the positions of the convolution kernel when it scans the input data twice adjacently. For example: when the input data is image data and the convolution stride is s0, the next scan skips s0-1 pixels.

[0114] It should be noted that the receptive field and convolution step size of each convolution kernel in the same convolution layer are the same. In addition, the number of channels of the convolution kernel in the convolution layer is the same as the number of channels of the input data. Therefore, the convolution kernel here can be recorded as F i p1 ×q1×r1 , i = 1, 2, ..., k. And the convolution step length corresponding to each convolution kernel is denoted as s1.

[0115] In this application, the interpretation of the convolution layer is that the convolution calculation is converted into a linear expression for the input data of the convolution layer and the k convolution kernels contained in the convolution layer. In other words, for the specific input data and the model of the convolution layer, the convolution calculation is expressed as a mathematically equivalent linear expression of the form y=w×x+b. Among them, y is the output of the convolution layer, and w and b are the weights and biases equivalent to the convolution calculation, respectively. Since in the convolutional neural network, the input can be a multidimensional array, and the output is also a multidimensional array. Therefore, in the linear expression equivalent to the convolution calculation here, the weight w should be a matrix, and the bias b is also a matrix. In order to distinguish the weights and biases of subsequent pooling layers, fully connected layers, etc., the weight matrix of the determined convolution layer is recorded as Wc, and the bias matrix of the determined convolution layer is recorded as bc.

[0116] The following will specifically introduce how to convert the convolution calculation of the convolution layer into a linear expression, that is, how to determine the weight matrix Wc and the bias matrix bc.

[0117] (1) Expand the input multidimensional array to 1×n d The matrix is ​​denoted as x'. d =m1×n1×r1.

[0118] like Figure 3 As shown, the image data of 3×3×3 (m1=3, n1=3, r1=3) is taken as an example for explanation.

[0119] For example, the input image data includes data from three channels, namely channel 1 (also called R channel), channel 2 (also called G channel) and channel 3 (also called B channel). Each channel contains 3×3 elements. The elements contained in channel 1 are recorded as The elements contained in channel 2 are recorded as The elements contained in channel 3 are recorded as in, Figure 3 The various elements contained in channel 1 are shown in FIG. The processing of other channels can refer to the processing of the R channel and will not be repeated here.

[0120] It can be seen that the input image data is a multi-channel matrix. According to the channel order of the channel matrix and the arrangement order of the elements in the channel matrix, when the input image data is expanded, n d =m1×n1×r1=27. That is, the input image data is expanded into a 1×27 matrix

[0121] In some embodiments, the convolution layer may process the input data using a padding algorithm. That is, before the convolution kernel is calculated, the size of the input data is artificially increased by a padding algorithm to offset the effect of size shrinkage in the calculation. The padding algorithm may be, for example, zero padding, repeated boundary value padding, or other methods. For specific padding methods, reference may be made to the relevant technology in the art, which will not be described in detail here.

[0122] It should be noted that if the convolutional layer uses a padding algorithm, the input data needs to be padded first and then flattened when interpreting the convolutional layer.

[0123] (2) Construct the weight matrix Wc and bias matrix bc based on the expanded matrix x' and the convolution kernel contained in the convolution layer.

[0124] Exemplarily, the convolution sliding window S1 can be constructed based on the expanded matrix x' and the k convolution kernels contained in the convolution layer. In the present application, the convolution sliding window S1 is used to simulate the process of each convolution kernel regularly scanning the input data when working. Then, the weight matrix Wc and the bias matrix bc are constructed according to the convolution sliding window S1 combined with the calculation process of the convolution kernel. Then, the convolution calculation of the convolution layer can be equivalent to a linear expression. The following will explain the methods used in each process respectively.

[0125] a) Method for constructing convolution sliding window S1

[0126] Among them, the size of S1 is S1 width ×S1 height Among them, S1 width is the width of the convolution sliding window S1, S1 height is the height of the convolution sliding window S1, which can be calculated by formula 1 and formula 2 respectively, as follows:

[0127] S1 width =(q1-1)×(m1-p1)×r1+p1×q1×r1 (Formula 1)

[0128] S1 height =k (Formula 2)

[0129] It can be understood that each row in the convolution sliding window corresponds to a convolution kernel. Subsequently, the values ​​of each element in the convolution window will be determined by combining the order of parameters (i.e., weights and biases) used by each convolution kernel in calculation.

[0130] When the convolution layer includes one convolution kernel, or when the convolution layer includes multiple convolution kernels, the values ​​of each element in the convolution window can be determined according to the following method for each convolution kernel.

[0131] For example: Still Figure 3 As shown, the matrix expanded with input data And the convolution layer contains a convolution kernel F1 2×2×3 (For example, convolution kernel 1) gives an example.

[0132] It can be seen that in the convolution kernel, i=1, p1=2, q1=2, r1=3, and the convolution kernel F1 2×2×3 Contains a bias b1 and 3 channel weights, each of which contains 2×2 weights, as follows:

[0133] Channel 1 contains the weights Channel 2 contains weights of Channel 3 contains the weights Figure 3 Only the weight of channel 1 is shown in the figure. The processing of other channels can refer to the processing of channel 1 and will not be repeated here.

[0134] Then, according to the expanded matrix x' and the convolution kernel F1 2×2×3 , substituting the corresponding data into Formula 1 and Formula 2, it can be determined that the size of the convolution sliding window S1 is 1×15. And combined with the convolution kernel F1 2×2×3 Contains weight values ​​and step size s1, and F1 2×2×3 The order of the parameters used in the calculation is used to determine the convolution sliding window S1:

[0135] [a1 a5 a9 a2 a6 a10 0 0 0 a3 a7 a11 a4 a8 a12].

[0136] Another example: the matrix expanded with input data And the convolution layer contains a convolution kernel F1 2×3×3 Give an example.

[0137] It can be seen that in the convolution kernel, i=1, p1=2, q1=3, r1=3. And the convolution kernel F1 2×3×3 Contains a bias b1 and 3 channel weights, each of which contains 2×3 weights, as follows:

[0138] Channel 1 contains the weights Channel 2 contains weights of Channel 3 contains the weights

[0139] Then, according to the expanded matrix x' and the convolution kernel F1 2×3×3 , substituting the corresponding data into Formula 1 and Formula 2, it can be determined that the size of the convolution sliding window S1 is 1×24. And combined with the convolution kernel F1 2×3×3 Contains weight values ​​and step size s1, and F1 2×3×3The order of parameters is used in the calculation, and the convolution sliding window S1 is determined to be [a1, a7, a13, a2, a8, a14, 0, 0, 0, a3, a9, a15, a4, a10, a16, 0, 0, 0, a5, a11, a17, a6, a12, a18].

[0140] When the convolution layer contains multiple convolution kernels, the elements of the multiple convolution kernels in the convolution window S1 are determined according to the above method.

[0141] For example: Still Figure 3 As shown, the matrix expanded with input data And the convolution layer contains two convolution kernels, namely F1 2×2×3 (convolution kernel 1) and (Convolution kernel 2) Give an example.

[0142] Among them, the convolution kernel F1 2×2×3 The structure and determination method will not be explained again, and reference may be made to the above description.

[0143] Convolution Kernel Contains a bias b1 and 3 channel weights, each of which contains 2×3 weights, as follows:

[0144] Channel 1 contains the weights Channel 2 contains weights of Channel 3 contains the weights

[0145] Then, according to the expanded matrix x' and the convolution kernel F1 2×2×3 , Substituting the corresponding data into Formula 1 and Formula 2, we can determine that the size of the convolution sliding window S1 is 15×2. Determine F1 2×2×3 The element values ​​in the convolution window S1 can refer to the description above to determine The element value in the convolution window S1 is determined by F1 2×2×3 The method for the element value in the convolution window S1 is similar and will not be described in detail.

[0146] Then, the convolution sliding window S1 is finally determined as follows:

[0147]

[0148] It should be noted that, when calculating in the same convolution layer, the order of calculation of each convolution kernel and input data has no effect on the calculation result. Therefore, the present application does not limit the arrangement order of each convolution kernel in the same convolution layer in the convolution window. For example, in this example, the convolution sliding window S1 can also be as follows:

[0149]

[0150] It should be noted that when calculating other layers later, if there are calculations corresponding to the convolution kernels here, their order needs to be adjusted to be consistent with the order here.

[0151] b) Method for constructing weight matrix Wc based on convolution sliding window S1

[0152] First, construct a weight matrix Wc initialized to 0. That is, all elements in Wc are initially zero, and the values ​​of the corresponding positions in Wc will be reset through the sliding window S1. The size of Wc is Wc width ×Wc height .Wc width is the width of the weight matrix Wc, Wc height is the height of the weight matrix Wc. Specifically, the width of the weight matrix Wc can be determined according to the input data of the convolution layer. The height of the weight matrix Wc is determined according to the receptive field and convolution step of one or more convolution kernels contained in the convolution layer, as well as the input data.

[0153] For example, Wc width and Wc height They can be calculated by formula 3 and formula 4 respectively, as follows:

[0154] W width =m1×n1×r1 (Formula 3)

[0155]

[0156] Among them, mathematical symbols It indicates the operation of rounding down to an integer. It can be seen that the width of the weight matrix Wc is the same as the width of the matrix x' of the expanded input data. It can be understood as the number of times the convolution sliding window is reset.

[0157] It should be noted that the present application simulates the convolution calculation process of the convolution kernel by moving the convolution sliding window S1 in the weight matrix Wc and resetting the element values ​​in the weight matrix Wc with the element values ​​of the convolution sliding window S1. Because when the convolution kernel is scanning the input data (referring to the input data when it is not expanded), it moves according to the step size s1 of the convolution kernel. It can be understood that it moves in the matrix of the input data according to the matrix of the convolution kernel. In other words, when moving the convolution sliding window S1, it is also necessary to consider the position where the convolution kernel moves when scanning the input data and reset the weight matrix Wc.

[0158] Exemplarily, when the convolution kernel scans the input data (referring to the input data when it is not expanded) for convolution calculation, the position of the first weight of the convolution kernel in the input data is recorded as (e1, f1), indicating the position of the e1th row and the f1th column. The position of the first element in the convolution sliding window S1 in the weight matrix Wc is recorded as (v1, h1), indicating that the first element in the convolution sliding window S1 is located at the position of the v1th row and the h1th column in the weight matrix Wc. When the sliding window S1 resets the element value in the weight matrix Wc, these two positional relationships can refer to Formula 5 and Formula 6, as follows:

[0159] h1=(e1-1)×m1×r1×s1+(f1-1)×r1×s1+1 (Formula 5)

[0160] v1=(i'-1)×k+1 (Formula 6)

[0161] Wherein, i' represents the i'th time resetting the element value in the weight matrix Wc according to the convolution sliding window S1, and i' is an integer greater than or equal to 1. and

[0162] After each movement of the convolution sliding window S1, the value of the element on Wc covered by the convolution sliding window S1 is reset to the corresponding value in the convolution sliding window S1.

[0163] For example: See Figure 4A , still with the matrix expanded by the input data And the convolution layer contains a convolution kernel F1 2×2×3 (For example, convolution kernel 1) gives an example.

[0164] According to the method provided above, it can be determined that the convolution sliding window S1 is a 1×15 matrix, such as Figure 4A as shown in .

[0165] And, by substituting the parameters (m1, n1, r1) of the input data into Formula 3 and Formula 4, it can be determined that the width of the weight matrix Wc is 27, the height is 4, and the size is 4×27. Initialize the elements in the weight matrix Wc to 0, such as Figure 4A as shown in .

[0166] When the element values ​​in the weight matrix are reset for the first time, the elements in the 1st row and 1st column of the convolution kernel coincide with the 1st row and 1st column of the input data (the multidimensional array before flattening). At this time, i'=1, e1=1, f1=1. According to Formula 5 and Formula 6, it can be determined that the position where the convolution sliding window S1 should be located in the weight matrix Wc is (1, 1). That is, the elements in the 1st row and 1st column of the convolution sliding window S1 coincide with the elements in the 1st row and 1st column of the weight matrix Wc. And the values ​​of the elements that coincide with the position of the convolution sliding window S1 are reset to the values ​​of the elements in the corresponding positions in the convolution sliding window. That is, the weight matrix Wc after the first reset is obtained as follows: Figure 4A as shown in .

[0167] After the first reset, the convolution kernel moves according to the convolution step size s1 (taking s1=1 as an example), and the elements in the 1st row and 1st column of the convolution kernel coincide with the 1st row and 2nd column of the input data (the multidimensional array before flattening). This is the second reset, that is, i'=2, e1=1, f1=2. According to Formula 5 and Formula 6, it can be determined that the position of the convolution sliding window S1 in the weight matrix Wc should be (2, 4). That is, the elements in the 1st row and 1st column of the convolution sliding window S1 coincide with the elements in the 2nd row and 4th column of the weight matrix Wc. And the values ​​of the elements that coincide with the position of the convolution sliding window S1 are reset to the values ​​of the elements in the corresponding positions in the convolution sliding window. That is, the weight matrix Wc after the second reset (i'=2) is obtained as follows Figure 4B as shown in .

[0168] After the second reset, the convolution kernel continues to move according to the convolution step s1 (taking s1=1 as an example). At this time, the elements in the 1st row and 1st column of the convolution kernel coincide with the 2nd row and 1st column of the input data (the multidimensional array before flattening). This is the third reset, that is, i'=3, e1=2, f1=1. According to Formula 5 and Formula 6, it can be determined that the position of the convolution sliding window S1 in the weight matrix Wc should be (3, 10). That is, the elements in the 1st row and 1st column of the convolution sliding window S1 coincide with the elements in the 3rd row and 10th column of the weight matrix Wc. And the values ​​of the elements that coincide with the position of the convolution sliding window S1 are reset to the values ​​of the elements in the corresponding positions in the convolution sliding window. That is, the weight matrix Wc after the third reset is obtained as follows: Figure 4B as shown in .

[0169] By analogy, the weight matrix Wc after the fourth reset is obtained as follows: Figure 4B In this example, the reset of the weight matrix Wc is completed, that is, the weight matrix Wc after the fourth reset is recorded as the weight matrix of the convolution layer determined by this application.

[0170] Another example: Figure 5A As stated, Figure 3The matrix of the expanded input data shown is And the convolution layer contains two convolution kernels, namely F1 2×2×3 (convolution kernel 1) and (Convolution kernel 2) Give an example.

[0171] According to the method provided above, it can be determined that the convolution sliding window S1 is a 2×15 matrix, such as Figure 5A as shown in .

[0172] And, by substituting the parameters (m1, n1, r1) of the input data into Formula 3 and Formula 4, it can be determined that the width of the weight matrix Wc is 27, the height is 8, and the size is 8×27. Initialize the elements in the weight matrix Wc to 0, such as Figure 5A as shown in .

[0173] When the element values ​​in the weight matrix are reset for the first time (i'=1), the element in the first row and first column of the convolution kernel coincides with the first row and first column of the input data (the multidimensional array before flattening). At this time, the element in the first row and first column of the convolution sliding window S1 coincides with the element in the first row and first column of the weight matrix Wc. And the value of the element that coincides with the position of the convolution sliding window S1 is reset to the value of the element in the corresponding position in the convolution sliding window. That is, the weight matrix Wc after the first reset is obtained as follows: Figure 5B as shown in .

[0174] After the first reset, the convolution kernel moves according to the convolution step s1 (taking s1=1 as an example), and the elements in the 1st row and 1st column of the convolution kernel coincide with the 1st row and 2nd column of the input data (the multidimensional array before flattening). This is the second reset, that is, i'=2, e1=1, f1=2. According to Formula 5 and Formula 6, it can be determined that the position of the convolution sliding window S1 in the weight matrix Wc should be (3, 4). That is, the elements in the 1st row and 1st column of the convolution sliding window S1 coincide with the elements in the 3rd row and 4th column of the weight matrix Wc. And the values ​​of the elements that coincide with the position of the convolution sliding window S1 are reset to the values ​​of the elements in the corresponding positions in the convolution sliding window. That is, the weight matrix Wc after the second reset (i'=2) is obtained as follows Figure 5B as shown in .

[0175] And so on, until all the elements in each row of the weight matrix Wc are reset, and the final weight matrix of the convolutional layer is obtained.

[0176] As can be seen from the above, since the constructed convolution sliding window S1 contains the weights corresponding to k convolution kernels, the convolution sliding window S1 is used as a unit to slide in the initialized weight matrix Wc and reset the element values ​​in the weight matrix Wc. Therefore, the weight values ​​of the k convolution kernels appear alternately in the reset weight matrix Wc. For example, you can refer to Figure 5B In the figure, k=2. After the second reset, in the reset weight matrix Wc, the first row corresponds to the weight value of convolution kernel 1, the second row corresponds to the weight value of convolution kernel 2, the third row corresponds to the weight value of convolution kernel 1, and the fourth row corresponds to the weight value of convolution kernel 2.

[0177] b) Method of constructing bias matrix bc

[0178] Since in a convolution layer, each convolution kernel corresponds to a bias, the bias matrix bc can be constructed according to formula 7 as follows:

[0179]

[0180] Among them, b1, b2, …b k They represent the bias values ​​of the 1st, 2nd,…, kth convolution kernels respectively.

[0181] Thus, the weight matrix Wc of the convolutional layer is constructed and assigned (the reset value) and the bias matrix bc.

[0182] It can be understood that the embodiment of the present application only provides a specific implementation method for constructing the weight matrix Wc and the bias matrix bc, but the protection scope of the present application is not limited to this. Any changes or substitutions based on the inventive concept of the present application and within the technical scope disclosed in the present application should be covered within the protection scope of the present application.

[0183] For example: the above embodiment is a convolution sliding window S1 constructed simultaneously based on k convolution kernels, that is, the height of the constructed convolution sliding window S1 is k, and then the convolution sliding window S1 is used as a unit to slide in the initialized weight matrix Wc and reset the element values ​​in the weight matrix Wc.

[0184] Of course, the convolution sliding window S1 can also be constructed according to the k convolution kernels respectively, that is, the height of the constructed convolution sliding window S1 is 1, and k convolution sliding windows S1 need to be constructed. These k convolution sliding windows S1 slide in the weight matrix Wc in turn and reset the element values ​​in the weight matrix Wc. In this case, the weight values ​​of the k convolution kernels appear in sequence in the reset weight matrix Wc. For example: the 1st to 4th rows in the reset weight matrix Wc correspond to the weight values ​​of convolution kernel 1, the 5th to 6th rows correspond to the weight values ​​of convolution kernel 2, and so on.

[0185] In this case, the constructed bias matrix bc also needs to be adjusted accordingly. At this time, the bias matrix bc can be constructed according to formula 8, as follows:

[0186]

[0187] It should also be noted that in subsequent calculations of other layers, if there are calculations corresponding to the convolution kernels here, the corresponding order needs to be adjusted to be consistent with the order here.

[0188] For other contents, please refer to the description of the relevant contents above, which will not be repeated here.

[0189] (3) Based on the constructed weight matrix Wc and bias matrix bc, the output of the convolutional layer is calculated and the output result is recorded as Zc.

[0190] After the previous conversion, the convolution calculation of the input data x has been converted into a linear expression, namely Wc·x'+bc.

[0191] In the convolutional layer of the convolutional neural network, an activation function can also be used to add nonlinear factors to enrich the expression of the convolutional neural network model. Then, the output result of the convolutional layer is Zc=Ac(Wc·x'+bc).

[0192] Among them, Ac(x) is the activation function. The commonly used activation function of the convolutional layer is generally the ReLU function, and sometimes the Sigmoid function is also used.

[0193] Obviously, the operation of the activation function makes the output result and the input data (Wc·x'+bc) no longer present a linear relationship as a whole. Therefore, based on the concept provided by the present application, it is also necessary to convert the activation function into a linear expression.

[0194] It is understandable that the operation of the activation function makes the overall relationship no longer linear, but for a specific element in the input data (Wc×x'+bc), it can be expressed as linear. To this end, the application converts the result of the activation function acting on an element into the product of the activation state (denoted as Sc) and the element, and the activation state can also be understood as the slope of the activation function.

[0195] Taking the ReLU function as an example, the activation function conversion and the meaning of the activation state are explained.

[0196] The ReLu function is max(0, x). This means that inputs greater than zero are retained, and values ​​less than zero are changed to zero. That is to say, when using the ReLu function, if the value of a specific element in the input data (Wc·x'+bc) is greater than zero, the original value of the element is retained, and if the value of the element is less than zero, the value of the element is changed to zero. In one example, the activation state Sc of the element can be represented as a matrix whose element values ​​contain 1 and 0. Among them, if the value of the element is greater than zero, it is multiplied by 1, which is an activated state, thereby retaining the value of the element. If the value of the element is less than zero, it is multiplied by 0, which is an inactive state (or called an inactive state, a deactivated state, etc.), thereby changing the value of the element to zero.

[0197] After introducing the activation state Sc, the output of the convolutional layer can be transformed as follows:

[0198]

[0199] Among them, mathematical symbols is the Hadamard product of the matrix. It can be seen that this application converts the calculation of the convolution layer into a mathematically equivalent linear expression, that is, the output result of the convolution layer is

[0200] 2. Equivalent conversion of pooling layer

[0201] After the convolutional layer performs feature extraction, the output data is passed to the pooling layer for selection and information filtering. The pooling layer contains a pre-set pooling function (also called a pooling kernel). The function of the pooling function is to replace the result of a single element in the input data with the feature map statistics of the adjacent area, thereby continuously reducing the spatial size of the data. The pooling layer selects the pooling area similar to the process of the convolution kernel sweeping over the input data, which is controlled by the pooling size, pooling step and padding.

[0202] Since the pooling layer does not require parameterization, and the pooling operations of each channel are independent of each other for the input data of the pooling layer, a single pooling kernel is usually used in a pooling layer. The pooling window of the pooling kernel is p2×q2, and the pooling step is s2. Here, the input data of a certain pooling layer is recorded as xp, and the input data xp is m2×n2×r2. The input data xp of this layer is the output result of the previous convolutional layer.

[0203] Similar to the conversion of the convolution layer, the explanation of the pooling layer is also to convert the pooling process into a linear expression for the input data xp of the pooling layer and the pooling kernel contained in the pooling layer. In other words, for the specific input data and the model of the pooling layer, the convolution calculation is expressed as a mathematically equivalent linear expression of the form y=w×x. Among them, y is the output of the pooling layer, w and b are the weights and biases equivalent to the pooling process, respectively. It can be understood that in the linear expression equivalent to the pooling process here, the weight w should be a matrix. Here, the weight matrix of the determined pooling layer is recorded as Wp.

[0204] The process of converting the pooling process into a linear expression is similar to the process of converting the convolution calculation into a linear expression. The following is a detailed introduction to the method of converting the pooling process into a linear expression:

[0205] (1) Expand the input multidimensional array to 1×n d The matrix is ​​denoted as xp'. d =m2×n2×r2.

[0206] like Figure 6 As shown, the input data is 4×4×2 (m2=4, n2=4, r2=2) as an example.

[0207] For example, the input data includes data from two channels, channel 1 and channel 2. Each channel contains 4×4 elements. The elements contained in channel 1 are recorded as The elements contained in channel 2 are recorded as in, Figure 6 The various elements included in channel 1 are shown in FIG. The processing of channel 2 can refer to the processing of channel 1 and will not be repeated here.

[0208] It can be seen that the input data of the pooling layer is a multi-channel matrix. According to the channel order of the channel matrix and the arrangement order of the elements in the channel matrix, when the input data of the pooling layer is expanded, n d =m2×n2×r2=32. That is, the input data of the pooling layer is expanded into a 1×32 matrix

[0209] In some embodiments, the pooling layer may also use a padding algorithm to process input data. That is, before the pooling kernel is calculated, the size of the input data is artificially increased by a padding algorithm to offset the effect of size shrinkage in the calculation. The padding algorithm may be, for example, zero padding, repeated boundary value padding, or other methods. For specific padding methods, reference may be made to the relevant technology in the art, which will not be described in detail here.

[0210] It should be noted that if the pooling layer adopts a padding algorithm, the input data needs to be padded first and then flattened when interpreting the pooling layer.

[0211] (2) Based on the expanded matrix xp', construct the weight matrix Wp of the pooling layer.

[0212] Exemplarily, the pooling sliding window S2 can be constructed based on the expanded matrix xp' and the pooling kernel. In the present application, the pooling sliding window S2 is used to simulate the process of the pooling kernel regularly scanning the input data when working. Then, the weight matrix Wp of the pooling layer is constructed according to the pooling sliding window S2 combined with the pooling process. That is, the calculation of the pooling layer is equivalent to that of the linear expression.

[0213] First, construct a pooling sliding window S2.

[0214] Among them, the size of S2 is S2 width ×1. Among them, S2 width is the width of the convolution sliding window S2, which can be calculated by formula 9 as follows:

[0215] S2 width =(q2-1)×m2×r2+(p2-1)×r2+1 (Formula 9)

[0216] According to the specific operation of the pooling kernel, the values ​​in the pooling sliding window S2 are initialized. For example: if the pooling kernel adopts the maximum pooling method, the elements corresponding to the pooling window in the pooling sliding window S2 can be initialized to 1, indicating that these elements will participate in this pooling process. The elements in the pooling sliding window S2 that are not corresponding to the pooling window are initialized to 0, indicating that these elements do not participate in this pooling process. It is important to distinguish the meaning of the pooling sliding window S2 and the pooling window. The pooling sliding window S2 is a window constructed in this application, with a size of S2. width ×1. The pooling window is the window corresponding to the pooling kernel, and its size is p2×q2.

[0217] For example: Figure 6 The input data xp (m2=4, n2=4, r2=2) and the pooling kernel (p2=2, q2=2) are used, and the maximum pooling method is taken as an example for explanation.

[0218] According to formula 9, the width of the pooling sliding window can be determined to be 11. Combined with the position of the pooling core in the input data xp, the pooling sliding window S2 can be determined to be:

[0219] [1 0 1 0 0 0 0 0 1 0 1].

[0220] Then, the weight matrix Wp of the pooling layer is constructed.

[0221] For example, a weight matrix Wp is first constructed with an initial value of 0. That is, all elements in Wp are initially zero, and the values ​​of the corresponding positions in Wp are subsequently reset through the sliding window S2. The size of Wp is Wp width ×Wp height . Wp width is the width of the weight matrix Wp, Wp height is the height of the weight matrix Wp. Specifically, the width of the weight matrix Wp can be determined according to the input data of the pooling layer, and the height of Wp can be determined according to the size of the pooling window and the pooling step size, as well as the input data.

[0222] For example, Wp width and Wp height They can be calculated by formula 10 and formula 11 respectively, as follows:

[0223] Wp width =m2×n2×r2 (Formula 10)

[0224]

[0225] It can be seen that the width of the weight matrix Wp is the same as the width of the matrix xp' of the expanded input data. It can be understood as the number of times the pooling sliding window S2 is reset.

[0226] Similar to moving the convolution sliding window S1 in the convolution layer, when moving the pooling sliding window S2, it is also necessary to consider the position of the pooling kernel when scanning the input data.

[0227] Similarly, when the pooling core scans the input data xp (referring to the input data when it is not expanded) for pooling calculation, the position of the first element of the pooling core in the input data is recorded as (e2, f2), indicating the position of the e2th row and the f2th column. The position of the first element in the pooling sliding window S2 in the weight matrix Wp is recorded as (v2, h2), indicating that the first element in the pooling sliding window S2 is located at the position of the v2th row and the h2th column in the weight matrix Wp. When the sliding window S2 resets the element value in the weight matrix Wp, the two positional relationships can be referred to Formula 12 and Formula 13, as follows:

[0228] h2=(e2-1)×m2×r2×s2+(f2-1)×r2×s2+1 (Formula 12)

[0229] v2=(i'-1)×r2+1 (Formula 13)

[0230] Wherein, i' represents the i'th time resetting the element value in the weight matrix Wp according to the pooling sliding window S2, and i' is an integer greater than or equal to 1. and

[0231] The process of moving the pooling sliding window S2 in the weight matrix Wp is similar to the process of moving the convolution sliding window S1 in the weight matrix Wc, and will not be repeated here.

[0232] For example: Still Figure 6 The input data xp (m2=4, n2=4, r2=2) and the pooling kernel (p2=2, q2=2, s2=1) are used, and the maximum pooling method is taken as an example for explanation.

[0233] like Figure 7 As shown, according to the above method, the pooling sliding window S2 is determined, and according to formula 10 and formula 11, the size of the weight matrix Wp can be determined to be 32×18. The specific method of moving the pooling sliding window S2 can be referred to the description of the upper mobile convolution sliding window S1, which will not be repeated here.

[0234] It should be noted that when calculating the convolution kernel of the convolution layer, multiple channels are involved in the calculation. When calculating the pooling kernel of the pooling layer, one channel is calculated first, and then another channel.

[0235] It should also be noted that when the maximum pooling method is used, the element with the largest value is selected from the multiple elements participating in the pooling process. That is to say, when resetting the element values ​​at the corresponding positions in the weight matrix Wp according to the pooling sliding window S2, the input values ​​corresponding to the weight value of 1 are also compared. The weight corresponding to the element with the largest value is retained as 1, and the rest are reset to 0. For example, please continue to refer to Figure 7 , after resetting the weight matrix Wp for the first time, it is necessary to compare the value of the input data corresponding to the weight value of 1. That is, compare and The value of . Keep the weight of the element with the largest value as 1, and change the weight of the rest to 0. And so on.

[0236] (2) Calculate the output of the pooling layer, which can be recorded as Zp

[0237] According to the flattened xp' and the constructed Wp, the calculation of the pooling layer can be expressed as a mathematically equivalent linear expression, that is, Zp = Wp × xp'.

[0238] 3. Equivalent conversion of fully connected layers

[0239] The fully connected layer in a convolutional neural network is equivalent to the hidden layer in a traditional feedforward neural network. The fully connected layer is usually built at the end of the hidden layer of a convolutional neural network and only transmits signals to other fully connected layers. The feature map loses its 3D structure in the fully connected layer, is expanded into a vector and passed to the next layer through an activation function. The fully connected layer plays the role of a "classifier" in the entire convolutional neural network.

[0240] It can be understood that in a convolutional neural network, the input of the first fully connected layer is the output of the previous pooling layer. The input of the fully connected layers of other layers is the output of the previous fully connected layer. Here, the input of the fully connected is recorded as

[0241] The fully connected layer processes the input data by: activating the n3 neurons contained in the input data (if the input data is a matrix, the input data needs to be expanded. The expansion method can refer to the method described above) using the activation function, and then weighting and summing the n3 neurons using the corresponding weights, and adding a bias. According to the different activation functions used by the fully connected layer, the fully connected layer can be divided into a linear fully connected layer and a nonlinear fully connected layer. The activation function here is recorded as

[0242] Based on the inventive concept of this application, it is necessary to convert the calculation of the activation function into a linear expression. In the following, the equivalent conversion of the linear fully connected layer and the nonlinear fully connected layer will be described respectively.

[0243] (1) Equivalent transformation of linear full connection

[0244] The activation function used in the linear fully connected layer is a linear function (for example, the activation function is the ReLU function, various improved versions of ReLU, Maxout function, etc.), so the activation function can be directly expressed as a piecewise linear function, that is,

[0245] For example: the linear activation function contained in the linear fully connected layer is the ReLU function. The ReLu function is max(0, x). This means that inputs greater than zero are retained, and values ​​less than zero are changed to zero. In other words, when using the ReLu function, if the value of a specific element in the input data is greater than zero, the original value of the element is retained, and if the value of the element is less than zero, the value of the element is changed to zero. In other words, the ReLu function can be expressed as a piecewise linear function. When the value of an element in the input data is greater than zero, the activation slope is 1, and when the value of an element in the input data is less than or equal to zero, the activation slope is 0.

[0246] Another example: the linear activation function contained in the linear fully connected layer is the leaky ReLU function. The leaky ReLU function is max(0,x)+leak*min(0,x). Among them, leak is a very small constant, which retains some values ​​of the negative axis so that the information of the negative axis is not completely lost. In other words, the leaky ReLU function can be expressed as such a piecewise linear function. When the value of the element in the input data is greater than zero, the activation slope is 1. When the value of the element in the input data is less than or equal to zero, the activation slope is leak, and leak is less than 1.

[0247] Assume that the convolutional neural network contains L fully connected layers, of which the fully connected layer currently calculated is the lth layer, which contains n3 neurons. The output result of this layer can be recorded as Z (l) .

[0248] Then, the fully connected layer of the lth layer includes n3 neurons. When the activation function is expressed as a piecewise linear function, the activation slope s contains n3 slope values, denoted as S (l) =[s1,s2,……s n3 ]. The activation intercept t contains n3 intercepts, denoted by T (l) =[t1,t2,……,t n3 ]. Then, the output of the fully connected layer at layer l is:

[0249]

[0250] Among them, W (l) is the weight matrix of the lth fully connected layer, b (l) is the bias matrix corresponding to this layer. Subsequently, for a linear fully connected neural network with L layers, its output is:

[0251]

[0252] Thus, the linear expression of the linear fully connected layer neural network is obtained, that is, the final output result of all linear fully connected layers.

[0253] (2) Equivalent transformation of nonlinear full connection

[0254] The activation function used in the nonlinear fully connected layer is a nonlinear function (for example, the activation function is a Tanh function, a Sigmoid function, etc.). Therefore, based on the concept of the present invention, it is still necessary to express the nonlinear activation function as a linear one. In one example, a linear fully connected neural network can be trained. To fit the nonlinear activation function here. The specific training method can refer to the relevant technology in this field, which is not limited here. It can be understood that according to the universal approximation theorem, the fully connected neural network can fit any function, and the loss of accuracy is small enough to be ignored.

[0255] As mentioned above, the linear fully connected neural network It can be directly expressed as a piecewise linear function, that is:

[0256] The activation slope Similar to the activation slope s of the linear fully connected layer, it is recorded here as Activate Intercept Similar to the activation slope t of the linear fully connected layer, it is recorded here as Then, the output of the fully connected layer at layer l is:

[0257]

[0258] Among them, W (l) is the weight matrix of the lth fully connected layer, b (l) is the bias matrix corresponding to this layer. Subsequently, for a linear fully connected neural network with L layers, its output is:

[0259]

[0260] Thus, the linear expression of the nonlinear fully connected layer neural network is obtained, that is, the final output result of all nonlinear fully connected layers.

[0261] 4. Combined with the above explanation methods of each layer, the overall architecture of the convolutional neural network is explained

[0262] See also Figure 2 The overall architecture of the convolutional neural network has been explained above. Here, the overall architecture of the convolutional neural network and the above-mentioned method of explaining each layer are combined to explain the overall architecture of the convolutional neural network in order to determine the decision factors of the convolutional neural network.

[0263] According to the expression of the output result of the pooling layer determined above, and the expression of the output result of the convolution layer, the output of the lth layer can be expressed as:

[0264]

[0265] in, is the output result of the pooling layer of layer l, is the equivalent weight matrix of the pooling layer of layer l, The output result of the convolutional layer of the lth layer is the input data of the pooling layer of the lth layer. It is equivalent to the output result of the convolutional layer of layer l-1.

[0266] Therefore, the features extracted by the convolutional layer and the pooling layer can be That is, the output result of the last pooling layer (i.e., the mth pooling layer) is expressed as:

[0267]

[0268] Will Substituting into the calculation formula of the fully connected layer, the output result of the last fully connected layer is determined as:

[0269]

[0270] Therefore, the final equivalent model, that is, the explanation model, can be expressed as:

[0271]

[0272] Where f(x) is the decision function of the convolutional neural network, that is, all the calculations of the convolutional neural network. Softmax is a normalized exponential function that can make the range of each element between (0,1) and the sum of all elements equal to 1. The explanation model determined by this application is mathematically equivalent to the decision function of the original convolutional neural network.

[0273] Subsequently, the decision basis of the convolutional neural network, i.e., the decision factor, can be determined based on the explanation model and the input x. The decision basis can be represented by an equivalent weight matrix According to the characteristics of linear expression, it is easy to understand that for different inputs x, the size of the weight value directly reflects the degree of influence of the input on the linear expression result. In other words, in linear expression, the larger the weight value, the greater the influence of the corresponding input on the result of linear expression, which can be considered as the greater contribution to the result of linear expression.

[0274] It can be seen that when the calculation of the complex convolutional neural network is equivalently converted into a linear expression, the decision basis can be clearly obtained, that is, the factors that play a key role in the decision-making process.

[0275] For example, if the input is image data, then based on the weight values ​​corresponding to each element in the weight matrix, it can be determined which pixel points (positions) in the image data play a key role in the decision-making process.

[0276] In a specific application scenario, a feature explanation graph can be generated based on the weight matrix in the explanation model. For example, in the feature explanation graph, the depth of color can be used to reflect the size of each weight value in the weight matrix. The darker the color, the larger the corresponding weight value, which shows that the image data at that location is more important to the decision-making process. In other words, the feature explanation graph visually expresses the decision factors in the decision-making process, making it easier for users to intuitively understand the decision-making of the convolutional neural network model.

[0277] For example: In the auxiliary diagnosis of medical equipment, the convolutional neural network used in auxiliary diagnosis can be converted into an equivalent explanation model, and the positions in the patient's medical imaging data that are key factors can be determined based on the size of each weight value in the weight matrix (or the color depth in the feature explanation map). On the one hand, patients can better understand the decision conclusions of the convolutional neural network. For example, the explanation model determines that the lung area in the image data is the key factor. And the convolutional neural network decides that it is a lung problem. Patients can better understand and trust the auxiliary diagnosis of medical equipment. On the other hand, doctors can also manually check the key factors based on the explanation model. For example, if the explanation model determines that the lung area in the image data is the key factor, and the convolutional neural network decides that it is a chest problem. Then, the doctor can manually check the image data to correctly determine the lesion.

[0278] In another application scenario, security analysts can also analyze, verify, and evaluate the reliability and security of convolutional neural networks based on the explanatory model in order to reduce the potential risks of convolutional neural networks during deployment.

[0279] It is understandable that the above-mentioned application scenarios of the explanation model of the convolutional neural network provided by the present application are illustrative, and do not limit the use of the explanation model in other application scenarios. For example, in the scenario of autonomous driving, the explanation model can also be used to detect malicious input (i.e., the input data is specially constructed by the attacker) to prevent malicious input from causing the convolutional neural network to make incorrect decisions, such as causing the car to deviate from the correct road. The explanation model can also be used in other scenarios such as intelligent analysis of surveillance videos. The embodiments of the present application are not limited to this.

[0280] like Fig. 8A As shown, it is a flow chart of a method for determining a decision factor of a convolutional neural network provided in an embodiment of the present application, the method comprising:

[0281] S801, obtaining an input image, and identifying a convolutional neural network model of the input image.

[0282] The convolutional neural network model for identifying the input includes at least one convolutional layer, at least one pooling layer, and at least one fully connected layer. The specific structure of the convolutional neural network can be found in the above description.

[0283] S802. Determine a second weight matrix corresponding to each convolutional layer according to one or more convolution kernels contained in each convolutional layer in the at least one convolutional layer; determine a third weight matrix corresponding to each pooling layer according to a pooling window matrix contained in each pooling layer in the at least one pooling layer; and determine a fourth weight matrix corresponding to each fully connected layer according to an activation function contained in each fully connected layer in the at least one fully connected layer.

[0284] Among them, the method for determining the second weight matrix according to the convolution kernel contained in the convolution layer can refer to the method for constructing the weight matrix Wc of the convolution layer above. The method for determining the third weight matrix according to the pooling layer contained in the convolution layer can refer to the method for constructing the weight matrix Wp of the pooling layer above. The method for determining the fourth weight matrix according to the activation function of the fully connected layer can refer to the method for constructing the weight matrix W of the fully connected layer above. (l) I will not go into details here.

[0285] S803: Determine a first weight matrix according to the second weight matrix, the third weight matrix and the fourth weight matrix, wherein the first weight matrix has the same size as the pixel size of the input image.

[0286] Exemplarily, the second weight matrix, the third weight matrix and the fourth weight matrix are multiplied to obtain the first weight matrix.

[0287] S804. Determine, according to the weight values ​​of each element in the first weight matrix, one or more pixels in the input image corresponding to the positions of each element in the first weight matrix as decision factors of the convolutional neural network model.

[0288] It should be noted that the above embodiment is described by taking the example that the convolutional neural network model includes at least one convolutional layer, at least one pooling layer and at least one fully connected layer. In another example, it is also mentioned above that the convolutional neural network model may not include a pooling layer. Then, in this example, the method provided in the embodiment of the present application can also be used to interpret the convolutional neural network model and determine the key decision factors.

[0289] like Figure 8B As shown, it is a flow chart of a method for determining a decision factor of a convolutional neural network provided in an embodiment of the present application, the method comprising:

[0290] S901. Obtain an input image and a convolutional neural network model for recognizing the input image, wherein the convolutional neural network model includes at least one convolutional layer and at least one fully connected layer.

[0291] S902. Determine a second weight matrix corresponding to each convolutional layer according to one or more convolution kernels contained in each convolutional layer in the at least one convolutional layer; determine a third weight matrix corresponding to each fully connected layer according to an activation function contained in each fully connected layer in the at least one fully connected layer.

[0292] The method for determining the second weight matrix according to the convolution kernel contained in the convolution layer can refer to the method for constructing the weight matrix Wc of the convolution layer above. The method for determining the third weight matrix according to the activation function of the fully connected layer can refer to the method for constructing the weight matrix W of the fully connected layer above. (l) I will not go into details here.

[0293] S903: Determine a first weight matrix according to the second weight matrix and the third weight matrix, wherein the first weight matrix has the same size as a pixel size of the input image.

[0294] Exemplarily, the second weight matrix and the third weight matrix are multiplied to obtain the first weight matrix.

[0295] S904. Determine, according to the weight values ​​of each element in the first weight matrix, one or more pixels in the input image corresponding to the positions of each element in the first weight matrix as decision factors of the convolutional neural network model.

[0296] For the rest of the content, please refer to the relevant description above and will not be repeated here.

[0297] It is understandable that, in order to realize the above functions, the above-mentioned terminals etc. include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present invention.

[0298] The embodiment of the present application can divide the functional modules of the above-mentioned terminal etc. according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0299] In the case of dividing each functional module into corresponding functional modules, Fig. 9 FIG. 1 shows a possible structural diagram of the electronic device involved in the above embodiment. Fig. 9 As shown, the electronic device includes: an acquisition unit 901 and a determination unit 902.

[0300] Among them, the acquisition unit 901 is used to acquire an input image and identify a convolutional neural network model of the input image, wherein the convolutional neural network model includes at least one convolutional layer, at least one pooling layer, and at least one fully connected layer.

[0301] A determination unit 902 is used to determine a second weight matrix corresponding to each convolution layer according to one or more convolution kernels included in each convolution layer in the at least one convolution layer; determine a third weight matrix corresponding to each pooling layer according to a pooling window matrix included in each pooling layer in the at least one pooling layer; determine a fourth weight matrix corresponding to each fully connected layer according to an activation function included in each fully connected layer in the at least one fully connected layer;

[0302] The determining unit 902 is further configured to determine a first weight matrix according to the second weight matrix, the third weight matrix and the fourth weight matrix, wherein the first weight matrix has the same size as the pixel size of the input image;

[0303] The determination unit 902 is also used to determine one or more pixels in the input image corresponding to the positions of each element in the first weight matrix according to the weight values ​​of each element in the first weight matrix as decision factors of the convolutional neural network model.

[0304] In another example, when the convolutional neural network model for identifying the input image includes at least one convolutional layer and at least one fully connected layer, but does not include a pooling layer, then the acquisition unit 901 is used to acquire the input image and the convolutional neural network model for identifying the input image, wherein the convolutional neural network model includes at least one convolutional layer and at least one fully connected layer;

[0305] A determining unit 902 is configured to determine a second weight matrix corresponding to each convolutional layer according to one or more convolution kernels included in each convolutional layer in the at least one convolutional layer; and determine a third weight matrix corresponding to each fully connected layer according to an activation function included in each fully connected layer in the at least one fully connected layer;

[0306] The determining unit 902 is further configured to determine a first weight matrix according to the second weight matrix and the third weight matrix, wherein the first weight matrix has the same size as the pixel size of the input image;

[0307] The determination unit 902 is also used to determine one or more pixels in the input image corresponding to the positions of each element in the first weight matrix according to the weight values ​​of each element in the first weight matrix as decision factors of the convolutional neural network model.

[0308] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0309] In the case of adopting an integrated unit, the acquisition unit 901 and the determination unit 902 may be integrated together and may be the processor 110 of the electronic device 100 .

[0310] The present application also provides a chip system, such as Fig.10 As shown, the chip system includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 and the interface circuit 1102 can be interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (such as the memory of the electronic device 100). For another example, the interface circuit 1102 can be used to send signals to other devices (such as the processor 1101). Exemplarily, the interface circuit 1102 can read the instructions stored in the memory and send the instructions to the processor 1101. When the instructions are executed by the processor 1101, the electronic device can execute the various steps performed by the electronic device 100 (such as a mobile phone) in the above-mentioned embodiment. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0311] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0312] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0313] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

[0314] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining decision factors of a convolutional neural network, characterized in that include: Obtaining an input image, and identifying a convolutional neural network model of the input image, wherein the convolutional neural network model includes at least one convolutional layer, at least one pooling layer, and at least one fully connected layer; Determine a second weight matrix corresponding to each convolution layer according to one or more convolution kernels included in each convolution layer in the at least one convolution layer; determine a third weight matrix corresponding to each pooling layer according to a pooling window matrix included in each pooling layer in the at least one pooling layer; determine a fourth weight matrix corresponding to each fully connected layer according to an activation function included in each fully connected layer in the at least one fully connected layer; Determine a first weight matrix according to the second weight matrix, the third weight matrix and the fourth weight matrix, wherein the first weight matrix has the same size as the pixel size of the input image; According to the weight values ​​of the elements in the first weight matrix, one or more pixels in the input image corresponding to the positions of the elements in the first weight matrix are determined as decision factors of the convolutional neural network model.

2. The method according to claim 1, characterized in that The determining, according to one or more convolution kernels included in each convolution layer in the at least one convolution layer, a second weight matrix corresponding to each convolution layer comprises: Determine the width of the second weight matrix corresponding to each convolutional layer according to the input data of each convolutional layer; The height of the second weight matrix corresponding to each convolution layer is determined according to the one or more convolution kernel receptive fields and the convolution step size of each convolution layer, and the size of the input data of each convolution layer.

3. The method according to claim 2, characterized in that The input data of the first convolutional layer in the at least one convolutional layer is a multi-channel matrix of m1×n1×r1, where m1×n1 is the size of the channel matrix, r1 is the number of channel matrices, and includes k convolution kernels, the receptive fields of the k convolution kernels are p1×q1, and the convolution step size is s1. The width of the second weight matrix corresponding to the first convolutional layer is Wc width =m1×n1×r1; The height of the second weight matrix corresponding to the first convolutional layer is 4. The method according to claim 2 or 3, characterized in that: According to one or more convolution kernels included in each convolution layer in the at least one convolution layer, a second weight matrix corresponding to each convolution layer is determined, specifically: Initialize the value of each element in the second weight matrix corresponding to each convolutional layer to zero; According to the weight values ​​of one or more convolution kernels contained in each convolution layer, the values ​​of each element in the second weight matrix corresponding to each convolution layer are set according to the order of the channel matrix in the input data, the arrangement order of the elements in the channel matrix and the convolution calculation order.

5. The method according to claim 1, characterized in that The determining, according to the size of the pooling window included in each pooling layer in the at least one pooling layer, a third weight matrix corresponding to each pooling layer comprises: Determine the width of the third weight matrix corresponding to each pooling layer according to the input data of each pooling layer; The height of the third weight matrix corresponding to each pooling layer is determined according to the pooling window matrix and the pooling step size of each pooling layer and the input data of each pooling layer.

6. The method according to claim 5, characterized in that The input data of the first pooling layer in the at least one pooling layer is a multi-channel matrix of m2×n2×r2, where m2×n2 is the size of the channel matrix, r2 is the number of channel matrices, and the size of the pooling window is p2×q2, and the pooling step size is s2. The width of the third weight matrix corresponding to the first pooling layer is Wp width =m2×n2×r2; The height of the third weight matrix corresponding to the first pooling layer is 7. The method according to claim 5 or 6, characterized in that: According to the pooling window included in each pooling layer in the at least one pooling layer, a third weight matrix corresponding to each pooling layer is determined, specifically: Using the pooling window matrix contained in each pooling layer and the pooling method adopted by each pooling layer, calculate the pooling result corresponding to the input data of each pooling layer; According to the pooling results corresponding to the input data of each pooling layer, the values ​​of each element in the third weight matrix corresponding to each pooling layer are set according to the order of the channel matrix, the arrangement order of the elements in the channel matrix, and the pooling calculation order.

8. The method according to claim 1, characterized in that A first fully connected layer in the at least one fully connected layer comprises a linear activation function and a fifth weight matrix, wherein the fifth weight matrix is ​​a weight matrix of the first fully connected layer, The step of determining, according to the activation function included in each fully connected layer in the at least one fully connected layer, a fourth weight matrix corresponding to each fully connected layer comprises: The first fully connected layer includes the linear activation function represented as a first piecewise linear function, wherein the plurality of activation slopes included in the first piecewise linear function form a first activation slope matrix; The first activation slope matrix is ​​multiplied by the fifth weight matrix to obtain a fourth weight matrix corresponding to the first fully connected layer.

9. The method according to claim 1, characterized in that: The second fully connected layer in the at least one fully connected layer comprises a nonlinear activation function and a sixth weight matrix, and the sixth weight matrix is ​​a weight matrix of the second fully connected layer; The step of determining, according to the activation function included in each fully connected layer in the at least one fully connected layer, a fourth weight matrix corresponding to each fully connected layer comprises: Calculate output data of the second fully connected layer according to the input data of the second fully connected layer, the nonlinear activation function included therein, and the sixth weight matrix; Training a linear fully connected neural network based on the input data and output data of the second fully connected layer; The linear activation function in the linear fully connected neural network obtained by training is expressed as a second piecewise linear function; the second piecewise linear function includes a plurality of activation slopes forming a second activation slope matrix, and the fully connected neural network obtained by training includes a seventh weight matrix; The second activation slope matrix is ​​multiplied by the seventh weight matrix to obtain a fourth weight matrix corresponding to the second fully connected layer.

10. A method for determining a decision factor of a convolutional neural network, characterized in that include: Obtaining an input image, and identifying a convolutional neural network model of the input image, wherein the convolutional neural network model includes at least one convolutional layer and at least one fully connected layer; Determine a second weight matrix corresponding to each convolution layer according to one or more convolution kernels included in each convolution layer in the at least one convolution layer; Determine a third weight matrix corresponding to each fully connected layer according to the activation function included in each fully connected layer in the at least one fully connected layer; Determine a first weight matrix according to the second weight matrix and the third weight matrix, wherein the first weight matrix has the same size as the pixel size of the input image; According to the weight values ​​of the elements in the first weight matrix, one or more pixels in the input image corresponding to the positions of the elements in the first weight matrix are determined as decision factors of the convolutional neural network model.

11. The method according to claim 10, characterized in that The determining, according to one or more convolution kernels included in each convolution layer in the at least one convolution layer, a second weight matrix corresponding to each convolution layer comprises: Determine the width of the second weight matrix corresponding to each convolutional layer according to the input data of each convolutional layer; The height of the second weight matrix corresponding to each convolution layer is determined according to the one or more convolution kernel receptive fields and the convolution step size of each convolution layer, and the size of the input data of each convolution layer.

12. The method according to claim 11, characterized in that The input data of the first convolutional layer in the at least one convolutional layer is a multi-channel matrix of m1×n1×r1, where m1×n1 is the size of the channel matrix, r1 is the number of channel matrices, and includes k convolution kernels, the receptive fields of the k convolution kernels are p1×q1, and the convolution step size is s1. The width of the second weight matrix corresponding to the first convolutional layer is Wc width =m1×n1×r1; The height of the second weight matrix corresponding to the first convolutional layer is 13. The method according to claim 11 or 12, characterized in that: According to one or more convolution kernels included in each convolution layer in the at least one convolution layer, a second weight matrix corresponding to each convolution layer is determined, specifically: Initialize the value of each element in the second weight matrix corresponding to each convolutional layer to zero; According to the weight values ​​of one or more convolution kernels contained in each convolution layer, the values ​​of each element in the second weight matrix corresponding to each convolution layer are set according to the order of the channel matrix in the input data, the arrangement order of the elements in the channel matrix and the convolution calculation order.

14. The method according to any one of claims 10 to 12, characterized in that: A first fully connected layer in the at least one fully connected layer comprises a linear activation function and a fourth weight matrix, wherein the fourth weight matrix is ​​a weight matrix of the first fully connected layer, The determining, according to the activation function included in each fully connected layer in the at least one fully connected layer, a third weight matrix corresponding to each fully connected layer comprises: The first fully connected layer includes the linear activation function represented as a first piecewise linear function, wherein the plurality of activation slopes included in the first piecewise linear function form a first activation slope matrix; The first activation slope matrix is ​​multiplied by the fourth weight matrix to obtain a third weight matrix corresponding to the first fully connected layer.

15. The method according to any one of claims 10 to 12, characterized in that: The second fully connected layer in the at least one fully connected layer comprises a nonlinear activation function and a fifth weight matrix, and the fifth weight matrix is ​​a weight matrix of the second fully connected layer; The determining, according to the activation function included in each fully connected layer in the at least one fully connected layer, a third weight matrix corresponding to each fully connected layer comprises: Calculate output data of the second fully connected layer according to the input data of the second fully connected layer, the nonlinear activation function included therein, and the fifth weight matrix; Training a linear fully connected neural network based on the input data and output data of the second fully connected layer; The linear activation function in the linear fully connected neural network obtained by training is expressed as a second piecewise linear function; the second piecewise linear function includes a plurality of activation slopes forming a second activation slope matrix, and the fully connected neural network obtained by training includes a sixth weight matrix; The second activation slope matrix is ​​multiplied by the sixth weight matrix to obtain a third weight matrix corresponding to the second fully connected layer.

16. A device for determining a decision factor of a convolutional neural network, characterized in that include: An acquisition unit, used to acquire an input image and identify a convolutional neural network model of the input image, wherein the convolutional neural network model includes at least one convolutional layer, at least one pooling layer, and at least one fully connected layer; A determination unit, configured to determine a second weight matrix corresponding to each convolution layer according to one or more convolution kernels included in each convolution layer in the at least one convolution layer; determine a third weight matrix corresponding to each pooling layer according to a pooling window matrix included in each pooling layer in the at least one pooling layer; and determine a fourth weight matrix corresponding to each fully connected layer according to an activation function included in each fully connected layer in the at least one fully connected layer; The determining unit is further configured to determine a first weight matrix according to the second weight matrix, the third weight matrix and the fourth weight matrix, wherein the first weight matrix has the same size as the pixel size of the input image; The determination unit is also used to determine one or more pixels in the input image corresponding to the positions of each element in the first weight matrix according to the weight values ​​of each element in the first weight matrix, as decision factors of the convolutional neural network model.

17. The device according to claim 16, characterized in that The determining, according to one or more convolution kernels included in each convolution layer in the at least one convolution layer, a second weight matrix corresponding to each convolution layer comprises: Determine the width of the second weight matrix corresponding to each convolutional layer according to the input data of each convolutional layer; The height of the second weight matrix corresponding to each convolution layer is determined according to the one or more convolution kernel receptive fields and the convolution step size of each convolution layer, and the size of the input data of each convolution layer.

18. The device according to claim 17, characterized in that The input data of the first convolutional layer in the at least one convolutional layer is a multi-channel matrix of m1×n1×r1, where m1×n1 is the size of the channel matrix, r1 is the number of channel matrices, and includes k convolution kernels, the receptive fields of the k convolution kernels are p1×q1, and the convolution step size is s1. The width of the second weight matrix corresponding to the first convolutional layer is Wc width =m1×n1×r1; The height of the second weight matrix corresponding to the first convolutional layer is 19. The device according to claim 17 or 18, characterized in that According to one or more convolution kernels included in each convolution layer in the at least one convolution layer, a second weight matrix corresponding to each convolution layer is determined, specifically: Initialize the value of each element in the second weight matrix corresponding to each convolutional layer to zero; According to the weight values ​​of one or more convolution kernels contained in each convolution layer, the values ​​of each element in the second weight matrix corresponding to each convolution layer are set according to the order of the channel matrix in the input data, the arrangement order of the elements in the channel matrix and the convolution calculation order.

20. The device according to claim 16, characterized in that The determining, according to the size of the pooling window included in each pooling layer in the at least one pooling layer, a third weight matrix corresponding to each pooling layer comprises: Determine the width of the third weight matrix corresponding to each pooling layer according to the input data of each pooling layer; The height of the third weight matrix corresponding to each pooling layer is determined according to the pooling window matrix and the pooling step size of each pooling layer and the input data of each pooling layer.

21. The device according to claim 20, characterized in that The input data of the first pooling layer in the at least one pooling layer is a multi-channel matrix of m2×n2×r2, where m2×n2 is the size of the channel matrix, r2 is the number of channel matrices, and the size of the pooling window is p2×q2, and the pooling step size is s2. The width of the third weight matrix corresponding to the first pooling layer is Wp width =m2×n2×r2; The height of the third weight matrix corresponding to the first pooling layer is 22. The device according to claim 20 or 21, characterized in that According to the pooling window included in each pooling layer in the at least one pooling layer, a third weight matrix corresponding to each pooling layer is determined, specifically: Using the pooling window matrix contained in each pooling layer and the pooling method adopted by each pooling layer, calculate the pooling result corresponding to the input data of each pooling layer; According to the pooling results corresponding to the input data of each pooling layer, the values ​​of each element in the third weight matrix corresponding to each pooling layer are set according to the order of the channel matrix, the arrangement order of the elements in the channel matrix, and the pooling calculation order.

23. The device according to claim 16, characterized in that A first fully connected layer in the at least one fully connected layer comprises a linear activation function and a fifth weight matrix, wherein the fifth weight matrix is ​​a weight matrix of the first fully connected layer, The step of determining, according to the activation function included in each fully connected layer in the at least one fully connected layer, a fourth weight matrix corresponding to each fully connected layer comprises: The first fully connected layer includes the linear activation function represented as a first piecewise linear function, wherein the plurality of activation slopes included in the first piecewise linear function form a first activation slope matrix; The first activation slope matrix is ​​multiplied by the fifth weight matrix to obtain a fourth weight matrix corresponding to the first fully connected layer.

24. The device according to claim 16, characterized in that The second fully connected layer in the at least one fully connected layer comprises a nonlinear activation function and a sixth weight matrix, and the sixth weight matrix is ​​a weight matrix of the second fully connected layer; The step of determining, according to the activation function included in each fully connected layer in the at least one fully connected layer, a fourth weight matrix corresponding to each fully connected layer comprises: Calculate output data of the second fully connected layer according to the input data of the second fully connected layer, the nonlinear activation function included therein, and the sixth weight matrix; Training a linear fully connected neural network based on the input data and output data of the second fully connected layer; The linear activation function in the linear fully connected neural network obtained by training is expressed as a second piecewise linear function; the second piecewise linear function includes a plurality of activation slopes forming a second activation slope matrix, and the fully connected neural network obtained by training includes a seventh weight matrix; The second activation slope matrix is ​​multiplied by the seventh weight matrix to obtain a fourth weight matrix corresponding to the second fully connected layer.

25. A device for determining a decision factor of a convolutional neural network, characterized in that include: An acquisition unit, configured to acquire an input image and identify a convolutional neural network model of the input image, wherein the convolutional neural network model includes at least one convolutional layer and at least one fully connected layer; A determination unit, configured to determine a second weight matrix corresponding to each convolutional layer according to one or more convolution kernels included in each convolutional layer in the at least one convolutional layer; and determine a third weight matrix corresponding to each fully connected layer according to an activation function included in each fully connected layer in the at least one fully connected layer; The determining unit is further configured to determine a first weight matrix according to the second weight matrix and the third weight matrix, wherein the first weight matrix has the same size as the pixel size of the input image; The determination unit is also used to determine one or more pixels in the input image corresponding to the positions of each element in the first weight matrix according to the weight values ​​of each element in the first weight matrix, as decision factors of the convolutional neural network model.

26. The device according to claim 25, characterized in that The determining, according to one or more convolution kernels included in each convolution layer in the at least one convolution layer, a second weight matrix corresponding to each convolution layer comprises: Determine the width of the second weight matrix corresponding to each convolutional layer according to the input data of each convolutional layer; The height of the second weight matrix corresponding to each convolution layer is determined according to the one or more convolution kernel receptive fields and the convolution step size of each convolution layer, and the size of the input data of each convolution layer.

27. The device according to claim 26, characterized in that The input data of the first convolutional layer in the at least one convolutional layer is a multi-channel matrix of m1×n1×r1, where m1×n1 is the size of the channel matrix, r1 is the number of channel matrices, and includes k convolution kernels, the receptive fields of the k convolution kernels are p1×q1, and the convolution step size is s1. The width of the second weight matrix corresponding to the first convolutional layer is Wc width =m1×n1×r1; The height of the second weight matrix corresponding to the first convolutional layer is 28. The device according to claim 26 or 27, characterized in that According to one or more convolution kernels included in each convolution layer in the at least one convolution layer, a second weight matrix corresponding to each convolution layer is determined, specifically: Initialize the value of each element in the second weight matrix corresponding to each convolutional layer to zero; According to the weight values ​​of one or more convolution kernels contained in each convolution layer, the values ​​of each element in the second weight matrix corresponding to each convolution layer are set according to the order of the channel matrix in the input data, the arrangement order of the elements in the channel matrix and the convolution calculation order.

29. The device according to any one of claims 25 to 27, characterized in that A first fully connected layer in the at least one fully connected layer comprises a linear activation function and a fourth weight matrix, wherein the fourth weight matrix is ​​a weight matrix of the first fully connected layer, The determining, according to the activation function included in each fully connected layer in the at least one fully connected layer, a third weight matrix corresponding to each fully connected layer comprises: The first fully connected layer includes the linear activation function represented as a first piecewise linear function, wherein the plurality of activation slopes included in the first piecewise linear function form a first activation slope matrix; The first activation slope matrix is ​​multiplied by the fourth weight matrix to obtain a third weight matrix corresponding to the first fully connected layer.

30. The device according to any one of claims 25 to 27, characterized in that The second fully connected layer in the at least one fully connected layer comprises a nonlinear activation function and a fifth weight matrix, and the fifth weight matrix is ​​a weight matrix of the second fully connected layer; The determining, according to the activation function included in each fully connected layer in the at least one fully connected layer, a third weight matrix corresponding to each fully connected layer comprises: Calculate output data of the second fully connected layer according to the input data of the second fully connected layer, the nonlinear activation function included therein, and the fifth weight matrix; Training a linear fully connected neural network based on the input data and output data of the second fully connected layer; The linear activation function in the linear fully connected neural network obtained by training is expressed as a second piecewise linear function; the second piecewise linear function includes a plurality of activation slopes forming a second activation slope matrix, and the fully connected neural network obtained by training includes a sixth weight matrix; The second activation slope matrix is ​​multiplied by the sixth weight matrix to obtain a third weight matrix corresponding to the second fully connected layer.

31. An electronic device, characterized in that: include: A processor, a memory and a touch screen, wherein the memory and the touch screen are coupled to the processor, the memory is used to store computer program codes, and the computer program codes include computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes the method for determining the decision factor of a convolutional neural network as described in any one of claims 1 to 15.

32. A computer storage medium, characterized in that It includes computer instructions, which, when executed on a terminal, cause the terminal to execute the method for determining the decision factor of a convolutional neural network as described in any one of claims 1 to 15.

33. A computer program product, characterized in that When the computer program product runs on a computer, the computer is caused to execute the method for determining the decision factor of a convolutional neural network as claimed in any one of claims 1 to 15.

Images