Visual interactive learning method, system and device for convolutional neural network
Through the visual interactive learning method for convolutional neural networks, the forward propagation process and mathematical calculation of convolutional neural networks are demonstrated, and the problem of low interpretability of convolutional neural networks is solved, helping learners to understand the internal workflow of the model.
Patent Information
- Application Number
- CN202210335322.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-31
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Convolutional neural networks have low interpretability in the field of image classification, making it difficult for beginners to understand the internal workflow and decision-making mechanism of the model. The existing visualization methods cannot effectively demonstrate the overall structure and mathematical operations.
Using a visual interactive learning method for convolutional neural networks, the model is trained and normalized by determining the network structure and training data set, obtaining intermediate results, displaying the forward propagation process of the model and mathematical calculations of each layer, and using animation and color Mapping allows learners to understand the model intuitively.
Fully show the neural network model structure, explain mathematical operations, help learners to deeply understand the workflow and decision-making mechanism of convolutional neural networks, and make up for the shortcomings of existing methods.
Smart Images

Figure CN114863204B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition and data processing, and in particular to a visualization method, system and device for the internal network structure of a convolutional neural network. Background Art
[0002] Since the advent of convolutional neural networks in 1989, they have been widely used in image classification, speech recognition, and natural language processing, and have achieved great success, especially in the field of image classification. Convolutional neural networks are widely used because they avoid complex pre-processing of images and simply input the original image directly into the model to get the desired result.
[0003] With the widespread application of convolutional neural networks, people have an increasingly urgent need to deeply understand convolutional neural network models. People do not want to manipulate unknown things to achieve their own goals. However, it is like a "black box" with extremely low interpretability, which is extremely challenging for beginners. Beginners can only virtualize a multi-layer convolutional neural network model in their minds based on theoretical knowledge. They must not only understand how each layer works, but also understand how to choose different algorithms to complete data conversion between different network layers. Therefore, a key challenge in learning convolutional neural networks (CNNs) is the complex interaction between low-level mathematical operations and the high-level integration of such operations within the network.
[0004] In order to explain convolutional neural networks, the methods currently proposed on the market can be divided into the following three types: one is to use a highly interpretable model to fit the decision boundary of the convolutional neural network model, but it can only explain the local part but not the whole; the second is to extract decision rules based on rule methods, which are only applicable to shallow convolutional neural networks; the third is to qualitatively analyze convolution features. They all only show local sample decisions or partial convolution parameters or the final recognition results, so they cannot help beginners to understand the convolutional neural network as a whole, and have not achieved good results. Summary of the invention
[0005] In view of the problems existing in the prior art, the present invention proposes a visual interactive learning method, system and device for convolutional neural networks.
[0006] The technical solution adopted by the method of the present invention is: a visual interactive learning method for convolutional neural networks, comprising the following steps:
[0007] Step 1: Determine a network structure of a convolutional neural network and a training data set for training the convolutional neural network;
[0008] Step 2: training the convolutional neural network according to the training data set to obtain a trained convolutional neural network;
[0009] Step 3: After normalizing the image to be identified, input it into the trained convolutional neural network for model prediction;
[0010] Step 4: Get the parameters and intermediate results in the model prediction process and save them in global variables;
[0011] Step 5: According to the pre-set hierarchical structure of the convolutional neural network, the overall structure of the forward propagation of the convolutional neural network model is visualized;
[0012] Step 6: Based on the intermediate results generated during the prediction process, the specific processing and mathematical calculations of each layer of the network are displayed in the form of animation.
[0013] The technical solution adopted by the system of the present invention is: a visual interactive learning system for convolutional neural networks, including the following modules:
[0014] Module 1, used to determine the network structure of the convolutional neural network and the training data set for training the convolutional neural network;
[0015] Module 2, used to train the convolutional neural network according to the training data set to obtain a trained convolutional neural network;
[0016] Module 3 is used to normalize the image to be identified and then input it into the trained convolutional neural network for model prediction;
[0017] Module 4 is used to obtain the parameters and intermediate results in the model prediction process and save them in global variables;
[0018] Module 5 is used to visualize the overall structure of the forward propagation of the convolutional neural network model according to the pre-set hierarchical structure of the convolutional neural network;
[0019] Module 6 is used to display the specific processing and mathematical calculations of each layer of the network in the form of animation based on the intermediate results generated during the prediction process.
[0020] The technical solution adopted by the device of the present invention is: a visual interactive learning device for convolutional neural networks, comprising:
[0021] one or more processors;
[0022] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the visual interactive learning method for convolutional neural networks.
[0023] The present invention can not only fully demonstrate the model structure of the neural network, but also well explain the mathematical operations therein, which helps to deepen learners' understanding of the workflow and decision-making mechanism of convolutional neural networks, and makes up for the shortcomings of related visualization learning methods on the market. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a flow chart of a method according to an embodiment of the present invention;
[0025] Figure 2 A schematic diagram of the structure of a convolutional neural network (CNN) according to an embodiment of the present invention;
[0026] Figure 3 This is a visualization effect diagram of an embodiment of the present invention;
[0027] Figure 4 This is a visualization effect diagram of the convolutional layer of an embodiment of the present invention;
[0028] Figure 5 This is a visualization effect diagram of the pooling layer in an embodiment of the present invention;
[0029] Figure 6 This is a visualization effect diagram of the fully connected layer of an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0031] Please see Figure 1 The present invention provides a visual interactive learning method for convolutional neural networks, comprising the following steps:
[0032] Step 1: Determine a network structure of a convolutional neural network and a training data set for training the convolutional neural network;
[0033] like Figure 2 As shown, the convolutional neural network CNN model used in this embodiment adopts a six-layer hierarchical structure of convolution layer-pooling layer-convolution layer-pooling layer-flattening layer-fully connected layer. At the same time, the data set adopts a 28*28 single-channel image data set. In order to ensure the accuracy of model training, 400 hand-drawn images of size 28*28 are prepared for each classification category and saved as binary file data. Since there are 3 categories of data in this embodiment, a total of 1200 pictures of these 3 categories are the input of this embodiment.
[0034] Step 2: training the convolutional neural network according to the training data set to obtain a trained convolutional neural network;
[0035] In this embodiment, the parameters are set in advance, and then the model training is performed based on the tensorflow.js framework; the parameters of each layer of the network are set, such as the size and step size of the two convolution kernels in the convolution layer, the dimension of the output space, the excitation function, the bias, etc.; the pooling layer needs to set the pooling method, and the more commonly used methods are maximum pooling and mean pooling; the default excitation function in the fully connected layer is the softmax function. When the data set is loaded, the model training function cnn.train() is called through tensorflow.js. Table 1 below is a parameter table of the model network layer of this embodiment.
[0036]
[0037] As shown in Table 1, the input of the convolution layer conv1 is a 28*28 array, and the size of the parameter Weights is [5,5,1,8]. It can be concluded that the convolution kernel size used in this layer is 5*5, and the dimension of the output space is 8, so there are 8 output feature maps with a size of 24*24. It should be noted that the pooling layer and the flattening layer do not need to generate parameters, so there are no Weights and bias in these two layers. The input of the fully connected layer dense can be simply understood as an array of size 128. For each category, the CNN model generates a set of parameter Weights of size 128 and a bias bias, and the predicted value Predict of the category can be obtained by calculation. The calculation formula of the predicted value is as follows:
[0038]
[0039] Since the current model only uses 3 categories, the output size of the fully connected layer is 3.
[0040] Step 3: After normalizing the image to be identified, input it into the trained convolutional neural network for model prediction;
[0041] This embodiment uses HTML5 Canvas to build a 280*280 drawing board in the Web interface. The user can freely draw an image as the image to be recognized by clicking the mouse to trigger the event. When the user clicks the "Predict" button, the pixel array RGBArr of the image drawn by the user is obtained through the Canva function context.getImageData(), and its size is 280*280*4. Since the input of the model is 28*28 single-channel image data, RGBArr needs to be converted into a single-channel image pixel array PixelArr with a length of 28*28. After completing the transformation of the drawing board image shape, the image is normalized, and the normalization operation is: new_pixel=pixel / 255, where pixel represents the single-channel image pixel value after the drawing board is transformed, and new_pixel represents the normalized pixel value, and then the image data is model predicted. In this implementation, new_pixel=pixel / 255 is a normalization operation on the predicted image input by the user, and (255-p) / 255 is a normalization operation on the 1200 training images.
[0042] In this embodiment, the method of normalizing the image to be recognized is (255-p) / 255, where p represents the value of the current pixel. Step 4: Obtain the parameters and intermediate results in the model prediction process and save them in global variables;
[0043] This embodiment obtains the parameters and intermediate results in the model prediction process by calling the API interface of tensorflow.js, and saves them into global variables to facilitate subsequent visualization work.
[0044] Step 5: According to the pre-set hierarchical structure of the convolutional neural network, the overall structure of the forward propagation of the convolutional neural network model is visualized;
[0045] In this embodiment, the output data of each layer obtained in step 4 is mapped to RGB pixel data according to certain mapping rules, and then the RGB pixel data is converted into a corresponding base64 encoded image for visualization on a Web page. The specific color mapping method is shown in equations 3, 4, and 5. When single-channel processed data is mapped in this way, negative pixel values can be mapped to red, zero pixel values can be mapped to white, and positive pixel values can be mapped to blue. This method of using color to measure pixel values allows users to clearly see the positive and negative nature of image data during processing.
[0046]
[0047]
[0048]
[0049] Among them, R, G, B represent the values corresponding to the three color channels of red, green, and blue after color mapping, maxp represents the absolute value of the maximum pixel value in the current image, and p represents the value of the current pixel.
[0050] Step 6: Based on the intermediate results generated during the prediction process, the specific processing and mathematical calculations of each layer of the network are displayed in the form of animation.
[0051] This embodiment uses the front-end data visualization tool Echarts to complete the visualization of the CNN network hierarchy. The specific steps are as follows:
[0052] Step 6.1: Calculate the screen canvas size and reasonably layout and coordinate the entire canvas according to the network structure;
[0053] The canvas content automatically adapts to the screen size, which can avoid the display effect of the web page being affected by changes in window specifications. The pre-set coordinates can accurately display the image at the corresponding position, making it easier to draw curves.
[0054] Step 6.2: According to the position coordinates determined in step 6.1, add the image element to the canvas and display the output feature maps of the first four layers of the model at the corresponding positions;
[0055] Step 6.3: Visualize each neuron of the flattened layer in the form of a rect element in an Echarts graphic, with the same color as the source pixel;
[0056] Step 6.4: Draw the lines of the model network;
[0057] Add the bezierCurve element in Echarts graphic between the user's hand drawing and the convolution layer conv1, and between the pooling layer pooling1 and the convolution layer conv2 to draw Bezier curves. Add the line element in Echartsgraphic between other layers to draw straight lines.
[0058] Step 6.5: Add interactive effects;
[0059] A floating trigger event is added to the output feature map to display the input image of the output map in the current level through a highlight effect; a click trigger event is added to the output feature map to display the specific operation details of each level in the form of a modal box.
[0060] Figure 3 The following is a visualization of the model network structure. After the user draws an image on the drawing board and clicks the prediction button, the page displays the overall visualization of the model prediction, as shown in the figure below. Figure 3As shown in the figure. The first column is the user's hand-drawn picture, which is the input of the model. The second column is the image after the first convolution, the third column is the image after the first pooling, the fourth column is the image after the second convolution, the fifth column is the image after the second pooling, the sixth column is the flat layer neuron, and the seventh column shows the predicted probabilities of the three classification categories. Figure 3 In the figure, the neurons in the convolutional layer and the fully connected layer are fully connected to the neurons in the previous layer, while other neurons are only connected one-to-one with one neuron in the previous layer, which makes it easier for users to understand the origin of the input feature maps of neurons in each layer.
[0061] This embodiment visualizes the often-overlooked flatten layer to help users understand how the high-dimensional pooling layer pooling2 is transitionally connected to the one-dimensional fully connected layer dense through the flatten layer. The output feature map after the second pooling is traversed in row-first order, and the tensor of size [4,4,8] output by pooling2 is converted to a one-dimensional tensor of size [null,128]. This embodiment represents each neuron in the flatten layer as a small rectangular block whose color is the same as its source pixel in the previous layer. The neurons in the flatten layer are not only connected to the 8 output feature maps of the previous layer pooling2, but also to the predicted probabilities output by the next layer dense. Users can view the connections before and after the neurons by hovering the mouse, and the highlighted lines show the connections between their associated inputs and classification results.
[0062] Step 6.6: Based on the intermediate results generated during the prediction process, the specific processing and mathematical calculations of each layer of the network are displayed in the form of animation.
[0063] After the user clicks on the output feature map of the convolutional layer or pooling layer, the system pops up a modal box to display the calculations undergone by each neuron in the output feature map in the form of an animation.
[0064] Figure 4 The figure shows the computational visualization of the output feature map in the convolutional layer conv1. Figure 4 As shown, the left and right canvases in the modal box are the input matrix image and output matrix image of the layer respectively. The timer controls the movement of the small rectangular blocks in the input matrix and the output matrix to simulate the mathematical operations performed by the convolution layer. The pixel area in the small rectangular box is the pixel area currently involved in the convolution operation. At the same time, the modal box performs a dot product operation on the pixel matrix in the small rectangular box and the convolution kernel in text form to obtain the calculation result. Since the CNN model in this method uses the Relu excitation function, it is also necessary to perform a Relu operation on the calculation result to ensure the non-negativity of the output pixel value. The formula of the Relu function is as follows:
[0065]
[0066] This synchronization between input, output, and mathematical functions allows users to better understand how the kernel convolves the matrix in the convolutional layer.
[0067] Figure 5 Shown is the computational visualization of the output feature map in the pooling layer, and Figure 4 There are similarities in page layout. Figure 5 As shown in the figure, the left and right canvases in the modal box are the input matrix image and output matrix image of the layer respectively. The timer controls the movement of the small rectangular blocks in the input matrix and the output matrix to simulate the internal process during the pooling operation. The pixel area in the small rectangular box is the operation area currently participating in the maximum pooling. At the same time, the modal box displays the pixel matrix in the small rectangular box and the maximum pixel value output in text form.
[0068] When you click the last column of the hierarchical network, which is the final predicted probability, a modal box pops up showing the details of the fully connected layer, such as Figure 6 The modal box shows in detail the calculation process from the flatten layer to the fully connected layer dense and the normalization of the calculation results. The canvas in the modal box displays the linear calculation of the flatten layer data, the bias, the calculation result result, and the prediction probability prob obtained by normalization through the softmax function in turn, which effectively helps users sort out the detailed workflow and specific mathematical calculations of the fully connected layer.
[0069] This embodiment uses the hand-drawn image of the user on the web interface as the input of the convolutional neural network model, and uses it to start the forward propagation process of the convolutional neural network model, extracts the training parameters of the sample at each layer and the intermediate results generated, and displays them on the web page in a very friendly way. Users can not only see very intuitively how the convolutional neural network CNN processes the input image data into the final prediction result, but also understand the basic mathematical operations in an interactive way, which is conducive to establishing an overall impression of the convolutional neural network and achieving a good learning effect.
[0070] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.
Claims
1. A visual interactive learning method for convolutional neural networks, characterized in that: The following steps are involved: Step 1: Determine a network structure of a convolutional neural network and a training data set for training the convolutional neural network; Step 2: training the convolutional neural network according to the training data set to obtain a trained convolutional neural network; Step 3: After normalizing the image to be identified, input it into the trained convolutional neural network for model prediction; Step 4: Get the parameters and intermediate results in the model prediction process and save them in global variables; Step 5: According to the pre-set hierarchical structure of the convolutional neural network, the overall structure of the forward propagation of the convolutional neural network model is visualized; Among them, the processing results of the input image after passing through each layer of the convolutional neural network are subjected to RGB color mapping, and the single-channel processed data is converted into RGB image data and presented; Among them, the color mapping method is: Among them, R, G, and B represent the values corresponding to the red, green, and blue color channels after color mapping. maxp Indicates the absolute value of the maximum pixel value in the current image p Indicates the value of the current pixel; Step 6: Based on the intermediate results generated during the prediction process, the specific processing and mathematical calculations of each layer of the network are displayed in the form of animation; The specific implementation includes the following sub-steps: Step 6.1: Calculate the screen canvas size in the front-end data visualization tool Echarts and layout and coordinate the entire canvas according to the network structure; Step 6.2: After determining the position coordinates, add the image element to the canvas and display the output feature maps of the first four layers of the convolutional neural network at the corresponding positions; Step 6.3: Visualize each neuron of the flattened layer in the form of a rect element in the Echarts graphic, with the same color as the source pixel; Step 6.4: Draw the lines of the convolutional neural network model network; Add the bezierCurve element in Echarts graphic between the input image to be recognized and the convolution layer conv1, and between the pooling layer pooling1 and the convolution layer conv2 to draw the Bezier curve; add the line element in Echartsgraphic between other layers to draw a straight line; Step 6.5: Add interactive effects; The output feature graph adds a floating trigger event, which displays the input image of the output graph in the current level through a highlight effect; the output feature graph adds a click trigger event, which displays the specific operation details of each level in the form of a modal box; Step 6.6: Based on the intermediate results generated during the prediction process, the specific processing and mathematical calculations of each layer of the network are displayed in the form of animation.
2. The visual interactive learning method for convolutional neural networks according to claim 1, characterized in that: The method of normalizing the image to be identified in step 3 is (255- p ) / 255, where p Indicates the value of the current pixel.
3. A visual interactive learning system for convolutional neural networks, characterized in that: Includes the following modules: Module 1, used to determine the network structure of the convolutional neural network and the training data set for training the convolutional neural network; Module 2, used to train the convolutional neural network according to the training data set to obtain a trained convolutional neural network; Module 3 is used to normalize the image to be identified and then input it into the trained convolutional neural network for model prediction; Module 4 is used to obtain the parameters and intermediate results in the model prediction process and save them in global variables; Module 5 is used to visualize the overall structure of the forward propagation of the convolutional neural network model according to the pre-set hierarchical structure of the convolutional neural network; Among them, the processing results of the input image after passing through each layer of the convolutional neural network are subjected to RGB color mapping, and the single-channel processed data is converted into RGB image data and presented; Among them, the color mapping method is: Among them, R, G, and B represent the values corresponding to the red, green, and blue color channels after color mapping. maxp Indicates the absolute value of the maximum pixel value in the current image p Indicates the value of the current pixel; Module 6 is used to display the specific processing and mathematical calculation of each layer of the network in the form of animation based on the intermediate results generated during the prediction process; Module 6 includes the following submodules: Module 6.1 is used to calculate the screen canvas size in the front-end data visualization tool Echarts and layout and coordinate the entire canvas according to the network structure; Module 6.2 is used to determine the position coordinates, add the image element to the canvas, and display the output feature maps of the first four layers of the convolutional neural network at the corresponding position; Module 6.3, for visualizing each neuron of the flattened layer in the form of a rect element in an Echarts graphic, with the same color as the source pixel; Module 6.4, for drawing connections of convolutional neural network model networks; Add the bezierCurve element in Echarts graphic between the input image to be recognized and the convolution layer conv1, and between the pooling layer pooling1 and the convolution layer conv2 to draw the Bezier curve; add the line element in Echartsgraphic between other layers to draw a straight line; Module 6.5, for adding interactive effects; The output feature graph adds a floating trigger event, which displays the input image of the output graph in the current level through a highlight effect; the output feature graph adds a click trigger event, which displays the specific operation details of each level in the form of a modal box; Module 6.6 is used to display the specific processing and mathematical calculations of each layer of the network in the form of animation based on the intermediate results generated during the prediction process.
4. The visual interactive learning system for convolutional neural networks according to claim 3, characterized in that: The method for normalizing the image to be identified in module 3 is (255- p ) / 255, where p Indicates the value of the current pixel.
5. A visual interactive learning device for convolutional neural networks, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the visual interactive learning method for convolutional neural networks as described in any one of claims 1 to 2.
Citation Information
Patent Citations
Distributed hand-painted picture identification system and method based on convolutional neural network
CN108197632A
A convolutional neural network training result detection and visualization method based on a genetic algorithm
CN109598341A
Method and system for cell image segmentation using multi-stage convolutional neural networks
US20190228268A1
