Scatter plot-based classification network auxiliary analysis method and device

Abstract

The application relates to a scatter plot-based classification network auxiliary analysis method and device, wherein the method comprises the following steps: training a classification network by using a data set, obtaining each high-dimensional feature vector output by any network layer in the classification network and forming a high-dimensional feature group, performing linear mapping on each high-dimensional feature vector in the high-dimensional feature group to obtain a low-dimensional feature vector corresponding to each high-dimensional feature vector; constructing a scatter plot based on a low-dimensional space, determining the classification category to which the low-dimensional feature vector corresponding to each projection point in the scatter plot belongs according to the annotation information of the projection point in the corresponding sample image, and judging the classification rationality of each sample image in the data set according to the classification category to which the low-dimensional feature vector corresponding to each projection point belongs and the distribution state of the projection point. The technical scheme can qualitatively guide and analyze the classification network structure, data set division, training strategy and the like, and is beneficial to optimizing the training process of the classification model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a classification network auxiliary analysis method and device based on scatter plot. BACKGROUND

[0002] In recent years, artificial intelligence and big data have become the focus of attention in various fields at home and abroad. In the field of computer vision, image algorithms based on deep learning have a wide range of applications. Using the one-to-one correspondence between images and annotation information to participate in the training of convolutional neural networks can enable the network to complete classification, target detection and semantic segmentation and other work.

[0003] Deep learning technology is increasingly used in industrial detection, but most of the time deep learning is treated as a black box, and the model is driven by data. The training results are quite different from the expected results, but there is no way to start and analyze, which will cause poor explainability in the model training process. How to analyze the reasons for poor classification model training according to the output of the training results is an important problem to be discussed.

[0004] Now many deep neural networks cannot make decisions that are completely understood from a human perspective. Although many models have reached the level of human recognition, we still have some reservations about the prediction results because we do not fully understand what the model's prediction is based on and we do not know when it will make mistakes. This is an important factor that prevents models from being deployed in some critical areas, such as transportation, medicine, law and other fields. In these fields, the ability of the model cannot be fully trusted, and it is difficult to imagine that even if 5G autonomous driving is realized one day, people will really give up the steering wheel and sleep in the car. The answer is no, which requires analyzing the training results of errors to guide the training of the model or the adjustment of some hyperparameters in the training process. By explaining or guiding the model to achieve human trust in the model, a safer and more reliable application is created, thereby promoting the progress of the entire AI industry. SUMMARY

[0005] The technical problem solved by the present application is how to evaluate and analyze the training process of the classification network. To solve the above technical problem, the present application provides a classification network auxiliary analysis method and device based on scatter plot.

[0006] According to a first aspect, one embodiment provides a classification network-assisted analysis method based on scatter plots, comprising: training a classification network using a preset dataset; the dataset includes multiple sample images, and the classification network includes at least one network layer; obtaining each high-dimensional feature vector output by any network layer in the classification network and forming a high-dimensional feature group; performing linear mapping on each high-dimensional feature vector in the high-dimensional feature group, and obtaining low-dimensional feature vectors corresponding to each high-dimensional feature vector through vector selection; constructing a scatter plot based on a low-dimensional space; the scatter plot includes projection points of the low-dimensional feature vectors corresponding to each high-dimensional feature vector; determining the classification category to which the low-dimensional feature vectors corresponding to each projection point belong based on the annotation information of each projection point in the corresponding sample image in the scatter plot; and judging the rationality of the classification of each sample image in the dataset based on the classification category to which the low-dimensional feature vectors corresponding to each projection point belong and the distribution status of each projection point.

[0007] The step of obtaining the high-dimensional feature vectors output by any network layer in the classification network and forming a high-dimensional feature group includes: taking any network layer in the classification network as the analysis object; during the feature learning process of the classification network on each sample image in the dataset, using the feature map output by the network layer taken as the analysis object as a high-dimensional feature vector; and forming the high-dimensional feature group by obtaining the high-dimensional feature vectors corresponding to each sample image in the dataset, wherein the high-dimensional feature group is represented as X = {x1, x2, ... x...} i …,x m}, where x i Let represent the i-th high-dimensional feature vector, and m represent the total number of all high-dimensional feature vectors.

[0008] The step of linearly mapping each high-dimensional feature vector in the high-dimensional feature group and obtaining the corresponding low-dimensional feature vectors through vector selection includes: centering each high-dimensional feature vector in the high-dimensional feature group to obtain a centralized set denoted as X. c The covariance matrix of the high-dimensional eigenvectors is calculated based on the centered set, and the covariance matrix is ​​expressed as ε = X. c X c T / m; Obtain the eigenvectors and corresponding eigenvalues ​​of the covariance matrix, and arrange the eigenvectors by row according to the magnitude of the eigenvalues; Obtain the first few rows of eigenvectors and form a matrix P; Obtain the low-dimensional feature set by multiplying the matrix P and the centered set, and represent it as Y = PX. c The low-dimensional feature set includes the low-dimensional feature vectors corresponding to the high-dimensional feature vectors obtained through vector selection.

[0009] The centering processing includes: calculating vector mean of each high-dimensional feature vector in the high-dimensional feature set; obtaining the centering result of each high-dimensional feature vector by operating the difference between each high-dimensional feature vector and the vector mean, and denoted as x i ′=x i —∑x i / m;statistical centering result corresponding to each high-dimensional feature vector to obtain the centering set, and denoted as X c ={x1′,x2′,...x i ′…,x m ′}。

[0010] In linear mapping of each high-dimensional feature vector in the high-dimensional feature set, the mapped features are classified by the centering processing, including: obtaining the number of preset classification categories and denoted as k; randomly selecting k high-dimensional feature vectors as center points, traversing all data corresponding to each high-dimensional feature vector and calculating the distance from each data point to the center point, and dividing each data point into the nearest center point group; calculating the average value of each cluster according to each center point group, and taking the average value as a new center point, and using the new center point to perform iterative calculation again until the k center points converge or reach a preset training number, at which time the iterative calculation stops; calculating the distance between each center point, if the distance between two center points is less than a preset threshold, merging the two center points, and obtaining the center points by merging until the center points cannot be merged any more; judging whether the number of categories is equal to the number of classification categories according to the current number of center points, if yes, the classification of the mapped features is completed; wherein, for the low-dimensional feature vector obtained by mapping, the corresponding abnormal boundary is obtained and the abnormal boundary is denoted as d i =x i ′ T Σ -1 x′;wherein, Σ represents a covariance matrix, if d i is less than a specified threshold, the abnormal boundary is marked as a first color to represent normal classification, otherwise the abnormal boundary is marked as a second color to represent abnormal classification.

[0011] For the mapping of a two-dimensional plane, each mapped feature is denoted as

[0012]

[0013]

[0014] Further, d

[0015]

[0016]

[0017]

[0018] In the two-dimensional plane, the effective boundary corresponding to the mapped low-dimensional feature vector is expressed by the formula

[0019]

[0020] Wherein, n represents the first moment or mean, μ represents the second central moment, λ represents the major axis and minor axis of the ellipse, and δ represents the distance threshold.

[0021] For the mapping of three-dimensional space, the matrix is set to be and is expressed as

[0022]

[0023] Wherein,

[0024]

[0025]

[0026] In the three-dimensional space, the distance d is calculated j And is expressed as

[0027] d j = (X j - μ) T H -1 (X j - μ).

[0028] Wherein, h represents the second central distance, d represents the distance, N represents the number of feature vectors in a specific category, X represents the jth feature vector, and μ represents the average value of the feature vectors in the group.

[0029] If the distance d j is less than a certain threshold, the data within the effective boundary corresponding to the mapped low-dimensional feature vector is normal data.

[0030] The classification rationality of each sample image in the data set is determined according to the classification category to which the low-dimensional feature vector corresponding to each projection point belongs and the distribution state of each projection point, including: for each sample image in the data set, if the sample image belongs to a classification category and is close in distribution to the projection point generated by the sample image of the same category, the sample image is a normal sample; otherwise, the sample image is an abnormal sample; when there is an abnormal sample in the data set, it is determined that the classification of each sample image in the data set does not have rationality, and the abnormal sample is removed.

[0031] According to a second aspect, in an embodiment, a computer processing device is provided, comprising: a memory storing a data set and a classification network, the data set comprising a plurality of sample images, the classification network comprising at least one network layer; a processor comprising: a training module configured to train the classification network using the data set; an obtaining module configured to obtain each high-dimensional feature vector output by any network layer of the classification network and form a high-dimensional feature group; a mapping module configured to perform linear mapping on each high-dimensional feature vector in the high-dimensional feature group to obtain a low-dimensional feature vector corresponding to each high-dimensional feature vector through vector selection; a constructing module configured to construct a scatter plot based on a low-dimensional space; the scatter plot comprising projection points of the low-dimensional feature vectors corresponding to each high-dimensional feature vector; and an analyzing module configured to determine a classification category to which the low-dimensional feature vector corresponding to each projection point belongs according to label information of the sample image corresponding to each projection point in the scatter plot, and determine classification rationality of each sample image in the data set according to the classification category to which the low-dimensional feature vector corresponding to each projection point belongs and a distribution state of each projection point.

[0032] According to a third aspect, in an embodiment, a computer readable storage medium is disclosed, the medium storing a program executable by a processor to implement the method of the first aspect.

[0033] The beneficial effects of the present application are:

[0034] According to the above embodiments, a classification network-assisted analysis method and apparatus based on scatter plots are provided. The classification network-assisted analysis method includes: training a classification network using a preset dataset; obtaining the high-dimensional feature vectors output by any network layer in the classification network and forming a high-dimensional feature group; performing linear mapping on each high-dimensional feature vector in the high-dimensional feature group, and obtaining the low-dimensional feature vectors corresponding to each high-dimensional feature vector through vector selection; constructing a scatter plot based on the low-dimensional space; determining the classification category to which the low-dimensional feature vectors corresponding to each projection point belong based on the annotation information of each projection point in the corresponding sample image; and judging the rationality of the classification of each sample image in the dataset based on the classification category to which the low-dimensional feature vectors corresponding to each projection point belong and the distribution status of each projection point. On the one hand, the technical solution uses scatter plots to visualize the projection points of the low-dimensional feature vectors corresponding to the high-dimensional feature vectors, thereby guiding the partitioning of the dataset. This not only enhances the visualization and analysis capabilities of multi-scale features but also improves the analytical capabilities of neural network feature extraction. On the other hand, the technical solution can use the classification categories to which the low-dimensional feature vectors corresponding to each projection point in the scatter plot belong, as well as their distribution in a plane or space, to determine the rationality of the classification of each sample image in the dataset. This can qualitatively guide and analyze aspects such as the classification network structure, dataset partitioning, and training strategies, which is beneficial for optimizing the training process of the classification model. Attached Figure Description

[0035] Figure 1 This is a structural diagram of a computer processing device in one embodiment of this application;

[0036] Figure 2 This is a schematic diagram of a scatter plot in one embodiment of this application;

[0037] Figure 3 This is a flowchart of a scatter plot-based classification network-assisted analysis method in one embodiment of this application;

[0038] Figure 4 This is a flowchart illustrating the formation of a high-dimensional feature group in one embodiment of this application;

[0039] Figure 5 This is a flowchart illustrating the process of obtaining a low-dimensional feature set in one embodiment of this application;

[0040] Figure 6 This is a flowchart illustrating the process of determining the reasonableness of sample image classification in one embodiment of this application;

[0041] Figure 7 This is a schematic diagram illustrating the working principle of a classification network in one embodiment of this application;

[0042] Figure 8 This is a schematic diagram illustrating the working principle of the classification network in another embodiment of this application. DETAILED DESCRIPTION

[0043] The application will be described in further detail below with reference to the drawings. Like elements in different embodiments are denoted by like reference numerals. In the following description, numerous specific details are described to provide a thorough understanding of the application. However, those of skill in the art will recognize that the application can be practiced without one or more of the specific details. In other instances, well-known structures have not been described in order to avoid obscuring the application. In the following description, numerous specific details are described to provide a thorough understanding of the application. However, those of skill in the art will recognize that the application can be practiced without one or more of the specific details. In some instances, well-known structures have not been described in order to avoid obscuring the application.

[0044] In addition, features, operations, or steps described in the specification can be combined in any suitable manner without departing from the scope of the application. Similarly, the various steps or actions in a method can be combined into a single step or action or further subdivided into multiple steps or actions. Thus, the various sequences and / or temporal arrangements of steps or actions described in the specification can be combined in a variety of ways, and the order of each step or action can be rearranged without departing from the scope of the application.

[0045] The serial numbers of components in the specification, such as "first", "second", etc., are used only to distinguish the described objects and do not have any sequential or technical meaning. The "connection" and "coupling" in the specification include direct and indirect connections (couplings) unless otherwise specified.

[0046] In order to accurately understand the technical solutions and inventive concepts of the application, some neural network training techniques will be briefly described.

[0047] In order to construct a classification model, the neural network usually needs to be trained to learn the features of the sample image, such as adjusting the hyperparameters by observing the loss function value (loss), accuracy, and other indicators of the data set and validation set, and finally making the neural network converge to the lowest range as possible. In application, the training result of the neural network is not satisfactory, and qualitative analysis of the result cannot explain the main reason for the model collapse. The application can analyze the main reason for the model collapse by analyzing the network layer of the neural network, and in turn guide or optimize the architecture, hyperparameters, and iterative optimization strategy of the model.

[0048] The existing characterization training effect indicators include loss, precision, recall, confusion matrix, and F1-Score. These training values have a common limitation that as an abstract value, it is difficult to intuitively show the dynamic process of the neural network learning certain features from the data set. The technical scheme of the present application mainly analyzes and processes the feature map output in the training process to obtain visual analysis results of the model, such as scatter plot analysis results. For the scatter plot analysis results, the dimension of the output feature is n-dimensional, which needs to be displayed in a two-dimensional plane or a three-dimensional space, that is, reduced to 2-3 dimensions and displayed. The fully connected weight reflects the contribution degree of each feature map to the classification result, and the activation value on the feature map reflects the response of the feature in the corresponding region; then, multiplying the activation value of the feature map by the corresponding weight can obtain the contribution degree of each position of each feature map to a certain class. If all the feature maps are superimposed and summed, the contribution degree of a certain region to a certain class can be reflected, and whether the features possessed by the region have classification specificity (i.e., whether the region possesses features sufficient to distinguish a certain class from other classes) can be judged, so that whether the neural network has learned useful features can be judged.

[0049] Embodiment one,

[0050] Please refer to Figure 1 The computer processing device disclosed in the embodiment mainly includes a memory 11 and a processor 12, which are described below.

[0051] The memory 11 stores a data set and a classification network. The data set includes a plurality of sample images, that is, a data set of a plurality of sample images. The sample image herein refers to an image of a target object taken and labeled, which contains detailed information such as object shape, texture, and grayscale, and the labeled results of these information. The classification network includes at least one network layer, such as a convolution layer, a pooling layer, or a fully connected layer of a convolutional neural network, and the classification network is stored in the memory 11 in the form of an application program.

[0052] It should be noted that the memory 11 can be a flash card, a solid state memory, a hard disk, or other non-volatile storage medium, as long as it can store the data set of the sample image and the application program corresponding to the classification network.

[0053] The processor 12 communicates with the memory 11, can read the data set in the memory 11, and call and load the application program corresponding to the classification network. The processor 12 can train the classification network by using the data set, process to obtain the high-dimensional feature group and the low-dimensional feature group, construct the scatter plot based on the low-dimensional space, and judge the classification rationality of each sample image in the data set by means of the scatter plot and in combination with the classification category to which the low-dimensional feature vector belongs.

[0054] It should be noted that the processor 12 can be implemented by software, hardware, firmware or a combination thereof, and can use a circuit, a single or multiple application specific integrated circuits (ASIC), a single or multiple general integrated circuits, a single or multiple microprocessors, a single or multiple programmable logic devices, or a combination of the foregoing circuits or devices, or other suitable circuits or devices, so that the processor 12 can perform the algorithm processing process mentioned in the technical scheme of the present application.

[0055] In the embodiment, the processor 12 can include a training module 121, an acquisition module 122, a mapping module 123, a construction module 124 and an analysis module 125, which are described as follows.

[0056] The training module 121 can read each sample image in the data set from the memory 11, and can read and load the application program corresponding to the classification network from the memory 11, and then train the classification network by using the data set, so as to learn the features of the sample images and finally construct a classification model.

[0057] The acquisition module 122 can acquire each high-dimensional feature vector output by any network layer in the classification network during the training process of the training module 121, and form a high-dimensional feature group. It can be understood that the network layer usually has processing such as convolution, activation and pooling, which will change the scale of the image while extracting the features of the image, such as different dimensions of feature vectors after different scale convolution units.

[0058] The mapping module 123 can obtain each high-dimensional feature vector in the high-dimensional feature group from the acquisition module 122, perform linear mapping on each high-dimensional feature vector in the high-dimensional feature group, and obtain the low-dimensional feature vector corresponding to each high-dimensional feature vector by vector selection.

[0059] The construction module 124 can construct a scatter plot based on a low-dimensional space according to the low-dimensional feature vectors generated in the mapping module 123. It can be understood that the scatter plot includes the projection points of the low-dimensional feature vectors corresponding to each high-dimensional feature vector. If it is based on a two-dimensional space, each projection point in the scatter plot has the characteristic of two-dimensional plane distribution; if it is based on a three-dimensional space, each projection point in the scatter plot has the characteristic of three-dimensional space distribution.

[0060] The analysis module 125 can analyze the scatter plot constructed in the construction module 124, determine the classification category to which each projection point in the scatter plot corresponds according to the label information of the sample image corresponding to each projection point, and determine the classification rationality of each sample image in the data set according to the classification category to which the low-dimensional feature vector corresponding to each projection point belongs and the distribution state of each projection point.

[0061] In the embodiment, when the scatter plot is analyzed, a certain layer output of the neural network that has been vectorized is selected as the analysis object. Assuming that the selected feature vector is l-dimensional, the feature vector can be represented by a point in an l-dimensional coordinate system. In order to measure the separability of the data set, it is equivalent to measure the separability of the points in the high-dimensional space. Since the coordinate system dimension is limited to two or three dimensions when the scatter plot is visualized and output, the covariance matrix of the data points is calculated, and the data points can be projected into a 2-3 dimensional space according to the eigenvalues and eigenvectors.

[0062] For the classification network, it is assumed that there are m output l-dimensional feature vectors X = {x1, x2,...x i …,x m}, which can be regarded as high-dimensional data points distributed in an l-dimensional coordinate system. The separability of the data set can be determined by analyzing the separability of the data points, and then it can be determined whether the current data category division is reasonable. If it is not reasonable, it needs to be determined which categories need to be merged, and then further category analysis is performed through subsequent judgment and classification.

[0063] In the embodiment, in order to form a high-dimensional feature group, the acquisition module 122 can perform the following processing:

[0064] (1) The acquisition module 122 takes any network layer in the classification network as the analysis object, such as taking the convolution layer in the neural network as the analysis object.

[0065] (2) In the process of feature learning of each sample image in the data set by the classification network, the acquisition module 122 takes the feature map output by the network layer as the analysis object as a high-dimensional feature vector.

[0066] (3) Since each sample image will output a corresponding feature map after being processed by the network layer as the analysis object, the acquisition module 122 can obtain the high-dimensional feature vector corresponding to each sample image in the data set, and form a high-dimensional feature group by using the high-dimensional feature vectors. For example, the high-dimensional feature group is represented as X = {x1, x2,...x i …,x m}, where x iwherein xi represents the ith high-dimensional feature vector, and m represents the total number of the high-dimensional feature vectors.

[0067] In one embodiment, in order to form the low-dimensional feature set, the mapping module 123 can perform the following processing:

[0068] (1) The mapping module 123 respectively performs a centering processing on each high-dimensional feature vector in the high-dimensional feature set, to obtain a centering set and represented as X c It can be understood that the centering set X c can be represented in a matrix form.

[0069] (2) The mapping module 123 calculates a covariance matrix of the high-dimensional feature vectors according to the centering set, and the covariance matrix can be represented as ε = X c X c T / m, i.e. ε = 1 / m x X c X c T ; wherein m represents the total number of the high-dimensional feature vectors, and the superscript T represents a transpose operation of the matrix.

[0070] (3) The mapping module 123 obtains the eigenvectors of the covariance matrix and the corresponding eigenvalues, and arranges the eigenvectors in rows according to the size of the eigenvalues; since the number of the eigenvalues and the number of the eigenvectors are the same and one-to-one corresponding, the larger the eigenvalue is, the more the corresponding eigenvector is arranged in the column vector. It should be noted that the eigenvalue and the eigenvector are both mathematical concepts, which are basic parameters of the matrix, and will not be described in detail here; moreover, the eigenvectors of the covariance matrix have the same dimension as the high-dimensional feature vectors described above.

[0071] (4) The mapping module 123 obtains a plurality of rows of the eigenvectors arranged in front and constructs a matrix P; for example, the eigenvectors corresponding to the first 2 or 3 values of the eigenvalues are selected to form a matrix P.

[0072] (5) The mapping module 123 obtains the low-dimensional feature set by the multiplication operation of the matrix P and the centering set, and represented as Y = PX c It can be understood that the low-dimensional feature set Y includes the low-dimensional feature vectors corresponding to each high-dimensional feature vector after the vector selection.

[0073] In one embodiment, the construction module 124 constructs a scatter plot based on the low-dimensional space according to each low-dimensional feature vector in the low-dimensional feature set Y, and the low-dimensional space here can be a two-dimensional plane or a three-dimensional space. For example, the scatter plot based on the low-dimensional space constructed by the construction module 124 can refer to Figure 2 , Figure 2The diagram illustrates the projections of the corresponding low-dimensional feature vectors onto a two-dimensional plane, forming projection points. If the dataset contains several sample images of three categories, then the projections onto the two-dimensional plane of the high-dimensional feature vectors generated by the classification network and the low-dimensional feature vectors after dimensionality reduction are: Figure 2 In the case described, the three categories of the sample images correspond to... Figure 2 The three color depths in the image correspond to a certain category of a sample image. Figure 2 The projection point of a specific color depth in the image.

[0074] It should be noted that since image features that contribute significantly to variance in the dataset are preserved after dimensionality reduction, the resulting low-dimensional feature vectors not only reflect the distribution of the original data but can also be visually represented through projection points, thus facilitating the analysis of scatter plots.

[0075] In one specific embodiment, in order to analyze the scatter plot, the analysis module 125 may perform the following processing:

[0076] (1) The analysis module 125 determines the classification category to which the low-dimensional feature vectors corresponding to each projection point belong based on the annotation information of the sample images corresponding to each projection point in the scatter plot.

[0077] (2) For each sample image in the dataset, the analysis module 125 determines whether the sample image belongs to a classification category and its projection points are similar to those of sample images of the same category in the distribution (i.e., the distribution of projection points in the scatter plot). Otherwise, the sample image is an abnormal sample.

[0078] (3) When there are abnormal samples in the dataset, the analysis module 125 determines that the classification of each sample image in the dataset is not reasonable and removes the abnormal samples from the memory 11.

[0079] In one specific embodiment, the computer processing device may further include a display ( Figure 1 (Not shown in the image) The display is connected to the processor 12 and is used to display the sample images and scatter plots involved in the processor 12. It can also display the results of the reasonableness judgment of the category classification of each sample image in the dataset. The purpose is to let the user understand the process and results of the program processing and realize the visualization operation of the classification network training process.

[0080] In this embodiment, the disclosed technical solution uses scatter plots to visualize the projection points of the low-dimensional feature vectors corresponding to the high-dimensional feature vectors, thereby guiding the partitioning of the dataset. This not only enhances the visualization and analysis capabilities of multi-scale features but also improves the analytical capabilities of neural network feature extraction.

[0081] Embodiment Two,

[0082] The embodiment discloses a scatter plot-based classification network auxiliary analysis method, which can be applied to Figure 1 the processor 12.

[0083] In the embodiment, the auxiliary analysis method disclosed includes steps 210-260, which can be specifically referred to Figure 3 and described below respectively.

[0084] Step 210, the classification network is trained using a preset data set.

[0085] Step 220, in the training process of the classification network, each high-dimensional feature vector output by any network layer in the classification network is obtained, and a high-dimensional feature group is formed.

[0086] Step 230, linear mapping is performed on each high-dimensional feature vector in the high-dimensional feature group, and each low-dimensional feature vector corresponding to each high-dimensional feature vector is obtained through vector selection.

[0087] Step 240, a scatter plot based on a low-dimensional space is constructed. The scatter plot includes projection points of low-dimensional feature vectors corresponding to each high-dimensional feature vector.

[0088] Step 250, according to the label information of each projection point in the corresponding sample image in the scatter plot, the classification category to which each projection point corresponds is determined.

[0089] Step 260, according to the classification category to which each projection point corresponds and the distribution state of each projection point, the classification rationality of each sample image in the data set is determined.

[0090] In the embodiment, referring to Figure 4 , the above-mentioned step 220 mainly relates to the process of forming a high-dimensional feature group, which can specifically include steps 221-223, which are described below respectively.

[0091] Step 221, any network layer in the classification network is taken as an analysis object; for example, a convolutional layer in a neural network is taken as an analysis object.

[0092] Step 222, in the process of feature learning of the classification network on each sample image in the data set, the feature map output by the network layer taken as the analysis object is taken as a high-dimensional feature vector.

[0093] Step 223, by obtaining each high-dimensional feature vector corresponding to each sample image in the data set, a high-dimensional feature group is formed. For example, the high-dimensional feature group can be represented as X={x1,x2,...x i..., x m}, wherein x i represents the i-th high-dimensional feature vector, and m represents the total number of high-dimensional feature vectors.

[0094] In the embodiment, referring to Figure 5 , the step 230 mainly involves a process of obtaining low-dimensional features, which can specifically include steps 231-235, which are described as follows.

[0095] In step 231, each high-dimensional feature vector in the high-dimensional feature set is respectively subjected to a centering process to obtain a centering set and represented as X c .

[0096] For example, the centering process can be described as follows: 1) calculating the vector mean of each high-dimensional feature vector in the high-dimensional feature set; 2) obtaining the centering result of each high-dimensional feature vector by operating the difference between each high-dimensional feature vector and the vector mean, and represented as x i ′ = x i —∑x i / m; 3) counting the centering results corresponding to each high-dimensional feature vector to obtain the centering set, and represented as X c ={x1′,x2′,...x i ′...x m ′}.

[0097] In step 232, the covariance matrix of the high-dimensional feature vector is calculated according to the centering set, which can be represented as ε = X c X c T / m, i.e., ε = 1 / m × X c X c T ; wherein m represents the total number of high-dimensional feature vectors, and the superscript T represents the transpose operation of the matrix.

[0098] In step 233, the eigenvectors of the covariance matrix and the corresponding eigenvalues are obtained, and the eigenvectors are arranged in rows according to the size of the eigenvalues. Since the number of eigenvalues and eigenvectors is the same and one-to-one corresponding, the larger the eigenvalue is, the more the corresponding eigenvector is arranged in the column vector.

[0099] In step 234, the eigenvectors arranged in the front several rows are obtained and constitute a matrix P. For example, the eigenvectors corresponding to the first 2 or 3 values of the eigenvalues are selected to form a matrix P.

[0100] In step 235, the low-dimensional feature set is obtained by multiplying the matrix P and the centering set, and represented as Y = PX cIt can be understood that the low-dimensional feature set includes low-dimensional feature vectors corresponding to the high-dimensional feature vectors respectively after the vector selection.

[0101] It should be noted that the step 230 is essentially a linear mapping of each high-dimensional feature vector in the high-dimensional feature set to obtain a low-dimensional feature vector corresponding to each high-dimensional feature vector, and the key is to classify the mapped features through the centering process.

[0102] In the linear mapping of each high-dimensional feature vector in the high-dimensional feature set, the mapped features are classified through the centering process, including:

[0103] (1) Obtain the number of preset classification categories and denote it as k, randomly select k high-dimensional feature vectors as center points, traverse all data corresponding to each high-dimensional feature vector and calculate the distance from each data point to the center point, and divide each data point into the nearest center point group;

[0104] (2) Calculate the average value of each cluster according to each center point group, and use the new center point for iterative calculation, until the k center points converge or reach the preset training times, at which time the iterative calculation stops.

[0105] (3) Calculate the distance between each center point, if the distance between two center points is less than a preset threshold, then merge the two center points, and obtain new center points by merging until the center points cannot be merged any more;

[0106] (4) Determine whether the number of categories is equal to the number of classification categories according to the current number of center points, if yes, the classification of the mapped features is completed; wherein, for the low-dimensional feature vectors obtained by mapping, the corresponding abnormal boundary is obtained and the abnormal boundary is expressed as d i = x i ′ T Σ -1 x′; wherein, Σ represents a covariance matrix, if d i is less than a specified threshold, the abnormal boundary is marked as a first color (such as green) to represent normal classification, otherwise the abnormal boundary is marked as a second color (such as red) to represent abnormal classification.

[0107] It can be understood that in the centralization process, k center points are selected randomly, where k is determined by the number of categories; then, all data are traversed and the distance of each point to the center point is calculated, and each data is divided into the nearest center point; the new center point can be obtained by calculating the average of each cluster; as long as the data is continuously traversed and the new center point is determined, the k center points no longer change (that is, converge) through multiple iteration calculations, so as to complete the centralization process. After the above iteration is performed, the distance of each center point is calculated, if the distance of two center points is less than a preset threshold, the two results are merged, and the new center point is calculated after merging, until no merging position; in addition, it is also necessary to judge whether the number of categories is equal to the number of categories.

[0108] It should be noted that the boundary of each category can be obtained by calculating the distance of each center point, and then the stability of the division can be judged according to the boundary.

[0109] Case one, for the mapping of two-dimensional plane, the feature of each mapping is represented as

[0110]

[0111]

[0112] Further calculation can obtain

[0113]

[0114]

[0115]

[0116] In the two-dimensional plane, the low-dimensional feature vector obtained by mapping is obtained, and the corresponding effective boundary is represented by the formula

[0117]

[0118] Wherein, n represents the first moment or mean, μ represents the second central moment, λ represents the long axis and short axis of the ellipse, and δ represents the distance threshold.

[0119] In this way, the center point is constructed as (n 1,0 , n 0,1 ), the radius is k x λ1, k x λ2, and the angle with the X-axis direction is θ. Usually k = 3, and the data points falling within the ellipse are normal data points.

[0120] Case two, for the mapping of three-dimensional space, the matrix is set as

[0121]

[0122] wherein,

[0123]

[0124]

[0125] In the three-dimensional space, the distance d j is calculated and expressed as

[0126] d j = (X j - μ) T H -1 (X j - μ) ;

[0127] wherein, h represents a second order central distance, d represents a distance, N represents the number of feature vectors in a specific category, X represents the jth feature vector, and μ represents the average value of feature vectors in a group.

[0128] It can be understood that if the distance d j is less than a certain threshold, the data within the effective boundary corresponding to the low-dimensional feature vector obtained by mapping is normal data, which is equivalent to the corresponding data being located within the ellipsoid.

[0129] In the embodiment, the step 240 mainly involves the process of constructing a scatter plot, which can be represented in a two-dimensional plane or in a three-dimensional space, and the purpose is to realize the visualization of the projection points in the scatter plot, because more-dimensional space is not easy to be visually represented. For example, in Figure 2 , the projections of the low-dimensional feature vectors corresponding to the high-dimensional feature vectors in the two-dimensional plane are illustrated, forming a plurality of projection points. Figure 2 The three color depths in represent the sample images of the three categories in the data set, and each projection point of a specific color depth corresponds to a sample image of a certain category in the data set.

[0130] In the embodiment, the step 250 mainly involves the process of determining the classification category to which the low-dimensional feature vector belongs, and the method adopted is to determine the classification category to which the low-dimensional feature vector corresponding to each projection point belongs according to the annotation information of the sample image corresponding to each projection point in the scatter plot. It can be understood that since each projection point corresponds to a sample image and the sample image has been annotated in advance, the annotation information of the sample image corresponding to each projection point reveals the classification category to which the low-dimensional feature vector corresponding to the projection point belongs.

[0131] Figure 6 In the embodiment, referring to , the step 260 mainly involves the process of judging the rationality of classification, which can specifically include steps 261-264, which are described as follows.

[0132] Step 261, for each sample image in the data set, if the sample image belongs to a classification category and is close to the projection point generated by the sample image of the same category in distribution, then the sample image is a normal sample; otherwise, the sample image is an abnormal sample.

[0133] Step 262, according to the manner of step 261, the normal or abnormal situation corresponding to each sample image in the data set can be obtained to obtain a sample situation record, and then it can be judged whether there is an abnormal sample in the sample situation record, if yes, go to step 263, otherwise go to step 264.

[0134] Step 263, when there is an abnormal sample in the data set, it is determined that the category division of each sample image in the data set is not reasonable, and the abnormal sample is removed. It can be understood that after removing the abnormal sample, the classification rationality of the remaining sample images in the data set is ensured, and then the remaining sample images are used to train the classification network, which can optimize the training process of the network, make the feature learning ability more reliable, and the feature learning result more accurate, and finally train a classification model with excellent performance.

[0135] Step 264, when there is no abnormal sample in the data set, i.e. all are normal samples, it is determined that the category division of each sample image in the data set is reasonable. In the case that the category division of each sample image in the data set is reasonable, each sample image in the data set can be directly used to train the classification network.

[0136] In the embodiment, the disclosed technical solution can determine the classification rationality of each sample image in the data set by means of the classification category to which each projection point in the scatter plot corresponds and the distribution state in the plane or space, which can qualitatively guide and analyze the classification network structure, data set division, training strategy and the like, and is beneficial to optimizing the training process of the classification model.

[0137] In one specific embodiment, the data set participating in the training of the classification network includes a plurality of sample images, the sample image is an image of a target object taken and labeled, and the image contains detailed information of the object shape, texture, gray scale and the like and the labeling results of the information. Moreover, the classification network here includes at least one network layer, such as the convolution layer, the pooling layer and the full connection layer of the neural network. It should be noted that the classification network can be a general convolutional neural network, such as Figure 7The network structure is shown in FIG. 1, and the principle of processing sample images by means of a convolutional neural network is also shown. The process mainly includes three stages. In the feature extraction stage, a convolutional network layer is usually used to extract image features to obtain a feature map. In the feature map vectorization stage, a global pooling operation is usually used to convert the feature map into a feature vector. In the classification stage, a fully connected network layer is usually used to process the feature vector to obtain the confidence of each classification result. Figure 7 In the convolutional neural network, each sample image is input into the network, and after feature extraction processing by one or more network layers of convolution, activation, and pooling, a feature map corresponding to each sample image is obtained. By extracting the feature, a corresponding feature vector is obtained. The feature vector is sent to a fully connected layer for estimation of the class confidence. The class confidence corresponding to the feature vector is obtained. Then, according to the class confidence, a class feature map can be obtained. Figure 7 In the convolutional neural network, each network layer is analyzed as an analysis object, so that the auxiliary analysis is performed, and all abnormal samples in the data set are excluded, thereby ensuring the rationality of the classification of the remaining sample images.

[0138] In one embodiment, the classification network can be a specially designed convolutional neural network, such as the network structure shown in FIG. 1. Figure 8 The network structure is shown in FIG. 1, and the principle of processing sample images by means of a convolutional neural network is also shown. The process mainly includes three stages. In the feature extraction stage, a convolutional network layer is usually used to extract image features to obtain a feature map. In the feature map vectorization stage, a global pooling operation is usually used to convert the feature map into a feature vector. In the classification stage, a fully connected network layer is usually used to process the feature vector to obtain the confidence of each classification result. Figure 8 In the convolutional neural network, each sample image is input into the network, and after feature extraction processing by one or more network layers of convolution, activation, and pooling, a feature map corresponding to each sample image is obtained. By extracting the feature, a corresponding feature vector is obtained. The feature vector is sent to a fully connected layer for estimation of the class confidence. The class confidence corresponding to the feature vector is obtained. Then, according to the class confidence, a class feature map can be obtained. Figure 8The two scale size feature maps help to obtain image information of multiple scales. It can be understood that, since each network layer in the convolutional neural network can cause training failure or model collapse, each network layer in the convolutional neural network can be taken as an analysis object respectively, so as to perform the foregoing auxiliary analysis, and then exclude all abnormal samples in the data set, and ensure the rationality of the class division of the remaining sample images. Figure 8

[0139] Those skilled in the art can understand that all or part of the functions of the various methods in the above embodiments can be realized by hardware or by a computer program. When all or part of the functions in the above embodiments are realized by a computer program, the program can be stored in a computer readable storage medium, and the storage medium can include a read-only memory, a random access memory, a magnetic disk, an optical disk, a hard disk, and the like. The above functions are realized by executing the program by a computer. For example, the program is stored in a memory of a device, and when the program in the memory is executed by a processor, the above functions are realized. In addition, when all or part of the functions in the above embodiments are realized by a computer program, the program can also be stored in a server, another computer, a disk, an optical disk, a flash disk, a mobile hard disk, or the like, and is downloaded or copied into a memory of a local device, or the system of the local device is updated, and when the program in the memory is executed by a processor, the above functions are realized.

[0140] The above application uses specific examples to describe the present application, which is only used to help understand the technical solutions of the present application, and does not limit the present application. According to the idea of the present application, those skilled in the art can make several simple deductions, deformations or substitutions.​

Claims

1. A method for scatter plot-based classification network assisted analysis, characterized in that, The method comprises the following steps: training a classification network by using a preset data set; the data set comprises a plurality of sample images, and the classification network comprises at least one network layer; obtaining each high-dimensional feature vector output by any network layer in the classification network and forming a high-dimensional feature group; performing linear mapping on each high-dimensional feature vector in the high-dimensional feature group to obtain a low-dimensional feature vector corresponding to each high-dimensional feature vector through vector selection; constructing a scatter plot based on a low-dimensional space; the scatter plot comprises projection points of the low-dimensional feature vectors corresponding to each high-dimensional feature vector; determining the classification category to which the low-dimensional feature vector corresponding to each projection point belongs according to the label information of the sample image corresponding to each projection point in the scatter plot; judging the classification rationality of each sample image in the data set according to the classification category to which the low-dimensional feature vector corresponding to each projection point belongs and the distribution state of each projection point; wherein, when performing linear mapping on each high-dimensional feature vector in the high-dimensional feature group, the mapped features are classified through centering processing, comprising: obtaining the number of preset classification categories and denoting it as k; randomly selecting k high-dimensional feature vectors as center points, traversing all data corresponding to each high-dimensional feature vector and calculating the distance from each data point to the center point, and dividing each data point into the nearest center point group; calculating the average value of each cluster according to each center point group and taking it as a new center point, and using the new center point to perform iterative calculation again until the k center points converge or a preset training number is reached, at which time the iterative calculation stops; calculating the distance between each center point, if the distance between two center points is less than a preset threshold, merging the two center points, and obtaining new center points through merging until the center points cannot be merged any more; judging whether the number of categories is equal to the number of classification categories according to the current number of center points, and if yes, completing the classification of the mapped features; wherein, for the low-dimensional feature vectors obtained through mapping, obtaining the corresponding abnormal boundary, and the abnormal boundary is represented as d i = x i ' T ∑ -1 x i ' wherein x i represents a centering result of each high-dimensional feature vector in the high-dimensional feature set, and Σ represents a covariance matrix, and if d i If the distance is less than a distance threshold, the abnormal boundary is marked as a first color to represent normal classification, and otherwise, the abnormal boundary is marked as a second color to represent abnormal classification.

2. The method of claim 1, wherein, the method comprises the following steps: taking any network layer in the classification network as an analysis object; in the process of feature learning of the classification network on each sample image in the data set, taking the feature map output by the network layer taken as the analysis object as a high-dimensional feature vector; By obtaining the high-dimensional feature vectors corresponding to each sample image in the dataset, a high-dimensional feature group is formed, and the high-dimensional feature group is represented as X = {x1, x2, ... x}. i …,x m }, where x i Let represent the i-th high-dimensional feature vector, and m represent the total number of all high-dimensional feature vectors.

3. The method of claim 2, wherein, the method comprises the following steps: respectively, to obtain a centralized set and denoted as X c ; A covariance matrix of the high-dimensional feature vectors is computed from the centralized set, denoted as ε = X c X c T / m; obtaining the eigenvectors and corresponding eigenvalues of the covariance matrix, and arranging the eigenvectors by rows according to the size of the eigenvalues; obtaining a plurality of rows of eigenvectors arranged in front and forming a matrix P; A low-dimensional feature group is obtained by a multiplication operation of the matrix P and the centralized set, and is expressed as Y=PX c The low-dimensional feature group includes low-dimensional feature vectors corresponding to the high-dimensional feature vectors respectively after vector selection.

4. The method of claim 3, wherein, the centering processing comprises: calculating the vector mean of each high-dimensional feature vector in the high-dimensional feature group; By operating the difference between each high-dimensional feature vector and the vector mean, a centralization result of each high-dimensional feature vector is obtained and denoted as x i = x i = x i / m; The centralization set is obtained by counting the centralization results corresponding to each high-dimensional feature vector respectively, and is denoted as X c = {x1′, x2′,... x i ′..., x m ′}.

5. The method of claim 1, wherein, for the mapping of a two-dimensional plane, each mapped feature is represented as further calculation obtains in the two-dimensional plane, the effective boundary corresponding to the low-dimensional feature vector obtained through mapping is represented by the formula where n p,q denotes the first moment or mean, μ p,q denotes the second central moment, λ1and λ2denote the major and minor axes of the ellipse, respectively, and δ denotes the distance threshold.

6. The method of claim 1, wherein, for the mapping of a three-dimensional space, a matrix is set and represented as wherein the parameters of the matrix are represented as In the three-dimensional space, the distance d is calculated j and is expressed as d j = (X j - μ) T ∑ -1 (X j - μ); wherein h represents a second order center distance, d represents a distance, N represents the number of feature vectors in a specific category, X represents the jth feature vector, and μ represents the average value of the feature vectors in the group; If the distance d j is less than a certain threshold, the low-dimensional feature vector obtained by mapping corresponds to normal data within the effective boundary.

7. The method of claim 3, wherein, the classification rationality of each sample image in the data set is determined according to the classification category to which each projection point corresponding to the low-dimensional feature vector respectively belongs and the distribution state of each projection point, and the classification rationality comprises: for each sample image in the data set, if the sample image belongs to a classification category and the distance between the center point of the projection point generated by the sample image and the projection point generated by the sample image of the sample image of the same category is less than the distance threshold, the sample image is a normal sample; otherwise, the sample image is an abnormal sample; when there is an abnormal sample in the data set, it is determined that the classification division of each sample image in the data set is not reasonable, and the abnormal sample is removed.

8. A computer processing device, characterized by, comprise: a memory storing a data set and a classification network, the data set comprising a plurality of sample images, and the classification network comprising at least one network layer; a processor comprising: a training module configured to train the classification network using the data set; an acquisition module configured to acquire each high-dimensional feature vector output by any network layer in the classification network and form a high-dimensional feature group; a mapping module configured to perform linear mapping on each high-dimensional feature vector in the high-dimensional feature group to obtain a low-dimensional feature vector corresponding to each high-dimensional feature vector through vector selection; a construction module configured to construct a scatter plot based on a low-dimensional space, wherein the scatter plot comprises projection points of the low-dimensional feature vectors corresponding to each high-dimensional feature vector; an analysis module configured to determine the classification category to which each projection point corresponding to the low-dimensional feature vector respectively belongs according to the annotation information of the sample image corresponding to each projection point in the scatter plot, and determine the classification rationality of each sample image in the data set according to the classification category to which each projection point corresponding to the low-dimensional feature vector respectively belongs and the distribution state of each projection point; wherein, when performing linear mapping on each high-dimensional feature vector in the high-dimensional feature group, the mapping module classifies the mapped features through centering processing, and the classification comprises: acquiring the number of preset classification categories and denoting the number as k; randomly selecting k high-dimensional feature vectors as center points, traversing all data corresponding to each high-dimensional feature vector, calculating the distance from each data point to the center points, and dividing each data point into the nearest center point group; calculating the average value of each cluster according to each center point group and taking the average value as a new center point, and performing iterative calculation using the new center point until the k center points converge or a preset number of training times is reached, at which point the iterative calculation stops; calculating the distance between each center point, and if the distance between two center points is less than a preset threshold, merging the two center points, and obtaining new center points through merging until the center points cannot be merged any more; determining whether the number of categories is equal to the number of classification categories according to the current number of center points, and if so, completing the classification of the mapped features; wherein, for the low-dimensional feature vector obtained through mapping, an abnormal boundary is obtained and the abnormal boundary is represented as wherein x i represents a centering result of each high-dimensional feature vector in the high-dimensional feature set, and Σ represents a covariance matrix, and if d i If the distance is less than a distance threshold, the abnormal boundary is marked as a first color to represent normal classification, and otherwise, the abnormal boundary is marked as a second color to represent abnormal classification.

9. A computer-readable storage medium, characterized in that, The medium has stored thereon a program that can be executed by the processor to implement the method of any one of claims 1-7.

Images