Scene image classification method and system based on dual hypergraph neural network

Abstract

The present invention discloses a scene image classification method and system based on a dual hypergraph neural network. The method comprises: obtaining a corresponding feature matrix based on an acquired intelligent video surveillance scene image to be classified; inputting the feature matrix into a multi-view hypergraph learning network of a trained scene image classification model; fusing hypergraphs learned by the multi-view hypergraph learning network on multiple views to obtain a hypergraph association matrix, wherein each view uses a different learnable similarity metric function; inputting the feature matrix and the hypergraph association matrix into a density-aware hypergraph attention network of the trained scene image classification model; utilizing a density-aware attention mechanism to mine density information in the data for hypergraph representation learning, and obtaining a category prediction result for the scene image to be classified. The present invention utilizes a dual hypergraph neural network to effectively combine the multi-view hypergraph learning network and the density-aware hypergraph attention network, thereby achieving higher scene image classification performance.

Description

Technical Field

[0001] The present invention relates to the technical field of scene image classification, and in particular to a scene image classification method, system, electronic device and storage medium based on a dual hypergraph neural network. Background Art

[0002] Scene image classification can intelligently identify and classify events and scene data collected in intelligent video surveillance, thereby improving its efficiency. However, current mainstream scene image classification technologies tend to process each scene image individually, ignoring the common features between similar scene images. As a result, the feature information they can obtain is relatively limited, especially when there are few labeled samples. This significantly limits the performance of scene image classification.

[0003] Graph neural networks (GNNs) have become a hot topic of research due to their ability to effectively mine and process graph-structured information in data. GNNs have been widely used in a variety of research fields, including document classification, image classification, personalized recommendations, and risk prediction. GNNs not only process the feature information of individual samples but also consider the correlations between samples, thereby obtaining more useful information than processing individual samples in isolation. Therefore, applying GNNs to scene image classification in intelligent video surveillance can effectively overcome these limitations. GNN-based scene image classification methods not only process the feature information of individual scene image samples but also effectively leverage the correlations between samples to mine common features across similar scene images. Therefore, compared to traditional methods, GNN-based scene image classification methods can obtain more useful information, particularly when labeled samples are limited, and can effectively improve scene image classification performance.

[0004] Based on this idea, a number of related methods have been studied. Most of these methods are based on artificially constructed graph structures, such as the k-nearest neighbor graph. However, these artificially constructed k-nearest neighbor graphs are primarily based on a fixed, single similarity metric function, such as the Euclidean distance, which is not suitable for accurately measuring the similarity between all samples. Furthermore, these artificially constructed graphs are primarily based on the original feature space of the samples, which often contains redundant information. Therefore, these artificially constructed graphs cannot accurately reflect the associations between samples. Furthermore, these artificially constructed graphs are suboptimal and unsuitable for downstream graph neural network-based scene image classification networks, thus limiting the performance of these downstream graph neural network-based scene image classification models. Although a small number of learnable graph modeling methods have emerged, these methods still use a single similarity metric function to measure the similarity between samples, which is still not accurate enough for measuring the similarity between samples. Therefore, how to comprehensively and accurately model scene image progress graphs remains a challenging problem.

[0005] Furthermore, most existing graph neural network-based scene image classification methods focus solely on pairwise relationships between data. However, in real applications, the relationships between samples are not limited to pairwise relationships; they are often more complex, one-to-many or many-to-many multivariate relationships. Exploring only pairwise relationships between samples and modeling them as graphs loses high-level semantic connections between data, especially for complex data sets such as image datasets. Furthermore, existing graph-based scene image classification methods only utilize superficial graph structural information and lack in-depth exploration and utilization of the implicit information within the graph data. Summary of the Invention

[0006] To address the shortcomings of the aforementioned prior art, the present invention provides a scene image classification method, system, electronic device, and storage medium based on a dual hypergraph neural network. The dual hypergraph neural network comprises a multi-view hypergraph learning network and a density-aware hypergraph attention network. This dual hypergraph neural network is capable of simultaneously utilizing both the feature information and the interrelationships between scene images, thereby obtaining more useful information than processing a single scene image alone. By effectively combining the multi-view hypergraph learning network and the density-aware hypergraph attention network, the dual hypergraph neural network achieves higher scene image classification performance.

[0007] The first object of the present invention is to provide a scene image classification method based on a dual hypergraph neural network.

[0008] The second object of the present invention is to provide a scene image classification system based on a dual hypergraph neural network.

[0009] A third object of the present invention is to provide an electronic device.

[0010] A fourth object of the present invention is to provide a storage medium.

[0011] The first object of the present invention can be achieved by adopting the following technical solutions:

[0012] A scene image classification method based on a dual hypergraph neural network, the method comprising:

[0013] Obtain a scene image to be classified; and obtain a feature matrix corresponding to the scene image based on the scene image to be classified; the scene image to be classified is at least one, and all of them are intelligent video surveillance scene images;

[0014] Inputting the feature matrix into a multi-view hypergraph learning network of a trained dual-hypergraph neural network, and fusing the hypergraphs learned by the multi-view hypergraph learning network on multiple views to obtain a hypergraph association matrix; each view in the multi-view hypergraph learning network uses a different learnable similarity metric function;

[0015] The feature matrix and the hypergraph association matrix are input into the density-aware hypergraph attention network of the trained dual-hypergraph neural network, and the density-aware attention mechanism is used to mine the density information in the data for hypergraph representation learning to obtain the category prediction results of the scene image to be classified; the density-aware attention mechanism is used to fuse the density information of vertices / hyperedges in the hypergraph into the vertex / hyperedge density attention.

[0016] Furthermore, the feature matrix is ​​input into the multi-view hypergraph learning network of the trained dual-hypergraph neural network, and the hypergraphs learned by the multi-view hypergraph learning network on multiple views are fused to obtain a hypergraph association matrix, including:

[0017] Using a multi-view fusion module, the hypergraphs learned on each view in the multi-view hypergraph learning network are fused to obtain a first hypergraph association matrix; each node in the hypergraph represents a scene image, and the hyperedges in the hypergraph represent high-order association relationships between scene images;

[0018] Calculating the similarity between scene images based on the feature matrix; constructing hyperedges based on the similarity to obtain an original hypergraph association matrix;

[0019] The first hypergraph association matrix and the original hypergraph association matrix are merged to obtain a hypergraph association matrix.

[0020] Furthermore, the hypergraph learned on any view in the multi-view hypergraph learning network includes:

[0021] Mapping the feature matrix from the original feature space to a low-dimensional embedding space using a low-dimensional embedding module;

[0022] Using the similarity learning module, the similarity between scene images is learned through the similarity measurement function in the low-dimensional embedding space;

[0023] The hyperedge construction module is used to construct hyperedges based on the learned similarity to obtain the hypergraph structure learned on the view.

[0024] Furthermore, the learning of the similarity between scene images by using a similarity measurement function in the low-dimensional embedding space includes:

[0025] The similarity between scene images is calculated through the similarity measurement function to obtain the similarity matrix;

[0026] Sparse sampling is performed on the similarity matrix to filter out similarities below a first preset threshold.

[0027] Furthermore, the method further comprises:

[0028] According to the feature matrix and the hypergraph association matrix, a vertex feature matrix and a hyperedge feature matrix are obtained by utilizing a hypergraph information propagation mechanism.

[0029] Furthermore, the feature matrix and the hypergraph association matrix are input into the density-aware hypergraph attention network of the trained dual-hypergraph neural network, and the density-aware attention mechanism is used to mine the density information in the data for hypergraph representation learning to obtain the category prediction result of the scene image to be classified, including:

[0030] Inputting the vertex feature matrix and the hyperedge feature matrix into the first density-aware hypergraph attention layer in the density-aware hypergraph attention network, and obtaining the feature embedding of the layer using the density-aware attention mechanism;

[0031] The features obtained by the first density-aware hypergraph attention layer are embedded in the second density-aware hypergraph attention layer in the input density-aware hypergraph attention network, and the category result is predicted based on the output of the second density-aware hypergraph attention layer.

[0032] Furthermore, inputting the vertex feature matrix and the hyperedge feature matrix into the first density-aware hypergraph attention layer in the density-aware hypergraph attention network and obtaining the feature embedding of the layer using the density-aware attention mechanism includes:

[0033] Calculating the density of vertices and hyperedges based on the vertex feature matrix and the hyperedge feature matrix according to the hypergraph density rule;

[0034] The first density-aware attention mechanism designed based on vertex density fuses the density of vertices and the attention between vertices and hyperedges to obtain a vertex density-aware attention matrix; vertex features are aggregated according to the vertex density-aware attention matrix to obtain an updated hyperedge feature matrix; wherein the first density-aware attention mechanism considers the distribution information of vertices;

[0035] A second density-aware attention mechanism designed based on hyperedge density fuses the density of hyperedges and the attention between hyperedges and vertices to obtain a hyperedge density-aware attention matrix; hyperedge features are aggregated based on the hyperedge density-aware attention matrix and the updated hyperedge feature matrix to obtain an updated vertex feature matrix; wherein the second density-aware attention mechanism considers the distribution information of hyperedges;

[0036] The updated vertex feature matrices obtained by each attention head in the first density-aware hypergraph attention layer are concatenated to obtain the feature embedding of this layer.

[0037] Furthermore, the hypergraph density rule includes a vertex density rule and a hyperedge density rule, wherein:

[0038] Vertex density rule: The density of a vertex is the sum of the similarities of neighboring nodes whose similarity to the target node is greater than a second preset threshold;

[0039] Hyperedge density rule: The density of a hyperedge is the sum of the densities of the vertices connected by the hyperedge.

[0040] The second object of the present invention can be achieved by adopting the following technical solutions:

[0041] A scene image classification system based on a dual hypergraph neural network, the system comprising:

[0042] A scene image acquisition module is used to acquire a scene image to be classified; based on the scene image to be classified, a feature matrix corresponding to the scene image is obtained; the scene image to be classified includes at least one intelligent video surveillance scene image;

[0043] A hypergraph learning module is configured to input the feature matrix into a multi-view hypergraph learning network of a trained dual-hypergraph neural network, and to obtain a hypergraph association matrix by fusing hypergraphs learned by the multi-view hypergraph learning network on multiple views; each view in the multi-view hypergraph learning network uses a different learnable similarity metric function;

[0044] The category prediction module is used to input the feature matrix and the hypergraph association matrix into the density-aware hypergraph attention network of the trained dual-hypergraph neural network, and use the density-aware attention mechanism to mine the density information in the data to perform hypergraph representation learning to obtain the category prediction results of the scene image to be classified; the density-aware attention mechanism is used to fuse the density information of vertices / hyperedges in the hypergraph into the vertex / hyperedge density attention.

[0045] The third object of the present invention can be achieved by adopting the following technical solutions:

[0046] An electronic device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned scene image classification method is implemented.

[0047] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0048] A storage medium stores a program, which, when executed by a processor, implements the above-mentioned scene image classification method.

[0049] The present invention has the following beneficial effects compared to the prior art:

[0050] The present invention provides a scene image classification method, device, computer equipment and storage medium based on a dual hypergraph neural network. The dual hypergraph neural network is composed of a multi-view hypergraph learning network and a density-aware hypergraph attention network. The multi-view hypergraph learning network is used to learn the hypergraph structure on multiple views, and a variety of learnable similarity measurement methods are used to accurately measure the similarity between scene images, so that the learned hypergraph structure can accurately model the high-order semantic relationship between scene images; the density-aware hypergraph attention network can effectively mine the density information implicit in the hypergraph structure, and the density-aware attention mechanism can achieve more accurate neighborhood feature aggregation, thereby improving the accuracy of scene image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0052] Figure 1 This is a schematic diagram of the principle of the scene image classification method based on a dual hypergraph neural network in Example 1 of the present invention.

[0053] Figure 2 This is a flowchart of a scene image classification method based on a dual hypergraph neural network according to embodiment 1 of the present invention.

[0054] Figure 3 Schematic diagram of the structure of the dual hypergraph neural network of Example 1 of the present invention.

[0055] Figure 4 Schematic diagram of the structure of the multi-view hypergraph neural network of Example 1 of the present invention.

[0056] Figure 5 This is a structural diagram of the density-aware hypergraph attention network of Example 1 of the present invention.

[0057] Figure 6 This is a structural block diagram of a scene image classification system based on a dual hypergraph neural network according to embodiment 2 of the present invention.

[0058] Figure 7 This is a structural block diagram of an electronic device according to embodiment 3 of the present invention. DETAILED DESCRIPTION

[0059] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. It should be understood that the specific embodiments described are only used to explain this application and are not used to limit this application.

[0060] Example 1:

[0061] like Figure 1 As shown, this embodiment provides a scene image classification method based on a dual hypergraph neural network, which mainly includes data construction, model structure design, model training, and model application. Among them, data construction is mainly based on intelligent video scene image data. The feature matrix and label matrix of the scene image are obtained according to the scene image data. The feature matrix and label matrix constitute the data set, and the feature matrix serves as the input of the model; the model structure design is mainly based on the dual hypergraph neural network framework, and the corresponding multi-view hypergraph learning network and density-aware hypergraph attention network are designed to accurately model the scene image hypergraph and deeply mine and utilize the density information of implicit information in the data to enhance the scene image classification performance; among them, the multi-view hypergraph learning network learns the hypergraph structure from multiple views, so that the learned hypergraph can more accurately model the high-order semantic relationship between samples; the density-aware hypergraph attention network is designed with a density-aware attention mechanism module, which mines and utilizes the density information in the data and realizes more accurate neighborhood feature aggregation based on the density-aware attention mechanism; model training is to train the designed model structure using the data set and adjust the model parameters according to the value of the loss function; model application is to apply the trained model to actual testing, that is, output the classification result of the scene image based on the input scene image data.

[0062] This example is based on the Pytorch framework and the Pycharm development environment: Pytorch is a deep learning development framework based on the Python language. It provides interfaces for numerous encapsulated functions and image processing functions used in deep learning, making it easy for users to build deep learning networks and enabling GPU-accelerated graphics computing. The Pycharm integrated development environment is one of the primary tools for Python programming, providing programming, testing, and debugging tools, and can call compilers on remote servers to run programs.

[0063] like Figure 2 As shown, the scene image classification method based on the dual hypergraph neural network provided in this embodiment specifically includes the following steps:

[0064] S201: Acquire a scene image dataset, and obtain a feature matrix and a label matrix corresponding to the scene image according to the scene image dataset.

[0065] This embodiment is based on the collection of scene images by intelligent video surveillance equipment, and all scene images constitute a scene image dataset. The scene image is encoded using the LC-KSVD algorithm to obtain a sparse coding vector of the scene image as the feature vector of the scene image. The above operation is performed on all samples in the scene image dataset, and then the feature vectors of all scene images are stacked to obtain a feature matrix of the scene image data. For the label of the scene image sample, it is converted into a one-hot vector representation, which is the label vector of the scene image. The labels of all labeled samples are subjected to the above operation to finally obtain a label matrix of the scene image data. The feature matrix and label matrix of the scene image data together constitute the data required by the model. The samples in the scene image dataset can be one scene image or multiple scene images.

[0066] In one embodiment, the sparse coding vector dimension of the scene image obtained by the LC-KSVD algorithm is 3000 dimensions, and the scene image data set used contains a total of 4485 samples, so the dimension of the feature matrix finally constructed is 4485×3000. For the labels of the scene image samples, they are converted into one-hot vector representations, and the target category of the scene image is marked with "1", and other categories are marked with "0" to obtain the label vector of the scene image, and all label vectors are stacked to obtain the label matrix of the scene image data. In this embodiment, the number of labeled samples is 500, and the number of scene image categories is 15, so the dimension of the label matrix finally obtained is 500×15.

[0067] S202. Build a scene image classification model based on a dual hypergraph neural network.

[0068] like Figure 2 As shown in Figure 2, the main network of this scene image classification model mainly consists of a multi-view hypergraph learning network and a density-aware hypergraph attention network. The multi-view hypergraph learning network in the main network takes the scene image feature matrix as input and outputs a hypergraph association matrix modeled by the scene image. Its structure adopts a multi-view hypergraph learning mechanism to learn the hypergraph structure from multiple views, making the hypergraph modeling of scene image data more accurate. The density-aware hypergraph attention network takes the scene image feature matrix and the hypergraph association matrix as input and outputs the classification structure of the scene image. Its structure adopts a density-aware hypergraph attention mechanism to deeply explore and utilize the density information implicit in the data to improve the classification performance of scene images.

[0069] (1) Multi-view hypergraph learning network.

[0070] like Figure 3 As shown, the multi-view hypergraph learning network in this embodiment is used to model the hypergraph structure of scene image data. Its input is scene image features, and its output is a hypergraph association matrix. Structurally, the multi-view hypergraph learning network primarily learns the hypergraph structure from multiple views and is primarily composed of a low-dimensional embedding module, a similarity learning module, a hyperedge construction module, and a multi-view fusion module.

[0071] (1-1) Hypergraph structure learned for each view.

[0072] First, a low-dimensional embedding module is used to map scene image features from the original feature space to a low-dimensional embedding space, which is expressed by the following formula:

[0073]

[0074] Among them, X0 is the original feature matrix of the scene image, P is the learnable embedding mapping matrix, That is the scene image feature embedding.

[0075] In one embodiment, the low-dimensional embedding module is implemented by a fully connected layer with 70 neurons.

[0076] Then, the similarity learning module is used to learn the similarity between samples through the similarity measurement function in the low-dimensional embedding space, which can be described by the following formula:

[0077]

[0078] Among them, sim(·,·) is the similarity measurement function; and are the feature embeddings of scene image samples i and j, respectively, where i, j = 0, 1, ..., N-1; the obtained S ij That is, the similarity between scene image samples i and j.

[0079] The similarity learning module also introduces sparse sampling of the learned sample similarity matrix to filter out similarities below the preset threshold δ1, which is defined as follows:

[0080]

[0081] In one embodiment, the network performs hypergraph learning on two views, and the network classifies samples in the two views using two similarity metrics, namely, cosine similarity and inner product similarity, respectively, with a preset threshold δ1 of 0.7 in both views.

[0082] Finally, the hyperedge building module is used to build the similarity matrix based on the learned Perform hyperedge construction, construct each column of the similarity matrix into a hyperedge, and obtain the hypergraph structure learned on this view.

[0083] In this embodiment, a hypergraph is used to model high-order relationships between scene images. Each node in the hypergraph represents a scene image, and the node's features are the features of its corresponding scene image. Hyperedges in the hypergraph represent high-order relationships between scene images. The dual hypergraph neural network uses both scene image feature information and high-order relationship information to predict the category of the scene image.

[0084] (1-2) The hypergraph structures learned from each view are fused to obtain the first hypergraph association matrix.

[0085] Finally, the multi-view fusion module is used to fuse the hypergraph structures learned on each view, which is defined as follows:

[0086]

[0087] Among them, V is the number of views used, v is the vth view, H (v) That is, the hypergraph association matrix learned on the v-th view.

[0088] The multi-view hypergraph learning network learns the hypergraph structure on multiple views and adopts a different learnable similarity measurement function in each view to achieve a comprehensive and accurate measurement of the similarity of scene image samples.

[0089] (1-3) The original hypergraph association matrix obtained from the auxiliary road network and the first hypergraph association matrix are merged to obtain a hypergraph association matrix.

[0090] In order to more accurately model the hypergraph of scene image data, the auxiliary road network is introduced, and the kNN algorithm is used to calculate the similarity between scene image samples in the original feature space. Based on the similarity matrix, hyperedges are constructed to obtain the original hypergraph association matrix H0.

[0091] In one embodiment, the k value in the kNN algorithm is set to 15.

[0092] The first hypergraph association matrix learned by the multi-view hypergraph learning network and the original hypergraph association matrix are then weightedly merged to obtain the final hypergraph association matrix, which is expressed by the following formula:

[0093]

[0094] Where η is the weight parameter for the weighted combination of the two correlation matrices.

[0095] In one embodiment, the weight parameter η=0.3.

[0096] (1-4) Loss function.

[0097] In order to improve the generalization of the multi-view hypergraph learning network, a consistency loss function is introduced to constrain the consistency of the hypergraph structure learned on each view. It is defined as follows:

[0098]

[0099] Among them, ‖·‖2 is the L2 norm.

[0100] The loss function of the multi-view hypergraph learning network is defined as follows:

[0101]

[0102] in, is the hypergraph Laplace, D v and D e are vertex degree matrix and hyperedge degree matrix respectively, where vertex degree is calculated as The calculation of the hyperedge degree is α, β, γ and μ are the weight hyperparameters of each term in the loss function; N is the number of samples; tr(·) is the trace of the matrix; · T represents the transpose operation; ‖·‖ F is the Frobenius norm; log(·) represents the logarithmic operation.

[0103] In one embodiment, α=0.8, β=0.3, γ=0.5, μ=0.4, and N=4485.

[0104] (2) Density-aware hypergraph attention network.

[0105] The density-aware hypergraph attention network is mainly used to learn the embedded representation of scene images and perform category prediction based on the learned embedded representation. Its input is the scene image feature matrix and the hypergraph association matrix obtained by the multi-view hypergraph learning network, and its output is the category prediction result of the scene image.

[0106] like Figure 4 As shown in Figure 1, the density-aware hypergraph attention network mainly consists of two density-aware hypergraph attention layers. A density-aware attention mechanism is designed to mine and utilize density information in the data to improve the performance of scene image classification. The input of the density-aware hypergraph attention layer is the node feature matrix and the hyperedge feature matrix. An auxiliary path network is introduced before the density-aware hypergraph attention network, and the hypergraph information propagation mechanism is used to obtain the node feature matrix and the hyperedge feature matrix. Its definition is as follows:

[0107]

[0108]

[0109] Among them, X and E are the obtained node feature matrix and hyperedge feature matrix respectively.

[0110] Each density-aware hypergraph attention layer mainly consists of a density-aware vertex aggregation module and a density-aware hyperedge aggregation module. The density-aware vertex aggregation module aggregates the information of connected vertices into hyperedges. Similarly, the density-aware hyperedge aggregation module is used to aggregate hyperedge information to update node representations.

[0111] (2-1) Density-aware attention vertex aggregation module.

[0112] (2-1-1)Vertex density rule.

[0113] In the density-aware attention vertex aggregation module, a density rule is designed for vertices in the hypergraph. The density of a vertex is defined as the sum of the similarities of neighboring nodes whose similarity to the target node is greater than a preset threshold δ2, which is defined as follows:

[0114]

[0115] in, Represents node x i density; Represents node x i Neighbor node x k Set; W is the learnable weight matrix.

[0116] In one embodiment, δ2 is set to 0.4.

[0117] (2-1-2) Density-aware attention mechanism.

[0118] Based on this density rule, a density-aware attention mechanism is designed to incorporate density information into attention. Compared to traditional attention mechanisms that only consider node similarity, the density-aware attention mechanism also considers node distribution information, thereby achieving more accurate attention learning.

[0119] In the density-aware attention mechanism:

[0120] First, the traditional attention mechanism Attention(·) is used to calculate the vertex x i and hyperedge e k The attention value between can be expressed by the following formula:

[0121]

[0122] Then, the density information and attention are fused to construct the density-aware attention mechanism, which is described by the following formula:

[0123]

[0124] in, is the normalized vertex density, a X is the attention value A collection of .

[0125] The traditional attention mechanism Attention(·) adopts a design similar to that in the graph attention network. Therefore, the complete definition of the designed density-aware attention mechanism is as follows:

[0126]

[0127] in, Represents the hyperedge e k The set of connected vertices; exp(·) is the exponential function; LeakyReLU(·) is the activation function; α X is the weight matrix to be trained; || represents a concatenation operation.

[0128] Based on the density-aware attention mechanism, the density-aware attention matrix DA is obtained. X , whose elements

[0129] (2-1-3) Vertex feature aggregation.

[0130] Finally, vertex feature aggregation is performed based on the density-aware attention matrix, which is defined as follows:

[0131]

[0132] Among them, ELU(·) is the activation function.

[0133] (2-2) Density-aware attention hyperedge aggregation module.

[0134] (2-2-1) Density rule of hyperedges.

[0135] In the density-aware attention hyperedge aggregation module, a density rule is designed for hyperedges in the hypergraph. The density of a hyperedge is defined as the sum of the densities of the vertices connected by the hyperedge, which is defined as follows:

[0136]

[0137] in, That is the hyperedge e k density.

[0138] (2-2-2) Density-aware attention mechanism.

[0139] Similar to the density-aware attention vertex aggregation module, the density-aware attention mechanism in the density-aware attention hyperedge aggregation module is defined as follows:

[0140]

[0141] in, Indicates connection to vertex x i The hyperedge set of α E is the weight matrix to be trained; is the normalized hyperedge density, a E is the set of hyperedge attention values.

[0142] Based on the density-aware attention mechanism, the density-aware attention matrix DA is obtained. E , whose elements

[0143] (2-2-3) Hyperedge feature aggregation.

[0144] Finally, hyperedge feature aggregation is performed based on the density-aware attention matrix, which is defined as follows:

[0145]

[0146] The above two modules are combined to construct a density-aware hypergraph attention layer, which can be expressed by the following formula:

[0147]

[0148] In the density-aware hypergraph attention layer, vertex features are first aggregated to hyperedges to generate hyperedge features, and then vertex features are obtained by aggregating connected hyperedge features. By using this vertex-hyperedge-vertex transformation mechanism, high-order relationships between data can be effectively represented.

[0149] The first layer of the density-aware hypergraph attention network uses a multi-head attention mechanism to enhance feature aggregation. The feature embedding output by this layer is obtained by connecting the outputs of each head and can be described by the following formula:

[0150]

[0151] in, represents the concatenation operation; T is the number of attention heads used.

[0152] A softmax(·) operation is added after the density-aware hypergraph attention network to obtain the category prediction results of the scene image.

[0153] In one embodiment, T=2; the output dimensions of the weight matrices of the two density-aware hypergraph attention layers are set to 64 and the number of sample categories C, respectively.

[0154] (2-3) Loss function.

[0155] The loss function of the density-aware hypergraph attention network is the cross-entropy loss, which is defined as follows:

[0156]

[0157] Where L is the set of labeled samples; Y is the label of the labeled sample; C is the number of scene image categories; Z is the scene image prediction result; ln(·) is the logarithmic operation.

[0158] By jointly optimizing the multi-view hypergraph learning network and the density-aware hypergraph attention network, we can learn the optimal hypergraph suitable for the downstream graph-based scene image classification task. At the same time, by learning a suitable hypergraph, we can improve the performance of the downstream density-aware hypergraph attention network.

[0159] The dual hypergraph neural network achieves higher scene image classification performance by effectively combining the above two sub-hypergraph neural networks.

[0160] S203: Use the scene image dataset to train a scene image classification model.

[0161] The scene image dataset obtained in step S201 is used to train the scene image classification model designed in step S202, so that the model converges reliably and obtains optimal model parameters.

[0162] By jointly optimizing the multi-view hypergraph learning network and the density-aware hypergraph attention network, the loss function of the model is a linear combination of the hypergraph learning loss of the multi-view hypergraph learning network and the cross entropy loss of the density-aware hypergraph attention network. During training, the overall loss function is optimized. Achieve model convergence. The loss function of the model is The definition is as follows:

[0163]

[0164] Among them, λ is the weight hyperparameter between the two loss functions.

[0165] By optimizing the loss function of the model, the network model converges and the optimal model parameters are obtained.

[0166] In one embodiment, λ is set to 1.1.

[0167] S204 : Input the scene image to be classified into the trained scene image classification model to obtain a category prediction result of the scene image to be classified.

[0168] The intelligent video surveillance scene image to be classified is used as the input of the model, and the optimal model parameters obtained by training in step S203 are used to perform category prediction on the input scene image sample, and the category prediction result of the scene image to be classified is output.

[0169] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above embodiments may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.

[0170] It should be noted that although the method operations of the above embodiments are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the depicted steps may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0171] Example 2:

[0172] like Figure 6 As shown, this embodiment provides a scene image classification system based on a dual hypergraph neural network, which includes a scene image acquisition module 601, a hypergraph learning module 602 and a category prediction module 603, wherein:

[0173] The scene image acquisition module 601 is used to acquire a scene image to be classified; and obtain a feature matrix corresponding to the scene image based on the scene image to be classified; the scene image to be classified is at least one, and all of them are intelligent video surveillance scene images;

[0174] A hypergraph learning module 602 is configured to input the feature matrix into a multi-view hypergraph learning network of a trained dual-hypergraph neural network, and fuse hypergraphs learned by the multi-view hypergraph learning network on multiple views to obtain a hypergraph association matrix; each view in the multi-view hypergraph learning network uses a different learnable similarity metric function;

[0175] The category prediction module 603 is used to input the feature matrix and the hypergraph association matrix into the density-aware hypergraph attention network of the trained dual-hypergraph neural network, and use the density-aware attention mechanism to mine the density information in the data to perform hypergraph representation learning to obtain the category prediction results of the scene image to be classified; the density-aware attention mechanism is used to fuse the density information of vertices / hyperedges in the hypergraph into the vertex / hyperedge density attention.

[0176] The specific implementation of each module in this embodiment can be found in the above-mentioned embodiment 1, and will not be described one by one here; it should be noted that the system provided in this embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0177] Example 3:

[0178] This embodiment provides an electronic device, which may be a computer, such as Figure 7 As shown, a processor 702, a memory, an input device 703, a display 704, and a network interface 705 are connected via a system bus 701. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium 706 and an internal memory 707. The non-volatile storage medium 706 stores an operating system, a computer program, and a database. The internal memory 707 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 702 executes the computer program stored in the memory, the scene image classification method of the above-mentioned embodiment 1 is implemented as follows:

[0179] Obtain a scene image to be classified; and obtain a feature matrix corresponding to the scene image based on the scene image to be classified; the scene image to be classified is at least one, and all of them are intelligent video surveillance scene images;

[0180] Inputting the feature matrix into a multi-view hypergraph learning network of a trained dual-hypergraph neural network, and fusing the hypergraphs learned by the multi-view hypergraph learning network on multiple views to obtain a hypergraph association matrix; each view in the multi-view hypergraph learning network uses a different learnable similarity metric function;

[0181] The feature matrix and the hypergraph association matrix are input into the density-aware hypergraph attention network of the trained dual-hypergraph neural network, and the density-aware attention mechanism is used to mine the density information in the data for hypergraph representation learning to obtain the category prediction results of the scene image to be classified; the density-aware attention mechanism is used to fuse the density information of vertices / hyperedges in the hypergraph into the vertex / hyperedge density attention.

[0182] Example 4:

[0183] This embodiment provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the scene image classification method of the above embodiment 1 is implemented as follows:

[0184] Obtain a scene image to be classified; and obtain a feature matrix corresponding to the scene image based on the scene image to be classified; the scene image to be classified is at least one, and all of them are intelligent video surveillance scene images;

[0185] Inputting the feature matrix into a multi-view hypergraph learning network of a trained dual-hypergraph neural network, and fusing the hypergraphs learned by the multi-view hypergraph learning network on multiple views to obtain a hypergraph association matrix; each view in the multi-view hypergraph learning network uses a different learnable similarity metric function;

[0186] The feature matrix and the hypergraph association matrix are input into the density-aware hypergraph attention network of the trained dual-hypergraph neural network, and the density-aware attention mechanism is used to mine the density information in the data for hypergraph representation learning to obtain the category prediction results of the scene image to be classified; the density-aware attention mechanism is used to fuse the density information of vertices / hyperedges in the hypergraph into the vertex / hyperedge density attention.

[0187] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0188] The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solution and inventive concept of the present invention within the scope disclosed by the present invention, which falls within the scope of protection of the present invention.

Claims

1. A scene image classification method based on a dual hypergraph neural network, characterized in that: The method comprises: Obtain a scene image to be classified; and obtain a feature matrix corresponding to the scene image based on the scene image to be classified; the scene image to be classified is at least one, and all of them are intelligent video surveillance scene images; Inputting the feature matrix into a multi-view hypergraph learning network of a trained dual-hypergraph neural network, and fusing the hypergraphs learned by the multi-view hypergraph learning network on multiple views to obtain a hypergraph association matrix; each view in the multi-view hypergraph learning network uses a different learnable similarity metric function; The feature matrix and the hypergraph association matrix are input into the density-aware hypergraph attention network of the trained dual-hypergraph neural network, and the density-aware attention mechanism is used to mine the density information in the data to perform hypergraph representation learning, thereby obtaining a category prediction result of the scene image to be classified; the density-aware attention mechanism is used to fuse the density information of the vertices in the hypergraph into the vertex density attention, and the density-aware attention mechanism is used to fuse the density information of the hyperedges in the hypergraph into the hyperedge density attention; The step of inputting the feature matrix into the multi-view hypergraph learning network of the trained dual-hypergraph neural network and fusing the hypergraphs learned by the multi-view hypergraph learning network on multiple views to obtain a hypergraph association matrix includes: Using a multi-view fusion module, the hypergraphs learned on each view in the multi-view hypergraph learning network are fused to obtain a first hypergraph association matrix; each node in the hypergraph represents a scene image, and the hyperedges in the hypergraph represent high-order association relationships between scene images; Calculating the similarity between scene images based on the feature matrix; constructing hyperedges based on the similarity to obtain an original hypergraph association matrix; Merging the first hypergraph incidence matrix and the original hypergraph incidence matrix to obtain a hypergraph incidence matrix; The method further comprises: According to the feature matrix and the hypergraph association matrix, a vertex feature matrix and a hyperedge feature matrix are obtained using a hypergraph information propagation mechanism; The feature matrix and the hypergraph association matrix are input into the density-aware hypergraph attention network of the trained dual-hypergraph neural network, and the density-aware attention mechanism is used to mine the density information in the data for hypergraph representation learning to obtain the category prediction result of the scene image to be classified, including: Inputting the vertex feature matrix and the hyperedge feature matrix into the first density-aware hypergraph attention layer in the density-aware hypergraph attention network, and obtaining the feature embedding of the layer using the density-aware attention mechanism; The features obtained by the first density-aware hypergraph attention layer are embedded in the second density-aware hypergraph attention layer in the input density-aware hypergraph attention network, and the category result is predicted based on the output of the second density-aware hypergraph attention layer.

2. The scene image classification method according to claim 1, characterized in that: The hypergraph learned on each view in the multi-view hypergraph learning network is obtained through the following process: Mapping the feature matrix from the original feature space to a low-dimensional embedding space using a low-dimensional embedding module; Using the similarity learning module, the similarity between scene images is learned through the similarity measurement function in the low-dimensional embedding space; The hyperedge construction module is used to construct hyperedges based on the learned similarity to obtain the hypergraph structure learned on the view.

3. The scene image classification method according to claim 2, characterized in that: The similarity between scene images is learned by using a similarity measurement function in a low-dimensional embedding space, including: The similarity between scene images is calculated through the similarity measurement function to obtain the similarity matrix; Sparse sampling is performed on the similarity matrix to filter out similarities below a first preset threshold.

4. The scene image classification method according to claim 1, characterized in that Inputting the vertex feature matrix and the hyperedge feature matrix into a first density-aware hypergraph attention layer in a density-aware hypergraph attention network, and obtaining feature embedding of the layer using a density-aware attention mechanism, comprises: Calculating the density of vertices and hyperedges based on the vertex feature matrix and the hyperedge feature matrix according to the hypergraph density rule; The first density-aware attention mechanism designed based on vertex density fuses the density of vertices and the attention between vertices and hyperedges to obtain a vertex density-aware attention matrix; vertex features are aggregated according to the vertex density-aware attention matrix to obtain an updated hyperedge feature matrix; wherein the first density-aware attention mechanism considers the distribution information of vertices; A second density-aware attention mechanism designed based on hyperedge density fuses the density of hyperedges and the attention between hyperedges and vertices to obtain a hyperedge density-aware attention matrix; hyperedge features are aggregated based on the hyperedge density-aware attention matrix and the updated hyperedge feature matrix to obtain an updated vertex feature matrix; wherein the second density-aware attention mechanism considers the distribution information of hyperedges; The updated vertex feature matrices obtained by each attention head in the first density-aware hypergraph attention layer are concatenated to obtain the feature embedding of this layer.

5. The scene image classification method according to claim 4, characterized in that: The hypergraph density rule includes a vertex density rule and a hyperedge density rule, wherein: Vertex density rule: The density of a vertex is the sum of the similarities of neighboring nodes whose similarity to the target node is greater than a second preset threshold; Hyperedge density rule: The density of a hyperedge is the sum of the densities of the vertices connected by the hyperedge.

6. A scene image classification system based on a dual hypergraph neural network, characterized in that: The system comprises: A scene image acquisition module is used to acquire a scene image to be classified; based on the scene image to be classified, a feature matrix corresponding to the scene image is obtained; the scene image to be classified is at least one, and all of them are intelligent video surveillance scene images; A hypergraph learning module is configured to input the feature matrix into a multi-view hypergraph learning network of a trained dual-hypergraph neural network, and to obtain a hypergraph association matrix by fusing hypergraphs learned by the multi-view hypergraph learning network on multiple views; each view in the multi-view hypergraph learning network uses a different learnable similarity metric function; A category prediction module is configured to input the feature matrix and the hypergraph association matrix into the density-aware hypergraph attention network of the trained dual-hypergraph neural network, and utilize the density-aware attention mechanism to mine the density information in the data for hypergraph representation learning to obtain a category prediction result for the scene image to be classified; the density-aware attention mechanism is configured to fuse the density information of vertices in the hypergraph into the vertex density attention, and the density-aware attention mechanism is configured to fuse the density information of hyperedges in the hypergraph into the hyperedge density attention; The step of inputting the feature matrix into the multi-view hypergraph learning network of the trained dual-hypergraph neural network and fusing the hypergraphs learned by the multi-view hypergraph learning network on multiple views to obtain a hypergraph association matrix includes: Using a multi-view fusion module, the hypergraphs learned on each view in the multi-view hypergraph learning network are fused to obtain a first hypergraph association matrix; each node in the hypergraph represents a scene image, and the hyperedges in the hypergraph represent high-order association relationships between scene images; Calculating the similarity between scene images based on the feature matrix; constructing hyperedges based on the similarity to obtain an original hypergraph association matrix; Merging the first hypergraph incidence matrix and the original hypergraph incidence matrix to obtain a hypergraph incidence matrix; The system further comprises: According to the feature matrix and the hypergraph association matrix, a vertex feature matrix and a hyperedge feature matrix are obtained using a hypergraph information propagation mechanism; The feature matrix and the hypergraph association matrix are input into the density-aware hypergraph attention network of the trained dual-hypergraph neural network, and the density-aware attention mechanism is used to mine the density information in the data for hypergraph representation learning to obtain the category prediction result of the scene image to be classified, including: Inputting the vertex feature matrix and the hyperedge feature matrix into the first density-aware hypergraph attention layer in the density-aware hypergraph attention network, and obtaining the feature embedding of the layer using the density-aware attention mechanism; The features obtained by the first density-aware hypergraph attention layer are embedded in the second density-aware hypergraph attention layer in the input density-aware hypergraph attention network, and the category result is predicted based on the output of the second density-aware hypergraph attention layer.

7. A storage medium storing a program, characterized in that: When the program is executed by a processor, the scene image classification method according to any one of claims 1 to 5 is implemented.

Images