Image scene recognition method and device, equipment and storage medium

By designing branch network model and loss function training, the problem of difficult to balance accuracy and speed in image scene recognition methods is solved, efficient recognition effect is achieved, and application scenarios are broadened.

CN119027790APending Publication Date: 2024-11-26CHANGCHUN UNIV OF SCI & TECH

Patent Information

Application Number
CN202411025598.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing image scene recognition methods are difficult to balance accuracy and speed, with low recognition efficiency and low accuracy, which limits their application scenarios.

Method used

A method of image scene recognition is designed, using a network model divided into three branches, including pre-trained image semantic segmentation network, graph neural network and convolutional neural network, and generating through graph generation module, graph embedding module and significant area The module extracts scene features and uses cross-entropy loss function for training to improve recognition speed and accuracy.

Benefits of technology

It realizes the recognition accuracy while improving the recognition speed, widens the application scenarios, and combines the graph neural network with node classification and traditional convolutional neural network, reduces the computational complexity, improves the accuracy of multiple scene recognition and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027790A_ABST
    Figure CN119027790A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of image recognition, and particularly relates to an image scene recognition method, which comprises the following steps of: 1, preparing a data set, selecting a scene recognition data set, dividing the data set into a training set, a test set and a verification set, and preprocessing an image; according to the method, scene recognition is carried out by designing the graph neural network of node classification, target information in the image is fully utilized to improve recognition precision, recognition is carried out by utilizing the graph neural network, a large number of images are converted into graph structures, the step of feature extraction of a traditional convolutional neural network is simplified, and meanwhile, the recognition efficiency is improved. Compared with a traditional convolutional neural network, the graph neural network model has fewer parameters, and through a local connection mode, the calculation complexity can be reduced, and the recognition speed of scene recognition is improved while the recognition precision is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to an image scene recognition method, device, equipment and storage medium. Background Art

[0002] Image scene recognition is a technology that uses computer vision to enable computers to understand scenes and objects in images, thereby realizing intelligent analysis and processing of image content. Image scene recognition can achieve efficient recognition of complex scenes and is widely used in fields such as intelligent monitoring, autonomous driving, and smart homes, providing more accurate and reliable data support for these fields and improving people's living and working environment. However, current image scene recognition methods have difficulty in achieving a balance between accuracy and speed.

[0003] The Chinese patent publication number is "CN115100496A", and its name is "An image recognition method for adaptive scenes". This method establishes a feature knowledge graph based on image features and the correlation between image features, and establishes several feature recognition models based on the correlation between image features. The corresponding feature recognition model is used to analyze and recognize the image features in the sub-image to obtain the corresponding feature information, which reduces the training amount of the feature recognition model and can improve the accuracy of the feature information, but there is a problem of low recognition efficiency. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides an image scene recognition method, device, equipment and storage medium. By solving the problems of low accuracy and poor real-time performance of the existing image scene recognition methods, the present invention improves the speed of the recognition algorithm while ensuring the recognition accuracy, thereby broadening the application scenarios.

[0006] (II) Technical solution

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] An image scene recognition method, apparatus, device and storage medium, including four aspects;

[0009] In a first aspect, the present invention provides an image scene recognition method, the method comprising the following steps:

[0010] Step 1: Prepare the data set, select the scene recognition data set, divide the data set into training set, test set, and validation set, and preprocess the images;

[0011] Step 2: Build a network model. The model is divided into three branches, and finally converges the scene features to the classification layer. One branch is used to extract the attributes of each target in the scene image, including a pre-trained image semantic segmentation network model and a graph neural network for node classification. The graph neural network mainly includes a graph generation module and a graph embedding module. One branch is used to extract the global scene attributes of the scene image, including a convolutional neural network. One branch is used to extract the local scene attributes of the scene image, including a salient area generation module.

[0012] Step 3: Train the network model and select a suitable loss function to train the network to minimize the value between the image recognition result and the true label. When the training reaches the set number of times or the value of the loss function reaches the preset range, the model parameters are considered to have been trained and saved.

[0013] Step 4: Fine-tune the network model and use the validation set to fine-tune the neural network model to further improve the model performance.

[0014] Step 5: Determine the model, solidify the network model parameters, and determine the final network model. If the image scene recognition task is performed later, the image to be recognized can be directly input into the end-to-end network model to obtain the recognition result.

[0015] Furthermore, in step 1, scene recognition uses the MIT-67Indoor dataset and the Place365 dataset to perform preprocessing such as image enhancement.

[0016] Furthermore, in step 2, the pre-trained image semantic segmentation network model can use mainstream semantic segmentation models such as U-Net model and DeepLab model to identify and segment targets in scene images for use by subsequent scene recognition network.

[0017] Furthermore, in the step 2, the graph generation module is used to generate a node feature matrix X in the graph structure and an adjacency matrix A for constructing the adjacency relationship of the edges in the graph structure. The graph embedding module maps the node and edge feature information of the graph structure into corresponding vectors. The convolutional neural network includes convolution block 1, convolution block 2, convolution block 3, convolution block 4, and an attention module. The attention module performs weighted calculation on the feature vectors of the two branches and then fuses them. The significant area generation module includes convolution block 5, convolution block 6, and convolution block 7. After feature extraction and global average pooling of the semantic map, the significant area of ​​the scene image is obtained. The classification layer includes a fully connected layer and a Softmax layer, which is used to classify the obtained feature information to obtain a scene score.

[0018] Furthermore, the step 3 selects a suitable loss function, specifically a cross entropy loss; the cross entropy loss is used to measure the difference between the output of the network and the true label, minimize the cross entropy loss, make the network's prediction as close to the true label as possible, and improve the accuracy of classification.

[0019] In a second aspect, the present invention provides an image scene recognition device, including an acquisition module, a preprocessing module, and a recognition module, characterized in that:

[0020] The acquisition module is used to acquire images of different scenes;

[0021] The preprocessing module is used to perform denoising and enhancement operations on the collected scene images;

[0022] The recognition module is used to recognize the scene in which the image is located and obtain the category of the image scene; its internal composition has been described in detail in the above-mentioned image scene recognition method and will not be repeated here.

[0023] In a third aspect, a computer device comprises an input / output unit, a central processing unit, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the image scene recognition method are implemented.

[0024] In a fourth aspect, the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image scene recognition method as described in any one of the first aspects.

[0025] (III) Beneficial effects

[0026] Compared with the prior art, the present invention provides an image scene recognition method, which has the following beneficial effects:

[0027] The present invention performs scene recognition by designing a graph neural network with node classification, which makes full use of the target information in the image to improve the recognition accuracy. The graph neural network is used for recognition to convert a large number of pictures into graph structures, which simplifies the feature extraction steps of traditional convolutional neural networks. At the same time, the graph neural network model has fewer parameters than the traditional convolutional neural network, and can reduce the complexity of calculation through local connection, thereby ensuring the recognition accuracy and improving the recognition speed of scene recognition.

[0028] The present invention, by designing a graph generation module, enables semantic images to be efficiently converted into graph structures, constructs nodes and adjacency matrices in the graph structure, uses the features of each image as nodes of the graph structure, and connects the nodes based on the similarities and differences between classes and within classes, using the similarities and differences of the images as edges of the graph structure, thereby making full use of the target attributes in the image scene and effectively improving the accuracy of recognition of multiple scenes.

[0029] The present invention designs a significant region generation module to generate a significant region map of a scene semantic image, guides feature block clipping based on the significant region, obtains local significant features of the scene, and fully extracts local details of the scene image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of an image scene recognition method of the present invention;

[0031] Figure 2 This is a schematic diagram of the structure of an image scene recognition device of the present invention;

[0032] Figure 3 A network structure diagram of an image scene recognition method of the present invention;

[0033] Figure 4 It is a structural diagram of convolution blocks 1 to 4 of an image scene recognition method of the present invention;

[0034] Figure 5 It is a structural diagram of convolution blocks 5 to 7 of an image scene recognition method of the present invention;

[0035] Figure 6 This is a structural diagram of an attention module of an image scene recognition method of the present invention;

[0036] Figure 7 A comparison table of related indicators of an image scene recognition method of the present invention and the prior art;

[0037] Figure 8 The figure is a schematic diagram of the structure of an image scene recognition device according to the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example

[0040] like Figure 1As shown, an image scene recognition method proposed in one embodiment of the present invention includes the following steps:

[0041] Step 1: Prepare the dataset. Prepare the dataset for training the image scene recognition network. Use the MIT-67Indoor dataset, which contains 67 indoor categories and a total of 15,620 images. Use the Place365-Standard dataset, which contains 365 scene categories and a total of 1.8 million images.

[0042] Step 2: Build a network model, such as Figure 3 As shown in Figure 1, the model is divided into three branches and finally converges to the classification layer. One branch is used to extract the attributes of each target in the scene image, including a pre-trained image semantic segmentation network model and a graph neural network for node classification. The graph neural network mainly includes a graph generation module and a graph embedding module. One branch is used to extract the global scene attributes of the scene image, including a convolutional neural network. One branch is used to extract the local scene attributes of the scene image, including a salient area generation module. The pre-trained semantic segmentation model is a DeepLab semantic segmentation model, which is used to extract the target attributes in the image scene, identify and segment the scenery and characters, and calculate the proportion of the identified target in the image, and then input it into the graph neural network. The graph neural network model includes a graph generation module and a graph embedding module. The graph generation module is used to generate the node feature matrix X in the graph structure and construct the adjacency matrix A of the edge adjacency relationship in the graph structure, and convert the proportion of the target in the image obtained by the semantic segmentation model into a node feature vector. Each target is a dimension, with a total of 37 dimensions, and the size of each dimension is determined by the proportion of the target in the image. If a target does not exist, the dimension is 0. At the same time, the module constructs the adjacency matrix A based on the inter-class similarity and intra-class difference. Specifically, for scene images of the same category, the module determines the difference between the images and decides whether to connect an edge between the corresponding nodes based on the relationship between the difference value and the preset threshold. The difference is defined as:

[0043]

[0044] Among them, I a and I b Represents two pictures of the same category, δ is the indicator function, θ d is the difference threshold, Represents image I a The eigenvalue in the i-th dimension.

[0045] For scene images of different categories, the module determines whether to connect an edge between corresponding nodes based on the similarity between the images, where the similarity is defined as:

[0046]

[0047] Among them, I a and I c Represents two pictures of different categories, δ is the indicator function, θ s is the similarity threshold.

[0048] The condition for connecting edges depends on the difference between corresponding dimensions and the comparison of similarity values ​​with a preset threshold θ.

[0049] The graph embedding module is composed of multiple graph convolution layers. The input is the adjacency matrix A of the graph data and the node feature matrix X. The graph convolution is used for processing to map the node and edge feature information of the graph into corresponding vectors to obtain the target features. The scene feature extraction branch is used to extract the global features of the scene image. It is composed of a convolutional neural network, including convolution block 1, convolution block 2, convolution block 3, and convolution block 4. Figure 4 As shown, the convolution kernel size of convolution blocks 1 to 4 is 3×3, the step size and padding are set to 1, and all activation functions use R-type functions, which are defined as follows:

[0050]

[0051] The attention module is composed of convolutional blocks, such as Figure 6 As shown in the figure, it includes a 1×1 convolution layer, a global average pooling layer, a fully connected layer, and an activation layer. The activation layer uses an R-type activation function to fuse the obtained features and target features into the global features after weighted calculation by the attention module.

[0052] The salient region generation module includes convolutional blocks 5, 6, and 7. Figure 5 As shown in the figure, the convolution kernel size is 3×3, the step size and padding are both set to 1, and all activation functions use R-type functions. The semantic map is subjected to feature extraction and global average pooling to obtain the salient area of ​​the scene image. The obtained features are weighted multiplied with the features of the scene image after feature extraction to obtain the local features of the scene image.

[0053] The classification layer consists of a fully connected layer and a Softmax layer. The obtained feature information is concatenated and classified to obtain the scene score.

[0054] Step 3: Train the network model and select the cross entropy loss function to train the network. This function is used to measure the difference between the network output and the true label, minimize the cross entropy loss, make the network prediction as close to the true label as possible, and improve the classification accuracy. The specific formula is as follows:

[0055] L=-zlogσ(y)-(1-z)log(1-σ(y)) (4)

[0056] where z represents the scene label of the input image, y refers to the output of the scene recognition network, and σ refers to the Softmax function that normalizes the scene probability to [0, 1].

[0057] The network is trained, the number of training times is set to 200, and the learning rate of the training process is set to 0.001, which can make the loss function converge quickly without causing the network to overfit. The Adam optimizer is used, and the loss function threshold is set to about 0.005. The model is continuously iterated and optimized until the number of training times reaches the set threshold or the value of the loss function reaches the set threshold range. The model parameters are considered to have been trained and the model parameters are saved.

[0058] Step 4: Fine-tune the network and use the validation set to adjust the network to further improve the model performance;

[0059] Step 5: Determine the model, solidify the network model parameters, and determine the final network model. If the image scene recognition task is performed later, the image to be recognized can be directly input into the end-to-end network model to obtain the recognition result.

[0060] Among them, convolution, activation function, splicing operation, and DeepLab model are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0061] The present invention uses a scene recognition method to achieve accurate and rapid recognition of scene images. By calculating the relevant indexes of the image obtained by the existing method, the feasibility and superiority of the method are further verified. The relevant indexes of the existing technology and the method proposed by the present invention are compared. Figure 7 As shown in the table, it can be seen that the method proposed in the present invention has better recognition results.

[0062] like Figure 2 As shown, a scene recognition device proposed in one embodiment of the present invention can execute the above method, including an acquisition module, a preprocessing module, and a recognition module.

[0063] The acquisition module is used to acquire the scene image to be recognized.

[0064] The preprocessing module is used to perform denoising and enhancement operations on the collected scene images.

[0065] The recognition module is used to process the preprocessed image. Its structure is divided into three branches and finally converges to the classification layer. The specific processing process is that the semantic segmentation model of one branch segments the target object in the input scene image, calculates the proportion of the recognized target in the image, and then inputs it into the graph generation module to generate the node feature matrix X and the edge adjacency matrix A, and then inputs the generated graph data into the graph embedding module to convert it into a corresponding vector. One branch is used to extract the global scene attributes of the scene image, and one branch is used to extract the local scene attributes of the scene image. Then the obtained feature fusion features are input into the classification layer to obtain the scene score.

[0066] like Figure 8 As shown, a computer device proposed in one embodiment of the present invention represents a schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present application. The computer device includes a central processing unit, a system memory, a large-capacity storage medium, an input part, an output part, a communication interface, and a display unit. Among them, the central processing unit, the system memory, and the large-capacity storage medium are connected through a system bus, and the input part, the output part, the communication interface, and the display unit are connected to the system bus through an input / output interface. Among them, the central processing unit is used to provide computing and control capabilities, the system memory includes a read-only memory and a random access memory, and provides an operating environment for an operating system and a computer program, and the large-capacity storage medium stores an operating system and a computer program. The input / output interface is connected through a system bus, wherein the input part includes a keyboard, a mouse, etc., the output part includes a liquid crystal display, a speaker, a projection device, etc., and the communication interface is used to communicate with an external terminal in a wired or wireless manner.

[0067] The present invention provides a computer storage medium, which is a device capable of storing data and instructions, wherein the medium stores instruction sets, code sets and programs that can be read and executed by a computer processor to implement the various steps of the method described in the present invention; optionally, the computer storage medium includes: read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk and other media capable of storing computer program codes.

[0068] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image scene recognition method, characterized in that: The method comprises the following steps: Step 1: Prepare the data set, select the scene recognition data set, divide the data set into training set, test set, and validation set, and preprocess the images; Step 2: Build a network model. The model is divided into three branches, and finally converges the scene features to the classification layer. One branch is used to extract the attributes of each target in the scene image, including a pre-trained image semantic segmentation network model and a graph neural network for node classification. The graph neural network mainly includes a graph generation module and a graph embedding module. One branch is used to extract the global scene attributes of the scene image, including a convolutional neural network. One branch is used to extract the local scene attributes of the scene image, including a salient area generation module. Step 3: Train the network model and select a suitable loss function to train the network to minimize the value between the image recognition result and the true label. When the training reaches the set number of times or the value of the loss function reaches the preset range, the model parameters are considered to have been trained and saved. Step 4: Fine-tune the network model and use the validation set to fine-tune the neural network model to improve model performance. Step 5: Determine the model, solidify the network model parameters, and determine the final network model. If the image scene recognition task is performed later, the image to be recognized can be directly input into the end-to-end network model to obtain the recognition result.

2. The image scene recognition method according to claim 1, characterized in that: In the step 1, scene recognition uses the MIT-67Indoor dataset and the Place365 dataset to perform preprocessing such as image enhancement.

3. The image scene recognition method according to claim 1, characterized in that: The step 2 is a pre-trained image semantic segmentation network model, which uses the mainstream semantic segmentation models of U-Net model and DeepLab model to identify and segment the targets in the scene image for use by the subsequent scene recognition network. The graph generation module is used to generate the node feature matrix X in the graph structure and the adjacency matrix A for constructing the adjacency relationship of the edges in the graph structure. The graph embedding module maps the node and edge feature information of the graph structure to corresponding vectors. The convolutional neural network includes convolution block 1, convolution block 2, convolution block 3, convolution block 4, and an attention module. The attention module performs weighted calculation on the feature vectors of the two branches and then fuses them. The significant area generation module includes convolution block 5, convolution block 6, and convolution block 7. After feature extraction and global average pooling of the semantic map, the significant area of ​​the scene image is obtained. The classification layer includes a fully connected layer and a Softmax layer, which is used to classify the obtained feature information to obtain a scene score.

4. The image scene recognition method according to claim 3, characterized in that: The convolution block one, convolution block two, convolution block three, convolution block four, convolution block five, convolution block six, and convolution block seven include a convolution layer, a normalization layer, and an activation layer. The convolution kernel size is 3×3, the step size and padding are both 1, and the activation layer is an R-type activation function.

5. The image scene recognition method according to claim 1, characterized in that: The step 3 selects a suitable loss function, specifically a cross entropy loss; the cross entropy loss is used to measure the difference between the network output and the true label, minimize the cross entropy loss, make the network prediction as close to the true label as possible, and improve the accuracy of classification.

6. An image scene recognition device, characterized in that: Including acquisition module, preprocessing module, recognition module, The acquisition module is used to acquire images of different scenes; The preprocessing module is used to perform an enhancement operation on the collected scene image; The recognition module is used to recognize the scene in which the image is located and obtain the category of the image scene.

7. A computer device, characterized in that: The computer device includes an input / output unit, a central processing unit, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, an image scene recognition method described in any one of 1-5 is implemented.

8. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, an image scene recognition method described in any one of 1-5 is implemented.

Citation Information

Patent Citations

  • Remote sensing image scene classification method coupling convolutional neural network and graph convolutional network

    CN111461258A

  • Vehicle re-identification method based on double sub-networks

    CN114067143A

  • Scene recognition method based on deep learning

    CN114972965A

  • Multi-scene dual-band image fusion method and device and storable medium

    CN118037561A

Cited By

  • Monitoring and early warning method for intelligent substation and control system thereof

    CN119696186A

  • A monitoring and warning method and its control system for intelligent substations

    CN119696186B

  • Intelligent construction site energy-saving control method and system based on Internet of Things

    CN119849880A