Scene recognition method and device based on self-attention convolutional neural network

Abstract

The invention provides a scene recognition method based on a self-attention convolutional neural network. The method comprises the steps of obtaining training image data and label data corresponding to the training image data; wherein the label data comprises a similar area and a key area, the similar area is an area of which the difference is within a preset range, and the key area is an area of which a fixed building appears within the preset range from the similar area; training the self-attention convolutional neural network through the training image data and the label data to obtain a trained self-attention convolutional neural network; and obtaining a to-be-recognized image, and processing the to-be-recognized image through the trained self-attention convolutional neural network to obtain a target similar region and a target key region. The positions of different points in the similar scene are determined by identifying the key area of the target, and the specific landmark building of the scene is determined, so that the interference of the surrounding environment is reduced when the scene is identified by the neural network.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a scene recognition method and device based on a self-attention convolutional neural network. Background Art

[0002] Generally speaking, the recognition task of specific scenes and landmark buildings is regarded as an instance retrieval task. Now, convolutional neural networks have been widely used in various recognition and detection tasks. Convolutional neural networks are also often used in instance retrieval tasks to extract image features. For example, NetVLAD replaces SIFT (Scale-invariant feature transform) in VLAD with convolution to extract local features, or uses an end-to-end method to train classification networks. However, existing convolutional neural networks still have some problems in specific scene and landmark building recognition tasks. For example, some neural networks have poor adaptability to day and night changes and changes in the surrounding environment, and neural networks trained based on classification methods cannot respond to scenes and locations outside of classification.

[0003] At the same time, the development of edge computing devices in recent years has considerable advantages over previous cloud deployment in terms of cost and flexibility in many scenarios. However, for edge computing devices, the limitations of their computing power and energy consumption are an issue that cannot be ignored. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a scene recognition method and device based on a self-attention convolutional neural network to overcome the above problems or at least partially solve the above problems, including:

[0005] A scene recognition method based on a self-attention convolutional neural network comprises the following steps:

[0006] Acquire training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas;

[0007] Training the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network;

[0008] An image to be identified is obtained, and the image to be identified is processed by the trained self-attention convolutional neural network to obtain a target similar area and a target key area.

[0009] Further, the step of acquiring training image data and label data corresponding to the training image data, wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas, comprises:

[0010] Get initial training images;

[0011] Performing enhancement processing on the initial training image to obtain enhanced image data;

[0012] Performing resistance processing on the initial training image to obtain resistance image data;

[0013] Training image data is generated based on the enhanced image data and the resistant image data.

[0014] Furthermore, the step of training the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network includes:

[0015] Training the self-attention convolutional neural network according to the training image data and the label data to obtain a primary self-attention convolutional neural network;

[0016] The primary self-attention convolutional neural network is re-parameterized to obtain a trained self-attention convolutional neural network.

[0017] Furthermore, the step of training the self-attention convolutional neural network based on the training image data and the label data to obtain a primary self-attention convolutional neural network includes:

[0018] Fix the weights of the self-attention convolutional neural network and train the attention region module to obtain the attention region module;

[0019] The weight of the attention region module is fixed, and the self-attention convolutional neural network is trained to obtain the self-attention region module;

[0020] A primary self-attention convolutional neural network is obtained based on the attention module and the self-attention module.

[0021] Furthermore, the step of re-parameterizing the primary self-attention convolutional neural network to obtain a trained self-attention convolutional neural network includes:

[0022] Convert the convolutional layers and pass-through layers in the primary self-attention convolutional neural network branch into 3×3 convolutional convolutional layers to be fused;

[0023] All the convolutional layers to be fused are fused into a single 3×3 convolutional layer to obtain a trained self-attention convolutional neural network.

[0024] Furthermore, it also includes:

[0025] Performing model distillation on the self-attention convolutional neural network to obtain a distilled neural network;

[0026] Testing the distilled neural network to determine the accuracy difference between the distilled neural network and the self-attention convolutional neural network;

[0027] When the accuracy difference is lower than a preset value, the distilled neural network is deployed to an edge computing device.

[0028] Furthermore, it also includes:

[0029] Determining the landmark building of the image to be identified according to the target similar area and the target key area;

[0030] A landmark recognition result is generated according to the landmark building.

[0031] A scene recognition device based on a self-attention convolutional neural network, comprising:

[0032] An acquisition module, used to acquire training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas;

[0033] A training module, used to train the self-attention convolutional neural network through the training image data and the label data to obtain a trained self-attention convolutional neural network;

[0034] The recognition module is used to obtain the image to be recognized, and process the image to be recognized through the trained self-attention convolutional neural network to obtain the target similar area and the target key area.

[0035] A device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the scene recognition method based on the self-attention convolutional neural network described in any one of the above items is implemented.

[0036] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the scene recognition method based on a self-attention convolutional neural network described in any one of the above items is implemented.

[0037] This application has the following advantages:

[0038] In the embodiments of the present application, compared with the poor adaptability of the neural network scene recognition in the prior art and the shortcomings of being unable to accurately recognize similar scenes, the present application provides a solution for identifying specific scenes by identifying key areas, specifically: a scene recognition method based on a self-attention convolutional neural network, comprising the steps of: obtaining training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas; the self-attention convolutional neural network is trained by the training image data and the label data to obtain a trained self-attention convolutional neural network; the image to be recognized is obtained, and the image to be recognized is processed by the trained self-attention convolutional neural network to obtain a target similar area and a target key area. By identifying the target key area, the positions of different points in the similar scene are determined, and the specific landmark buildings of the scene are determined according to the positions of the different points, so as to improve the recognition accuracy and reduce the interference of the surrounding environment when the neural network recognizes the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 It is a flowchart of a scene recognition method based on a self-attention convolutional neural network provided in one embodiment of the present application;

[0041] Figure 2 It is a schematic diagram of the change of the empty forehead skin of the self-attention convolutional neural network of a scene recognition method based on the self-attention convolutional neural network provided by an embodiment of the present application;

[0042] Figure 3 It is a structural block diagram of a scene recognition device based on a self-attention convolutional neural network provided in one embodiment of the present application;

[0043] Figure 4 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the objects, features and advantages of the present application more obvious and understandable, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0045] By analyzing the prior art, the inventors found that the prior art only identifies the main landmark scenes when identifying landmark buildings or specific scenes, and does not pay too much attention to other elements in the image. As a result, when identifying similar landmark buildings or specific scenes with similar scenes, they are easily interfered with each other, or affected by environmental changes, resulting in recognition errors.

[0046] Reference Figure 1-3 , showing a scene recognition method and device based on a self-attention convolutional neural network provided by an embodiment of the present application;

[0047] The method comprises:

[0048] S110, acquiring training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas;

[0049] S120, training the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network;

[0050] S130, obtaining an image to be identified, and processing the image to be identified by using the trained self-attention convolutional neural network to obtain a target similar area and a target key area.

[0051] In the embodiments of the present application, compared with the poor adaptability of the neural network scene recognition in the prior art and the shortcomings of being unable to accurately recognize similar scenes, the present application provides a solution for identifying specific scenes by identifying key areas, specifically: a scene recognition method based on a self-attention convolutional neural network, comprising the steps of: obtaining training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas; the self-attention convolutional neural network is trained by the training image data and the label data to obtain a trained self-attention convolutional neural network; the image to be recognized is obtained, and the image to be recognized is processed by the trained self-attention convolutional neural network to obtain a target similar area and a target key area. By identifying the target key area, the positions of different points in the similar scene are determined, and the specific landmark buildings of the scene are determined according to the positions of the different points, so as to improve the recognition accuracy and reduce the interference of the surrounding environment when the neural network recognizes the scene.

[0052] Next, a scene recognition method based on a self-attention convolutional neural network in this exemplary embodiment will be further described.

[0053] As described in step S110, training image data and label data corresponding to the training image data are obtained; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas.

[0054] It should be noted that the similar area is a landmark building or a specific scene, and the key area is other representative features near the landmark building or the specific scene. During recognition, there can be multiple key areas to improve the accuracy of recognition.

[0055] As an example, similar areas can be representative landmark buildings such as airport areas, temple areas, and lighthouse areas in the image, but similar landmark buildings are difficult to distinguish. Key areas can be the signs of shops next to the airport, sculptures in front of temples, and buildings in front of lighthouses. For these similar areas, key areas can provide more precise characteristic areas to further identify images with similar areas, so as to identify specific landmark buildings or specific scenes in similar areas, thereby improving recognition accuracy.

[0056] In one embodiment of the present invention, the specific process of "obtaining training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas" in step S110 can be further explained in combination with the following description.

[0057] Obtain initial training images as described in the following steps;

[0058] As described in the following steps, the initial training image is enhanced to obtain enhanced image data;

[0059] As described in the following steps, the initial training image is subjected to resistance processing to obtain resistance image data;

[0060] As described in the following steps, training image data is generated based on the enhanced image data and the resistance image data.

[0061] It should be noted that the initial training images are subjected to data enhancement processing, such as rotation, mirror flipping, multi-image stitching, affine transformation, and adversarial image data enhancement processing, such as blur, noise, texture, and occlusion, to obtain an enhanced data set. Thus, the training image data is obtained to ensure the training performance of the training image data and the recognition ability of the self-attention convolutional neural network after training.

[0062] As described in step S120, the self-attention convolutional neural network is trained using the training image data and the label data to obtain a trained self-attention convolutional neural network.

[0063] It should be noted that the self-attention convolutional neural network is mainly used for feature extraction of images submitted for review. The neural network is mainly composed of convolutional layers and self-attention layers. During training, the main structure is a multi-branch model structure, while in the prediction mode after re-parameterization, the main structure is a single-channel model structure. The structural changes are shown in the following figure. Figure 2 , mainly the changes in the convolution part.

[0064] The output of the self-attention convolutional neural network is a vector of dimension n×2560, where n is the batch size of the input image and 2560 is the dimension of the feature vector of a single image. It should be noted that in actual use, the first prediction, that is, the image batch size for predicting global features is generally 1, and there is only a second prediction of the key area for multiple batch inputs.

[0065] Attention feature processing layer: The main structure of the attention feature processing layer is Attention Pooling, which is mainly used for adaptive feature extraction, and adapts different features to image feature encoding and attention area modules. Its input is the n×2560-dimensional feature vector calculated by the self-attention convolutional neural network, and its output is the n×768-dimensional embedding vector. This module is used to adaptively distinguish the features required for classification at different levels and encode them into corresponding embedding vectors. Its structure is a single-layer multi-head attention structure.

[0066] Image feature comparison: After completing the image prediction of the similar area and the key area, first add the feature vector of the similar area and the feature vector of the key area into a vector of 1x 768 dimensions. After merging, perform L2 norm processing on the vector, and the formula is as follows:

[0067]

[0068]

[0069] Among them, v represents the vector of 1x768 dimension, a i is the component of the vector, eps is a very small number to avoid the denominator being 0, usually set to 10 -12 .

[0070] After L2 norm processing, the vector is used to calculate the cosine similarity distance with the scene / landmark features in the feature library. The formula is as follows:

[0071]

[0072] Where v represents the current vector, v t Represents the vector in the feature library. The calculation result is the distance. The smaller the distance, the higher the similarity. After all calculations are completed, the most similar scene / landmark below the threshold is taken as the recognition result. If all distances are greater than the threshold, it is judged as no result. In our experiments, the distance threshold is generally 0.8.

[0073] In one embodiment of the present invention, the specific process of "training the self-attention convolutional neural network through the training image data and the label data to obtain a trained self-attention convolutional neural network" in step S120 can be further explained in combination with the following description.

[0074] As described in the following steps, the self-attention convolutional neural network is trained according to the training image data and the label data to obtain a primary self-attention convolutional neural network;

[0075] As described in the following steps, the primary self-attention convolutional neural network is reparameterized to obtain a trained self-attention convolutional neural network.

[0076] In one embodiment of the present invention, the specific process of the step of "training the self-attention convolutional neural network based on the training image data and the label data to obtain a primary self-attention convolutional neural network" can be further explained in combination with the following description.

[0077] As described in the following steps, the weights of the self-attention convolutional neural network are fixed, and the attention region module is trained to obtain the attention region module;

[0078] As described in the following steps, the weights of the attention region module are fixed, and the self-attention convolutional neural network is trained to obtain the self-attention region module;

[0079] As described in the following steps, a primary self-attention convolutional neural network is obtained based on the attention module and the self-attention module.

[0080] It should be noted that based on the use of RepVGG pre-trained model weights, the training steps can be divided into two steps of the cycle until the network converges:

[0081] Training the attention region module: In this step, only the attention region module is trained by fixing the weights of the self-attention convolutional neural network;

[0082] Training the self-attention convolutional neural network: In this step, only the self-attention convolutional neural network is trained by fixing the weights of the attention region module;

[0083] It should be noted that during the training process, the ArcFace classification head is used after the image features to train the image feature encoding, and the feature comparison part is not considered; while the prediction process will remove the ArcFace classification head and only use the image features for comparison.

[0084] In one embodiment of the present invention, the specific process of the step of "re-parameterizing the primary self-attention convolutional neural network to obtain a trained self-attention convolutional neural network" can be further explained in combination with the following description.

[0085] As described in the following steps, the convolutional layers and the pass-through layers in the primary self-attention convolutional neural network branch are converted into 3×3 convolutional convolutional layers to be fused;

[0086] As described in the following steps, all the convolutional layers to be fused are fused into a single 3×3 convolutional layer to obtain a trained self-attention convolutional neural network.

[0087] It should be noted that for edge computing devices, the RepVGG structure is used as the structural reference of the skeleton network, and the model reparameterization technology is applied on this basis. The schematic diagram of the structural change is shown in Figure 2 The steps are mainly divided into two steps: the first step: convert the 1x1 convolution layer, 3x3 convolution layer and straight-through layer in the branch into the form of 3x3 convolution; the second step: add and fuse these convolution layers into a single 3x3 convolution layer.

[0088] Because most edge computing devices and dedicated computing devices are optimized for the most widely used 3x3 convolution in chip design, the reparameterization method is used to convert the original multi-branch convolutional neural network into a straight-through neural network composed entirely of 3x3 convolutions, which greatly improves the density on edge computing devices. Therefore, after using the reparameterization process, the neural network can still make fast predictions on edge computing devices with limited computing power and power consumption while maintaining its original accuracy.

[0089] In one embodiment of the present invention, it further includes:

[0090] Performing model distillation on the self-attention convolutional neural network to obtain a distilled neural network;

[0091] Testing the distilled neural network to determine the accuracy difference between the distilled neural network and the self-attention convolutional neural network;

[0092] When the accuracy difference is lower than a preset value, the distilled neural network is deployed to an edge computing device.

[0093] It should be noted that this solution has the characteristics of high computing density and simple structure, which enables it to better adapt to deployment on edge computing devices. It has been proven to be deployable on edge computing devices of various architectures. The engineering deployment is mainly divided into the following steps: model distillation, model format conversion, model quantization, model deployment and accuracy analysis.

[0094] Due to the limited deployment resources of edge computing devices, the model needs to be distilled.

[0095] The basic idea of ​​model distillation is to extract knowledge from a larger model (teacher model) and transfer it to a smaller model (student model). The specific method is to first use the teacher model to predict the data set and record the predicted probability distribution of each sample, and then define a hybrid loss function that combines soft label loss and hard label loss. The soft label loss measures the gap between the predicted probability distribution of the student model and the predicted probability distribution of the teacher model, while the hard label encourages the student model to correctly predict the true label. The formula for this loss is:

[0096]

[0097] Among them, the soft label loss uses MSE loss, and the hard label loss uses b. The model is used for training and prediction. The image features are first L2 normalized and then the cosine similarity is calculated. The parameter α is the weight to measure the proportion of soft labels and hard labels.

[0098] Using the input and output of the teacher model, in this technology, the input is the image for review and the output is the result of image recognition. This is used as training data to supervise the student model and use the previously defined loss function to eventually obtain a student model that has transferred the knowledge of the teacher model.

[0099] When performing model distillation, it is necessary to perform a temperature increase operation and combine the temperature parameter T when calculating softmax:

[0100]

[0101] This method can make the output results of different categories be considered in a balanced manner.

[0102] The original parameter volume of the model is about 44.5M, and the parameter volume of the small model obtained after distillation is about 11.2M, which greatly reduces the complexity of the model and the demand for computing resources, and improves the efficiency of the model's operation.

[0103] Model format conversion: For the neural network model trained using the pytorch framework, in order to adapt it to the migration and reuse of various platforms, it must first be converted into the open ONNX format for saving. At the same time, edge computing devices of various architectures have their own specific deployment formats and supported operators, which have been taken into account when designing the algorithm.

[0104] Model quantization: To reduce memory usage and achieve model compression and inference acceleration, the model needs to be quantized before being deployed to edge computing devices. The conventional approach is to convert the model's weight parameters from FP32 format to FP16, INT4 or INT8 format with lower precision, and perform linear asymmetric quantization on all channels of the network layer as a whole or each channel as an independent whole.

[0105] Taking the linear asymmetric quantization of floating-point numbers to fixed-point numbers as an example, the calculation principle is:

[0106]

[0107] Where [] is a rounding operation, s is the quantization scale factor, z is the quantization zero point, b is the quantization bit width. If the INT8 format is 8, clamp is a truncation operation.

[0108] Since rounding and truncation operations are required when quantizing the model, these two operations will cause loss of precision and affect the performance of the model to a certain extent. Further analysis and testing of the model's precision and performance can be performed to determine the extent of the loss caused by this step.

[0109] Taking into account the accuracy loss of the model during the quantization process from floating-point numbers to fixed-point numbers, the quantization error is added to the training process as part of the loss function. The final trained model can still guarantee the original performance after quantization.

[0110] Model deployment: After model conversion and quantization are completed, connect the edge computing device and deploy the generated configuration on it. After deployment, inference testing can be performed on the device side.

[0111] Precision analysis: If the inference results of the quantization model are abnormal, precision analysis is required to check the calculation accuracy of the operators in each layer of the network.

[0112] The accuracy analysis of the model will compare the quantized model output results of each layer of the model with the original model output results, calculate the cosine distance and Euclidean distance, and adjust the model quantization scheme if there is a large gap.

[0113] In one embodiment of the present application, it also includes:

[0114] Determining the landmark building of the image to be identified according to the target similar area and the target key area;

[0115] A landmark recognition result is generated according to the landmark building.

[0116] It should be noted that first, the image feature codes of the target similar areas are extracted through prediction, and several target key areas are determined; then secondary prediction is performed through several target key areas to extract the image feature codes of the target key areas; finally, the image feature codes of the target similar areas are merged with the image feature codes of the target key areas and then an l2_norm operation is performed, and the cosine similarity is calculated with the scene / landmark features in the feature library, and the most similar scene / landmark below the threshold is taken as the recognition result.

[0117] In the above prediction process, the model extracts multiple target key areas for secondary prediction in order to enhance the model's accurate recognition of scenes / landmarks under different shooting angles, day and night changes, different weather conditions, and different viewing ranges, and to accurately distinguish the subtle differences between landmarks / places. Since it is often difficult to distinguish similar landmarks, such as TV towers in various places, similar buildings in different blocks, and similar airports in various places, by only extracting global image codes, the neural network can often extract and encode the key features that distinguish these landmarks / places, such as the difference between airport terminals, to achieve accurate distinction of landmarks / places. In our method, the first prediction and the second prediction use a neural network with the same weights, and the input is also resized and padded to 640x640.

[0118] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0119] Reference Figure 3 , showing a scene recognition device based on a self-attention convolutional neural network provided by an embodiment of the present application;

[0120] Specifically include:

[0121] The acquisition module 310 is used to acquire training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, wherein the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas;

[0122] A training module 320 is used to train the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network;

[0123] The recognition module 330 is used to obtain the image to be recognized, and process the image to be recognized through the trained self-attention convolutional neural network to obtain the target similar area and the target key area.

[0124] In one embodiment of the present invention, the acquisition module 310 includes:

[0125] Initial acquisition submodule, due to acquiring the initial training image;

[0126] An enhancement submodule, which performs enhancement processing on the initial training image to obtain enhanced image data;

[0127] A resistance submodule, which obtains resistance image data by performing resistance processing on the initial training image;

[0128] The training image data generating submodule generates the training image data according to the enhanced image data and the resistance image data.

[0129] In one embodiment of the present invention, the training module includes:

[0130] A primary training submodule, training the self-attention convolutional neural network according to the training image data and the label data to obtain a primary self-attention convolutional neural network;

[0131] The reparameterization submodule reparameterizes the primary self-attention convolutional neural network to obtain a trained self-attention convolutional neural network.

[0132] In one embodiment of the present invention, the primary training submodule includes:

[0133] Attention region unit, since the weights of the self-attention convolutional neural network are fixed, the attention region module is trained to obtain the attention region module;

[0134] Self-attention region unit, due to the fixed attention region module weight, the self-attention convolutional neural network is trained to obtain the self-attention region module;

[0135] A primary self-attention convolutional neural network unit is obtained based on the attention module and the self-attention module.

[0136] In one embodiment of the present invention, the re-parameterization submodule includes:

[0137] A conversion unit, which converts the convolutional layer and the pass-through layer in the primary self-attention convolutional neural network branch into a 3×3 convolutional convolutional layer to be fused;

[0138] The fusion unit fuses all the convolutional layers to be fused into a single 3×3 convolutional layer, thereby obtaining a trained self-attention convolutional neural network.

[0139] In one embodiment of the present invention, it also includes:

[0140] A distillation module, used for performing model distillation on the self-attention convolutional neural network to obtain a distilled neural network;

[0141] A testing module, used to test the distilled neural network to determine the accuracy difference between the distilled neural network and the self-attention convolutional neural network;

[0142] A deployment module is used to deploy the distillation neural network to an edge computing device when the accuracy difference is lower than a preset value.

[0143] In one embodiment of the present invention, it also includes:

[0144] A landmark module, used for determining the landmark building of the image to be identified based on the target similar area and the target key area;

[0145] The result generation module is used to generate a landmark recognition result according to the landmark building.

[0146] Reference Figure 4 , showing a computer device of a scene recognition method based on a self-attention convolutional neural network of the present invention, which may specifically include the following:

[0147] The computer device 12 is in the form of a general-purpose computing device, and the components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0148] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor or a local bus 18 using any of a variety of bus 18 architectures. These architectures include, by way of example, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.

[0149] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0150] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although Figure 4Not shown, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42, which are configured to perform the functions of various embodiments of the present invention.

[0151] A program / utility 40 having a set (at least one) of program modules 42 may be stored in, for example, a memory, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0152] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, cameras, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs)), wide area networks (WANs), and / or public networks (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34, etc.

[0153] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing a scene recognition method based on a self-attention convolutional neural network provided in an embodiment of the present invention.

[0154] That is, when the processing unit 16 executes the above program, the following is achieved: obtaining training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas;

[0155] Training the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network;

[0156] An image to be identified is obtained, and the image to be identified is processed by the trained self-attention convolutional neural network to obtain a target similar area and a target key area.

[0157] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a scene recognition method based on a self-attention convolutional neural network as provided in all embodiments of the present application:

[0158] That is, when the program is executed by the processor, the following steps are implemented: obtaining training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas;

[0159] Training the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network;

[0160] An image to be identified is obtained, and the image to be identified is processed by the trained self-attention convolutional neural network to obtain a target similar area and a target key area.

[0161] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device.

[0162] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0163] The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0164] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0165] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0166] The above is a detailed introduction to the scene recognition method and device based on the self-attention convolutional neural network provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A scene recognition method based on self-attention convolutional neural network, characterized in that: Includes steps: Acquire training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas; Training the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network; An image to be identified is obtained, and the image to be identified is processed by the trained self-attention convolutional neural network to obtain a target similar area and a target key area.

2. The identification method according to claim 1, characterized in that: The step of acquiring training image data and label data corresponding to the training image data, wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas, comprises: Get initial training images; Performing enhancement processing on the initial training image to obtain enhanced image data; Performing resistance processing on the initial training image to obtain resistance image data; Training image data is generated based on the enhanced image data and the resistant image data.

3. The identification method according to claim 1, characterized in that: The step of training the self-attention convolutional neural network by using the training image data and the label data to obtain a trained self-attention convolutional neural network comprises: Training the self-attention convolutional neural network according to the training image data and the label data to obtain a primary self-attention convolutional neural network; The primary self-attention convolutional neural network is re-parameterized to obtain a trained self-attention convolutional neural network.

4. The method according to claim 3, characterized in that The step of training the self-attention convolutional neural network based on the training image data and the label data to obtain a primary self-attention convolutional neural network includes: Fix the weights of the self-attention convolutional neural network and train the attention region module to obtain the attention region module; The weight of the attention region module is fixed, and the self-attention convolutional neural network is trained to obtain the self-attention region module; A primary self-attention convolutional neural network is obtained based on the attention module and the self-attention module.

5. The method according to claim 3, characterized in that: The step of re-parameterizing the primary self-attention convolutional neural network to obtain a trained self-attention convolutional neural network includes: Convert the convolutional layers and pass-through layers in the primary self-attention convolutional neural network branch into 3×3 convolutional convolutional layers to be fused; All the convolutional layers to be fused are fused into a single 3×3 convolutional layer to obtain a trained self-attention convolutional neural network.

6. The method according to claim 1, characterized in that Also includes: Performing model distillation on the self-attention convolutional neural network to obtain a distilled neural network; Testing the distilled neural network to determine the accuracy difference between the distilled neural network and the self-attention convolutional neural network; When the accuracy difference is lower than a preset value, the distilled neural network is deployed to an edge computing device.

7. The method according to claim 1, characterized in that Also includes: Determining the landmark building of the image to be identified according to the target similar area and the target key area; A landmark recognition result is generated according to the landmark building.

8. A scene recognition device based on a self-attention convolutional neural network, characterized in that: include: An acquisition module, used to acquire training image data and label data corresponding to the training image data; wherein the label data includes similar areas and key areas, the similar areas are areas with differences within a preset range, and the key areas are areas where fixed buildings appear within a preset range from the similar areas; A training module, used to train the self-attention convolutional neural network through the training image data and the label data to obtain a trained self-attention convolutional neural network; The recognition module is used to obtain the image to be recognized, and process the image to be recognized through the trained self-attention convolutional neural network to obtain the target similar area and the target key area.

9. A device, characterized in that: It comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the scene recognition method based on a self-attention convolutional neural network as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the scene recognition method based on a self-attention convolutional neural network as described in any one of claims 1 to 7 is implemented.

Images