A scene recognition method and device based on a self-attention convolutional neural network

Abstract

The application provides a scene recognition method based on a self-attention convolutional neural network, comprising the steps of: acquiring training image data and label data corresponding to the training image data; wherein the label data comprises a similar region and a key region, the similar region is a region with a difference within a preset range, and the key region is a region in which a fixed building appears within a preset range from the similar region; training the self-attention convolutional neural network through the training image data and the label data to obtain a trained self-attention convolutional neural network; acquiring an image to be recognized, processing the image to be recognized through the trained self-attention convolutional neural network, and obtaining a target similar region and a target key region. By recognizing the target key region, different point positions in a similar scene are determined, specific landmark buildings of the scene are determined, and the interference of the surrounding environment when the neural network recognizes the scene is reduced.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a scene recognition method and apparatus based on a self-attention convolutional neural network. Background Technology

[0002] Generally, the task of recognizing specific scenes and landmarks is considered an instance retrieval task. Convolutional neural networks (CNNs) are now widely used in various recognition and detection tasks, and they are also frequently used in instance retrieval tasks for image feature extraction. For example, NetVLAD replaces SIFT (Scale-invariant feature transform) in VLAD with convolution to extract local features, or it uses an end-to-end approach to train classification networks. However, existing CNNs still have some problems in specific scene and landmark recognition tasks. For instance, some neural networks are poorly adapted to day-night changes and changes in the surrounding environment, and neural networks trained based on classification methods cannot respond to scenes and locations outside their classification.

[0003] Meanwhile, the development of edge computing devices in recent years has shown significant advantages over traditional cloud deployments in many scenarios in terms of cost and flexibility. However, the limitations of computing power and energy consumption for edge computing devices are a problem that cannot be ignored. Summary of the Invention

[0004] In view of the aforementioned problems, this application is proposed to provide a scene recognition method and apparatus based on a self-attention convolutional neural network that overcomes or at least partially solves the aforementioned problems, comprising:

[0005] A scene recognition method based on a self-attention convolutional neural network includes the following steps:

[0006] Acquire training image data and corresponding label data; wherein, the label data includes similar regions and key regions, the similar regions are regions with differences within a preset range, and the key regions are regions with fixed buildings within a preset range from the similar regions;

[0007] The self-attention convolutional neural network is trained using the training image data and the label data to obtain the trained self-attention convolutional neural network.

[0008] The image to be identified is acquired, and the trained self-attention convolutional neural network is used to process the image to obtain the target similar region and the target key region.

[0009] Further, the step of acquiring training image data and corresponding label data, wherein the label data includes similar regions and key regions, the similar regions being regions with differences within a preset range, and the key regions being regions with fixed buildings appearing within a preset range of the similar regions, includes:

[0010] Obtain the initial training images;

[0011] The initial training images are enhanced to obtain enhanced image data;

[0012] The initial training images are subjected to resistance processing to obtain resistance image data;

[0013] Training image data is generated based on the enhanced image data and the resistant image data.

[0014] Further, the step of training the self-attention convolutional neural network using the training image data and the label data to obtain the trained self-attention convolutional neural network includes:

[0015] The self-attention convolutional neural network is trained based on the training image data and the label data to obtain a primary self-attention convolutional neural network;

[0016] The primary self-attention convolutional neural network is reparameterized to obtain a trained self-attention convolutional neural network.

[0017] Further, the step of training the self-attention convolutional neural network based on the training image data and the label data to obtain a primary self-attention convolutional neural network includes:

[0018] By fixing the weights of the self-attention convolutional neural network, the attention region module is trained to obtain the attention region module;

[0019] By fixing the weights of the attention region modules, a self-attention convolutional neural network is trained to obtain the self-attention region modules;

[0020] A primary self-attention convolutional neural network is obtained based on the attention module and the self-attention module.

[0021] Further, the step of reparameterizing the primary self-attention convolutional neural network to obtain the trained self-attention convolutional neural network includes:

[0022] The convolutional layers and pass-through layers in the primary self-attention convolutional neural network branch are all converted into 3×3 convolutional layers to be fused.

[0023] All the convolutional layers to be fused are merged into a single 3×3 convolutional layer to obtain a trained self-attention convolutional neural network.

[0024] Furthermore, it also includes:

[0025] The self-attention convolutional neural network is model distilled to obtain a distilled neural network;

[0026] The distillation neural network is tested to determine the accuracy difference between the distillation neural network and the self-attention convolutional neural network;

[0027] When the accuracy difference is lower than a preset value, the distillation neural network is deployed to an edge computing device.

[0028] Furthermore, it also includes:

[0029] The landmark buildings in the image to be identified are determined based on the target similarity region and the target key region;

[0030] Based on the landmark buildings, generate landmark recognition results.

[0031] A scene recognition device based on a self-attention convolutional neural network includes:

[0032] An acquisition module is used to acquire training image data and label data corresponding to the training image data; wherein, the label data includes similar regions and key regions, the similar regions are regions with differences within a preset range, and the key regions are regions with fixed buildings within a preset range from the similar regions;

[0033] The training module is used to train the self-attention convolutional neural network using the training image data and the label data to obtain the trained self-attention convolutional neural network.

[0034] The recognition module is used to acquire the image to be recognized and process the image to be recognized through the trained self-attention convolutional neural network to obtain the target similar region and the target key region.

[0035] An apparatus includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the scene recognition method based on a self-attention convolutional neural network as described above.

[0036] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the scene recognition method based on a self-attention convolutional neural network as described above.

[0037] This application has the following advantages:

[0038] In the embodiments of this application, compared with the shortcomings of existing neural network scene recognition technologies, such as poor adaptability and inability to accurately identify similar scenes, this application provides a solution for identifying specific scenes by identifying key regions. Specifically, it is a scene recognition method based on a self-attention convolutional neural network, comprising the steps of: acquiring training image data and label data corresponding to the training image data; wherein, the label data includes similar regions and key regions, the similar regions being regions with differences within a preset range, and the key regions being regions with fixed buildings within a preset range from the similar regions; training the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network; acquiring an image to be identified, and processing the image to be identified using the trained self-attention convolutional neural network to obtain target similar regions and target key regions. By identifying the target key regions, the positions of different points in similar scenes are determined, and the specific landmark buildings in the scene are determined based on the positions of the different points, thereby improving the accuracy of recognition and reducing interference from the surrounding environment when the neural network identifies the scene. Attached Figure Description

[0039] To more clearly illustrate the technical solution of this application, the drawings used in the description of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the steps of a scene recognition method based on a self-attention convolutional neural network according to an embodiment of this application.

[0041] Figure 2 This is a schematic diagram illustrating the empty skinning change of the self-attention convolutional neural network in a scene recognition method based on a self-attention convolutional neural network provided in an embodiment of this application;

[0042] Figure 3 This is a structural block diagram of a scene recognition device based on a self-attention convolutional neural network according to an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, features, and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0045] The inventors discovered through analysis of existing technologies that when identifying landmark buildings or specific scenes, existing technologies only identify the main landmark scenes and do not pay much attention to other elements in the image. This leads to mutual interference when identifying similar landmark buildings or similar specific scenes, or the influence of environmental changes, resulting in identification errors.

[0046] Reference Figure 1-3 This illustration shows a scene recognition method and apparatus based on a self-attention convolutional neural network according to an embodiment of this application;

[0047] The method includes:

[0048] S110. Acquire training image data and label data corresponding to the training image data; wherein, the label data includes similar regions and key regions, the similar regions are regions with differences within a preset range, and the key regions are regions where fixed buildings appear within a preset range from the similar regions;

[0049] S120. The self-attention convolutional neural network is trained using the training image data and the label data to obtain the trained self-attention convolutional neural network.

[0050] S130. Obtain the image to be identified, and process the image to be identified through the trained self-attention convolutional neural network to obtain the target similar region and the target key region.

[0051] In the embodiments of this application, compared with the shortcomings of existing neural network scene recognition technologies, such as poor adaptability and inability to accurately identify similar scenes, this application provides a solution for identifying specific scenes by identifying key regions. Specifically, it is a scene recognition method based on a self-attention convolutional neural network, comprising the steps of: acquiring training image data and label data corresponding to the training image data; wherein, the label data includes similar regions and key regions, the similar regions being regions with differences within a preset range, and the key regions being regions with fixed buildings within a preset range from the similar regions; training the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network; acquiring an image to be identified, and processing the image to be identified using the trained self-attention convolutional neural network to obtain target similar regions and target key regions. By identifying the target key regions, the positions of different points in similar scenes are determined, and the specific landmark buildings in the scene are determined based on the positions of the different points, thereby improving the accuracy of recognition and reducing interference from the surrounding environment when the neural network identifies the scene.

[0052] The following will further explain a scene recognition method based on a self-attention convolutional neural network in this exemplary embodiment.

[0053] As described in step S110, training image data and corresponding label data are acquired; wherein, the label data includes similar regions and key regions, the similar regions are regions with differences within a preset range, and the key regions are regions where fixed buildings appear within a preset range from the similar regions.

[0054] It should be noted that the similar area refers to a landmark building or a specific scene, and the key area refers to other representative features near the landmark building or specific scene. In the identification process, there can be multiple key areas to improve the accuracy of the identification.

[0055] As an example, similar regions can be representative landmarks in an image, such as airport areas, temple areas, or lighthouse areas. However, similar landmarks are difficult to distinguish. Key regions can be shop signs next to an airport, sculptures in front of a temple, or buildings in front of a lighthouse. For such similar regions, key regions can provide more precise feature areas to further identify images with similar regions, thereby identifying specific landmarks or scenes within those similar regions and improving the accuracy of the identification.

[0056] In one embodiment of the present invention, the specific process of step S110, "acquiring training image data and label data corresponding to the training image data; wherein the label data includes similar regions and key regions, the similar regions are regions with differences within a preset range, and the key regions are regions with fixed buildings appearing within a preset range from the similar regions," can be further explained in conjunction with the following description.

[0057] Obtain the initial training images as described in the following steps;

[0058] The initial training image is enhanced to obtain enhanced image data as described in the following steps;

[0059] As described in the following steps, the initial training images are subjected to resistance processing to obtain resistance image data;

[0060] Training image data is generated based on the enhanced image data and the resistant image data, as described in the following steps.

[0061] It should be noted that data augmentation processing, such as rotation, mirror flipping, multi-image stitching, and affine transformation, is applied to the initial training images; as well as adversarial image data augmentation processing, such as blurring, noise reduction, texture enhancement, and occlusion, to obtain an augmented dataset. This yields the training image data, ensuring its training performance and guaranteeing the recognition capability of the self-attention convolutional neural network after training.

[0062] As described in step S120, the self-attention convolutional neural network is trained using the training image data and the label data to obtain the trained self-attention convolutional neural network.

[0063] It should be noted that self-attention convolutional neural networks are primarily used for feature extraction from images submitted for review. This neural network mainly consists of convolutional layers and self-attention layers. During training, its main structure exhibits a multi-branch model structure, while in the prediction mode after reparameterization, the main structure exhibits a single-path model structure. The structural changes are illustrated in the diagram below. Figure 2 The main change is in the convolution part.

[0064] The output of a self-attention convolutional neural network is an n×2560 dimensional vector, where n is the batch size of the input images and 2560 is the dimension of the feature vector of a single image. It's important to note that in practice, the batch size for the first prediction (predicting global features) is typically 1; multiple batches of input only result in a second prediction of key regions.

[0065] Attention Feature Processing Layer: The main structure of the attention feature processing layer is Attention Pooling, whose primary function is adaptive feature extraction, adapting different features for image feature encoding and the attention region module. Its input is an n×2560-dimensional feature vector calculated by a self-attention convolutional neural network, and its output is an n×768-dimensional embedding vector. This module is used to adaptively distinguish the features required for different classification levels and encode them into corresponding embedding vectors. Its structure is a single-layer multi-head attention structure.

[0066] Image feature comparison: After predicting similar and key regions, the feature vectors of the similar and key regions are first combined into a 1x768 dimensional vector. After merging, this vector is then subjected to L2 norm processing, using the following formula:

[0067]

[0068]

[0069] Where v represents the 1x768 dimension vector, a i For the components of the vector, eps is a very small number to avoid a denominator of 0, and is generally set to 10. -12 .

[0070] After L2 norm processing, the vector is used to calculate the cosine similarity distance with scene / landmark features in the feature library, as shown in the following formula:

[0071]

[0072] Where v represents the current vector, v t The vectors represent the features in the feature library, and the calculated result is the distance. The smaller the distance, the higher the similarity. After all calculations are completed, the most similar scene / landmark below the threshold is taken as the recognition result. If all distances are greater than the threshold, it is determined that there is no result. In our experiments, this distance threshold is generally set to 0.8.

[0073] In one embodiment of the present invention, the specific process of step S120, "training the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network," can be further explained in conjunction with the following description.

[0074] As described in the following steps, the self-attention convolutional neural network is trained based on the training image data and the label data to obtain a primary self-attention convolutional neural network;

[0075] The primary self-attention convolutional neural network is reparameterized as described in the following steps to obtain a trained self-attention convolutional neural network.

[0076] In one embodiment of the present invention, the specific process of the step "training the self-attention convolutional neural network based on the training image data and the label data to obtain a primary self-attention convolutional neural network" can be further explained in conjunction with the following description.

[0077] As described in the following steps, the weights of the self-attention convolutional neural network are fixed, and the attention region module is trained to obtain the attention region module;

[0078] As described in the following steps, the weights of the attention region modules are fixed, and the self-attention convolutional neural network is trained to obtain the self-attention region modules;

[0079] As described in the following steps, a primary self-attention convolutional neural network is obtained based on the attention module and the self-attention module.

[0080] It should be noted that, based on the RepVGG pre-trained model weights, the training process can be divided into two iterative steps until the network converges:

[0081] Training the attention region module: In this step, the attention region module is trained only by fixing the weights of the self-attention convolutional neural network;

[0082] Training the self-attention convolutional neural network: In this step, the weights of the attention region modules are fixed, and only the self-attention convolutional neural network is trained.

[0083] It is important to note that during training, the ArcFace classification head is used to train image feature encoding after image features, without considering the feature comparison part; while in the prediction process, the ArcFace classification head will be removed, and only image features will be used for comparison.

[0084] In one embodiment of the present invention, the specific process of the step "reparameterizing the primary self-attention convolutional neural network to obtain a trained self-attention convolutional neural network" can be further explained in conjunction with the following description.

[0085] As described in the following steps, the convolutional layers and pass-through layers in the primary self-attention convolutional neural network branch are all converted into 3×3 convolutional layers to be fused.

[0086] As described in the following steps, all the convolutional layers to be fused are fused into a single 3×3 convolutional layer to obtain a trained self-attention convolutional neural network.

[0087] It should be noted that, for edge computing devices, the RepVGG structure was used as the structural reference for the backbone network, and model reparameterization technology was applied on this basis. A schematic diagram of the structural changes is shown below. Figure 2 As shown. The process mainly consists of two steps: First, convert the 1x1 convolutional layer, 3x3 convolutional layer, and pass-through layer in the branch into 3x3 convolutional layers; Second, merge these convolutional layers into a single 3x3 convolutional layer.

[0088] Because most edge computing devices and dedicated computing devices are designed with 3x3 convolutions as the most widely used technology in mind, reparameterization transforms the original multi-branch convolutional neural network into a pass-through neural network composed entirely of 3x3 convolutions, significantly increasing density on edge computing devices. Therefore, after reparameterization, the neural network can still perform fast predictions on edge computing devices with limited computing power and power consumption while maintaining its original accuracy.

[0089] In one embodiment of the present invention, it further includes;

[0090] The self-attention convolutional neural network is model distilled to obtain a distilled neural network;

[0091] The distillation neural network is tested to determine the accuracy difference between the distillation neural network and the self-attention convolutional neural network;

[0092] When the accuracy difference is lower than a preset value, the distillation neural network is deployed to an edge computing device.

[0093] It should be noted that this solution features high computational density and simple structure, making it better suited for deployment on edge computing devices. It has been proven to be deployable on edge computing devices with various architectures. The engineering deployment mainly consists of the following steps: model distillation, model format conversion, model quantization, model deployment, and accuracy analysis.

[0094] Due to limited deployment resources for edge computing devices, model distillation is necessary.

[0095] The basic idea of ​​model distillation is to extract knowledge from a larger model (teacher model) and transfer it to a smaller model (student model). Specifically, the teacher model is first used to predict the dataset and record the predicted probability distribution for each sample. Then, a hybrid loss function is defined, combining soft-label loss and hard-label loss. The soft-label loss measures the difference between the predicted probability distribution of the student model and the teacher model, while the hard-label loss encourages the student model to correctly predict the true labels. The formula for this loss is:

[0096]

[0097] The model uses MSE loss for soft label loss and b for hard label loss. During training and prediction, the image features are first L2 normalized and then cosine similarity is calculated. The parameter α is the weight that measures the proportion of soft and hard labels.

[0098] Using the input and output of the teacher model (in this technique, the input is the image to be reviewed, and the output is the image recognition result), the student model is supervised and trained using the training data. By using the loss function defined earlier, a student model that has transferred the knowledge of the teacher model can be obtained.

[0099] When performing model distillation, a heating operation is required, and the temperature parameter T is considered when calculating the softmax.

[0100]

[0101] This method allows for a balanced consideration of output results from different classes.

[0102] The original model had approximately 44.5M parameters, while the smaller model obtained after distillation had approximately 11.2M parameters. This significantly reduced the model's complexity and computational resource requirements, and improved the model's efficiency.

[0103] Model format conversion: For neural network models trained using the PyTorch framework, in order to adapt them to migration and reuse across various platforms, they first need to be converted to the open ONNX format for storage. At the same time, edge computing devices of different architectures have their own specific deployment formats and supported operators, which has been taken into account when designing the algorithm.

[0104] Model quantization: To reduce memory usage and achieve model compression and inference acceleration, the model needs to be quantized before being deployed to edge computing devices. The common practice is to convert the model's weight parameters from FP32 format to the less precise FP16, INT4 or INT8 format, and to perform linear asymmetric quantization on all channels of the network layer as a whole or on each channel as an independent whole.

[0105] Taking linear asymmetric quantization for converting floating-point numbers to fixed-point numbers as an example, the calculation principle is as follows:

[0106]

[0107] Where [] represents rounding, s is the quantization scaling factor, z is the quantization zero point, b is the quantization bit width (e.g., 8 for INT8 format), and clamp is the truncation operation.

[0108] Since rounding and truncation operations are required during model quantization, these operations can lead to a loss of accuracy and affect model performance to some extent. Further analysis and testing of the model's accuracy and performance can be conducted to determine the extent of the loss caused by this step.

[0109] Considering the precision loss that occurs during the quantization process from floating-point to fixed-point numbers, the quantization error is incorporated as part of the loss function during training. As a result, the trained model can still maintain its original performance after quantization.

[0110] Model Deployment: After the model conversion and quantization are completed, connect to the edge computing device to deploy the generated configuration. After deployment, inference testing can be performed on the device side.

[0111] Accuracy analysis: If the inference results of the quantization model are abnormal, accuracy analysis is required to check the computational accuracy of the operators in each layer of the network.

[0112] The model accuracy analysis compares the quantized output of each layer of the model with the original output, calculates the cosine distance and Euclidean distance, and adjusts the model quantization scheme if there is a large discrepancy.

[0113] In one embodiment of this application, it further includes:

[0114] The landmark buildings in the image to be identified are determined based on the target similarity region and the target key region;

[0115] Based on the landmark buildings, generate landmark recognition results.

[0116] It should be noted that, firstly, image feature codes of similar target regions are extracted through prediction, and several key target regions are identified; then, secondary prediction is performed using these key target regions to extract image feature codes of the key target regions; finally, the image feature codes of similar target regions and key target regions are merged and subjected to L2-norm operation, and cosine similarity is calculated with scene / landmark features in the feature library. The most similar scene / landmark below the threshold is taken as the recognition result.

[0117] In the aforementioned prediction process, the model extracts multiple key target areas for secondary prediction to enhance its accuracy in identifying scenes / landmarks under different shooting angles, day / night variations, weather conditions, and framing ranges, and to accurately distinguish subtle differences between landmarks / locations. Since extracting only global image encoding often fails to differentiate similar landmarks, such as TV towers in different locations, similar buildings in different neighborhoods, and similar airports in different locations, the addition of secondary prediction of key target areas allows the neural network to extract and encode key features that distinguish these landmarks / locations, such as the differences between airport terminals, thus achieving accurate landmark / location differentiation. In our method, the first and second predictions use neural networks with the same weights, and the input is resized and padded to 640x640 pixels.

[0118] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0119] Reference Figure 3 This illustrates a scene recognition device based on a self-attention convolutional neural network according to an embodiment of this application;

[0120] Specifically, it includes:

[0121] The acquisition module 310 is used to acquire training image data and label data corresponding to the training image data; wherein, the label data includes similar regions and key regions, the similar regions are regions with differences within a preset range, and the key regions are regions with fixed buildings appearing within a preset range of the similar regions;

[0122] Training module 320 is used to train the self-attention convolutional neural network using the training image data and the label data to obtain the trained self-attention convolutional neural network.

[0123] The recognition module 330 is used to acquire the image to be recognized and process the image to be recognized through the trained self-attention convolutional neural network to obtain the target similar region and the target key region.

[0124] In one embodiment of the present invention, the acquisition module 310 includes:

[0125] The initial acquisition submodule is used to acquire the initial training images;

[0126] The enhancement submodule obtains enhanced image data by performing enhancement processing on the initial training image;

[0127] The resistance submodule obtains resistance image data by performing resistance processing on the initial training images;

[0128] The training image data generation submodule generates training image data based on the enhanced image data and the resistant image data.

[0129] In one embodiment of the present invention, the training module includes:

[0130] The primary training submodule trains the self-attention convolutional neural network based on the training image data and the label data to obtain a primary self-attention convolutional neural network.

[0131] The reparameterization submodule reparameterizes the primary self-attention convolutional neural network to obtain a trained self-attention convolutional neural network.

[0132] In one embodiment of the present invention, the primary training submodule includes:

[0133] Attention region unit: By fixing the weights of the self-attention convolutional neural network, the attention region module is trained to obtain the attention region module;

[0134] Self-attention region units are obtained by training a self-attention convolutional neural network with fixed weights for the attention region modules.

[0135] The primary self-attention convolutional neural network unit is obtained by combining the attention module and the self-attention module.

[0136] In one embodiment of the present invention, the reparameterization submodule includes:

[0137] The conversion unit converts both the convolutional layers and pass-through layers in the primary self-attention convolutional neural network branch into 3×3 convolutional layers to be fused.

[0138] The fusion unit merges all the convolutional layers to be fused into a single 3×3 convolutional layer, resulting in a trained self-attention convolutional neural network.

[0139] In one embodiment of the present invention, it further includes:

[0140] The distillation module is used to perform model distillation on the self-attention convolutional neural network to obtain a distilled neural network.

[0141] The testing module is used to test the distillation neural network and determine the accuracy difference between the distillation neural network and the self-attention convolutional neural network.

[0142] The deployment module is used to deploy the distillation neural network to an edge computing device when the accuracy difference is lower than a preset value.

[0143] In one embodiment of the present invention, it further includes:

[0144] The landmark module is used to determine the landmark buildings in the image to be identified based on the target similarity region and the target key region;

[0145] The result generation module is used to generate landmark recognition results based on the landmark buildings.

[0146] Reference Figure 4 The computer device illustrating a scene recognition method based on a self-attention convolutional neural network according to the present invention may specifically include the following:

[0147] The computer device 12 described above is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0148] Bus 18 refers to one or more of several types of bus 18 architectures, including memory bus 18 or memory controller, peripheral bus 18, graphics acceleration port, processor, or local bus 18 using any of the various bus 18 architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus 18, Micro Channel Architecture (MAC) bus 18, Enhanced ISA bus 18, Audio / Video Electronics Standards Association (VESA) local bus 18, and Peripheral Component Interconnect (PCI) bus 18.

[0149] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0150] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as a "hard disk drive"). Figure 4Not shown, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 configured to perform the functions of the embodiments of the present invention.

[0151] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory. Such program modules 42 include—but are not limited to—an operating system, one or more application programs, other program modules 42, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0152] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, camera, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN)), wide area network (WAN), and / or public networks (e.g., the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although... Figure 4 Not shown, it can be combined with computer device 12 to use other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing unit 16, external disk drive array, RAID system, tape drive and data backup storage system 34, etc.

[0153] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a scene recognition method based on a self-attention convolutional neural network provided in the embodiments of the present invention.

[0154] That is, when the processing unit 16 executes the above program, it achieves the following: acquiring training image data and label data corresponding to the training image data; wherein, the label data includes similar regions and key regions, the similar regions are regions with differences within a preset range, and the key regions are regions where fixed buildings appear within a preset range from the similar regions;

[0155] The self-attention convolutional neural network is trained using the training image data and the label data to obtain the trained self-attention convolutional neural network.

[0156] The image to be identified is acquired, and the trained self-attention convolutional neural network is used to process the image to obtain the target similar region and the target key region.

[0157] In this embodiment of the invention, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a scene recognition method based on a self-attention convolutional neural network as provided in all embodiments of this application:

[0158] That is, when the program is executed by the processor, it implements the following: acquiring training image data and label data corresponding to the training image data; wherein, the label data includes similar regions and key regions, the similar regions are regions with differences within a preset range, and the key regions are regions with fixed buildings within a preset range from the similar regions;

[0159] The self-attention convolutional neural network is trained using the training image data and the label data to obtain the trained self-attention convolutional neural network.

[0160] The image to be identified is acquired, and the trained self-attention convolutional neural network is used to process the image to obtain the target similar region and the target key region.

[0161] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-to-signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0162] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0163] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider). The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0164] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0165] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0166] The above provides a detailed description of a scene recognition method and apparatus based on a self-attention convolutional neural network provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A scene recognition method based on a self-attention convolutional neural network, characterized in that, Including the following steps: Acquire training image data and corresponding label data; wherein, the label data includes similar regions and key regions, the similar regions are regions with differences within a preset range, and the key regions are regions with fixed buildings within a preset range from the similar regions; The self-attention convolutional neural network is trained using the training image data and the label data to obtain a trained self-attention convolutional neural network. Specifically, the self-attention convolutional neural network is trained using the training image data and the label data to obtain a primary self-attention convolutional neural network. The primary self-attention convolutional neural network is then reparameterized to obtain a trained self-attention convolutional neural network. The convolutional layers and pass-through layers in the branches of the primary self-attention convolutional neural network are all converted into 3×3 convolutional layers to be fused. All the convolutional layers to be fused are then fused into a single 3×3 convolutional layer to obtain a trained self-attention convolutional neural network. The image to be identified is acquired, and the trained self-attention convolutional neural network is used to process the image to obtain the target similar region and the target key region.

2. The identification method according to claim 1, characterized in that, The step of acquiring training image data and corresponding label data, wherein the label data includes similar regions and key regions, the similar regions being regions with differences within a preset range, and the key regions being regions with fixed buildings within a preset range of the similar regions, includes: Obtain the initial training images; The initial training images are enhanced to obtain enhanced image data; The initial training images are subjected to resistance processing to obtain resistance image data; Training image data is generated based on the enhanced image data and the resistant image data.

3. The method according to claim 1, characterized in that, Also includes: The self-attention convolutional neural network is model distilled to obtain a distilled neural network; The distillation neural network is tested to determine the accuracy difference between the distillation neural network and the self-attention convolutional neural network; When the accuracy difference is lower than a preset value, the distillation neural network is deployed to an edge computing device.

4. The method according to claim 1, characterized in that, Also includes: The landmark buildings in the image to be identified are determined based on the target similarity region and the target key region; Based on the landmark buildings, generate landmark recognition results.

5. A scene recognition device based on a self-attention convolutional neural network, characterized in that, include: An acquisition module is used to acquire training image data and label data corresponding to the training image data; wherein, the label data includes similar regions and key regions, the similar regions are regions with differences within a preset range, and the key regions are regions with fixed buildings within a preset range from the similar regions; The training module is used to train the self-attention convolutional neural network using the training image data and the label data to obtain a trained self-attention convolutional neural network. Specifically, the self-attention convolutional neural network is trained using the training image data and the label data to obtain a primary self-attention convolutional neural network; the primary self-attention convolutional neural network is reparameterized to obtain a trained self-attention convolutional neural network; the convolutional layers and pass-through layers in the branches of the primary self-attention convolutional neural network are converted into 3×3 convolutional layers to be fused; all the convolutional layers to be fused are fused into a single 3×3 convolutional layer to obtain a trained self-attention convolutional neural network. The recognition module is used to acquire the image to be recognized and process the image to be recognized through the trained self-attention convolutional neural network to obtain the target similar region and the target key region.

6. A scene recognition device based on a self-attention convolutional neural network, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When executed by the processor, the computer program implements the scene recognition method based on a self-attention convolutional neural network as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program implements the scene recognition method based on a self-attention convolutional neural network as described in any one of claims 1 to 4.

Images