Visual scene recognition method based on adaptive dynamic feature aggregation

Abstract

This invention discloses a visual scene recognition method based on adaptive dynamic feature aggregation. The method acquires a visual scene image to be recognized; inputs the visual scene image into a trained visual scene recognition model for scene recognition, and obtains the visual scene recognition result. The visual scene recognition model is trained through the following steps: acquiring a visual scene training image and extracting features from it to obtain local features; performing multi-layer dynamic pooling aggregation on the local features to obtain concatenated features; selecting prototype features from the concatenated features based on a learnable prototype to obtain globally optimized features; and updating the parameters of the initialized visual scene recognition model based on the globally optimized features to obtain a trained visual scene recognition model. The visual scene recognition model provided by this method can effectively improve the robustness and recognition effect of visual scene recognition. This invention relates to the field of computer vision technology.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a visual scene recognition method based on adaptive dynamic feature aggregation. Background Technology

[0002] The goal of visual scene recognition is to find data images in a database that match the query image, thereby determining the visual scene of the query image.

[0003] Currently, existing technologies typically aggregate local features of the query image and match the resulting global features with the global features of the data image to determine the visual scene of the query image. This method is easily affected by changes in the shape and scale of key objects in the query image. Specifically, when the shape and scale of key objects change significantly, the robustness and recognition effect of visual scene recognition are unsatisfactory.

[0004] Therefore, the problems existing in the current technology still need to be solved and optimized. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the related art.

[0006] Therefore, one objective of this invention is to provide a visual scene recognition method based on adaptive dynamic feature aggregation, which can effectively improve the robustness and recognition effect of visual scene recognition.

[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of this application include:

[0008] In a first aspect, embodiments of this application provide a visual scene recognition method based on adaptive dynamic feature aggregation, including:

[0009] Acquire the visual scene image to be identified;

[0010] The visual scene image is input into a trained visual scene recognition model for scene recognition, and the visual scene recognition result is obtained.

[0011] The trained visual scene recognition model is obtained through the following steps:

[0012] Acquire visual scene training images and extract features from the visual scene training images to obtain local features;

[0013] The local features are aggregated by multi-layer dynamic pooling to obtain spliced ​​features, which are used to characterize the global image information and local image information corresponding to the training image of the visual scene;

[0014] Based on the learnable prototype, prototype feature selection is performed on the spliced ​​features to obtain globally optimized features;

[0015] Based on the global optimization features, the parameters of the initialized visual scene recognition model are updated to obtain the trained visual scene recognition model.

[0016] In addition, the method according to the above embodiments of this application may also have the following additional technical features:

[0017] Furthermore, in one embodiment of this application, the visual scene recognition model includes several cascaded dynamic pooling network frameworks, and the multi-layer dynamic pooling aggregation of the local features to obtain spliced ​​features includes:

[0018] The local features are input into the plurality of cascaded dynamic pooling network frameworks to obtain a plurality of image features, each of the image features corresponding to one of the dynamic pooling network frameworks;

[0019] All the image features are flattened to obtain flattened features corresponding to different feature levels;

[0020] All the flattened features are spliced ​​together to obtain the spliced ​​features.

[0021] Furthermore, in one embodiment of this application, the dynamic pooling network framework is used to perform the following steps:

[0022] Obtain input features, wherein the input features are the local features or the image features output by the previous dynamic pooling network framework;

[0023] The input features are subjected to dual-information dynamic pooling to obtain the image features output by the current dynamic pooling network framework.

[0024] Furthermore, in one embodiment of this application, the step of performing dual-information dynamic pooling on the input features to obtain the image features output by the current dynamic pooling network framework includes:

[0025] Obtain a target offset map corresponding to the input feature, wherein the target offset map is used to record the pooling sampling offset of each feature pixel of the input feature;

[0026] Based on the target offset map, the input features are dynamically pooled to obtain the image features, wherein the width of the image features is smaller than the width of the input features, and the height of the image features is smaller than the height of the input features.

[0027] Furthermore, in one embodiment of this application, obtaining the offset map corresponding to the input feature includes:

[0028] Global self-attention feature extraction is performed on the input features to obtain a global offset map;

[0029] Local convolution feature extraction is performed on the input features to obtain a local offset map;

[0030] The global offset map and the local offset map are stitched together and optimized to obtain the target offset map.

[0031] Furthermore, in one embodiment of this application, the step of performing prototype feature selection on the spliced ​​features based on the learnable prototype to obtain the globally optimized features includes:

[0032] Based on the learnable prototype, dynamic importance element learning is performed on the spliced ​​features to obtain a target feature matrix. The target feature matrix is ​​used to characterize the set of similarities between each learnable prototype and each feature element of the spliced ​​features.

[0033] Based on the target feature matrix, the spliced ​​features are element-aggregated to obtain element-aggregated features;

[0034] The element aggregation features are optimized to obtain the global optimized features.

[0035] Further, in this embodiment of the application, the step of dynamically learning important elements of the spliced ​​features based on the learnable prototype to obtain the target feature matrix includes:

[0036] Based on the learnable prototype, matrix multiplication is performed on the spliced ​​features to obtain an intermediate feature matrix;

[0037] The intermediate feature matrix is ​​normalized by row operations to obtain the target feature matrix.

[0038] Secondly, embodiments of this application provide a visual scene recognition system based on adaptive dynamic feature aggregation, comprising:

[0039] The first processing unit is used to acquire the visual scene image to be recognized;

[0040] The second processing unit is used to input the visual scene image into the trained visual scene recognition model to perform scene recognition and obtain the visual scene recognition result.

[0041] The trained visual scene recognition model is obtained through the following steps:

[0042] Acquire visual scene training images and extract features from the visual scene training images to obtain local features;

[0043] The local features are aggregated by multi-layer dynamic pooling to obtain spliced ​​features, which are used to characterize the global image information and local image information corresponding to the training image of the visual scene;

[0044] Based on the learnable prototype, prototype feature selection is performed on the spliced ​​features to obtain globally optimized features;

[0045] Based on the global optimization features, the parameters of the initialized visual scene recognition model are updated to obtain the trained visual scene recognition model.

[0046] Thirdly, embodiments of this application also provide an electronic device, including:

[0047] At least one processor;

[0048] At least one memory for storing at least one program;

[0049] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0050] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a processor-executable program, which, when executed by the processor, is used to implement the above-described method.

[0051] The advantages and beneficial effects of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application:

[0052] This application discloses a visual scene recognition method based on adaptive dynamic feature aggregation. The method acquires a visual scene image to be recognized; inputs the visual scene image into a trained visual scene recognition model for scene recognition, and obtains a visual scene recognition result. The trained visual scene recognition model is trained through the following steps: acquiring a visual scene training image and extracting features from the visual scene training image to obtain local features; performing multi-layer dynamic pooling aggregation on the local features to obtain stitched features, which represent global image information and local image information corresponding to the visual scene training image; selecting prototype features from the stitched features based on a learnable prototype to obtain globally optimized features; and updating the parameters of the initialized visual scene recognition model based on the globally optimized features to obtain the trained visual scene recognition model. This method captures global scene information and local object information at different feature levels by performing multi-layer dynamic pooling aggregation on local features of training images of visual scenes. Based on the captured global scene information and local object information, the aggregation region is dynamically adjusted, so that the method can still adaptively capture key object features of different scales and geometric shapes when the shape and scale of objects change. This can effectively improve the robustness and recognition effect of visual scene recognition. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0054] Figure 1 A flowchart illustrating a visual scene recognition method based on adaptive dynamic feature aggregation provided in an embodiment of this application;

[0055] Figure 2 A schematic diagram illustrating the training process of a visual scene recognition model provided in an embodiment of this application;

[0056] Figure 3 A schematic diagram of the structural framework of a visual scene recognition system based on adaptive dynamic feature aggregation provided in an embodiment of this application;

[0057] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0060] Currently, existing technologies typically aggregate local features of the query image and match the resulting global features with the global features of the data image to determine the visual scene of the query image. This approach usually treats all aggregated regions in the query image as equally important, without distinguishing the importance of the aggregated regions, and cannot adaptively prioritize important object information. This results in unsatisfactory robustness and recognition performance when the shape and scale of key objects vary greatly.

[0061] In view of this, embodiments of the present invention provide a visual scene recognition method based on adaptive dynamic feature aggregation. This method performs multi-layer dynamic pooling aggregation on local features of visual scene training images, capturing global scene information and local object information at different feature levels. Based on the captured global scene information and local object information, the aggregation region is dynamically adjusted. This allows the method to capture key object features of different scales and geometries even when object shape and scale change, by adaptively shifting focus to the most important spatial region, effectively improving the robustness and recognition performance of visual scene recognition. Furthermore, this method also utilizes prototype-based feature selection to eliminate redundant information in image features at different levels and highlight key spatial information, further enhancing the robustness of visual scene recognition.

[0062] Reference Figure 1 and Figure 2 In this embodiment of the application, a visual scene recognition method based on adaptive dynamic feature aggregation includes:

[0063] Step 110: Obtain the visual scene image to be recognized;

[0064] Step 120: Input the visual scene image into the trained visual scene recognition model for scene recognition to obtain the visual scene recognition result;

[0065] In this embodiment of the application, the visual scene image to be identified can be an image acquired by a camera device in a real environment. After acquiring the visual scene image, the visual scene image can be input into a pre-trained visual scene recognition model for scene recognition, so as to obtain a visual scene recognition result with object recognition information in a certain visual scene.

[0066] The trained visual scene recognition model is obtained through the following steps:

[0067] Step 130: Obtain visual scene training images and extract features from the visual scene training images to obtain local features;

[0068] In the embodiments of this application, the visual scene training images can be visual scene images from visual scene training sets publicly available on the Internet, or historical visual scene images collected in the actual environment. There are various ways to obtain the visual scene training images, which will not be described in detail here.

[0069] Furthermore, the feature extraction in step 130 can first involve obtaining a visual feature extractor, and then inputting the visual scene training image into the visual feature extractor to extract local features, thereby obtaining local features that record the local details of the visual scene training image.

[0070] Step 140: Perform multi-layer dynamic pooling aggregation on the local features to obtain spliced ​​features, which are used to characterize the global image information and local image information corresponding to the visual scene training image;

[0071] In this embodiment of the application, the multi-layer dynamic pooling aggregation in step 140 is used to extract image features at different feature levels corresponding to the visual scene training image. Specifically, for a certain level of dynamic pooling aggregation, it can be used to abstract the local features corresponding to the visual scene training image through adaptively dynamically changing aggregation pooling regions to extract image features with higher-level information. The dynamic pooling aggregation of other levels is similar. Then, the image features at all different feature levels are aggregated and stitched together to obtain the stitched features that represent the global image information and local image information corresponding to the visual scene training image.

[0072] In some embodiments, the visual scene recognition model includes several cascaded dynamic pooling network frameworks. Step 140, performing multi-layer dynamic pooling aggregation on the local features to obtain concatenated features, includes:

[0073] A1. Input the local features into the plurality of cascaded dynamic pooling network frameworks to obtain a plurality of image features, each of the image features corresponding to one of the dynamic pooling network frameworks;

[0074] A2. Flatten all the image features to obtain flattened features corresponding to different feature levels;

[0075] A3. Perform feature stitching on all the flattened features to obtain the stitched features.

[0076] In this embodiment, the specific number of dynamic pooling network frameworks can be set according to actual conditions, such as any one of 2, 4, 7, 8, etc. Step A1 can be to input local features into several cascaded dynamic pooling network frameworks for dynamic pooling, thereby obtaining an image feature output by each dynamic pooling network framework, and each image feature corresponds to a different feature level.

[0077] Understandably, for a given image feature, the feature flattening in step A2 can be achieved by inputting the image feature into a flattening layer for flattening, thereby obtaining a flattened feature in vector form. The same principle applies to other image features, and this can be easily deduced. Furthermore, after obtaining the flattened features corresponding to different feature levels, the feature concatenation in step A3 can be achieved by concatenating all the flattened vector features to obtain a concatenated feature. This concatenated feature contains both global and local image information corresponding to the training images of the visual scene.

[0078] In some embodiments, the dynamic pooling network framework is used to perform the following steps:

[0079] B1. Obtain input features, wherein the input features are the local features or the image features output by the previous dynamic pooling network framework;

[0080] In this embodiment, if the current dynamic pooling network framework is the first network framework among all cascaded dynamic pooling network frameworks, the input features obtained by the current dynamic pooling network framework can be local features of the visual scene training image; or, if the current dynamic pooling network framework is the second or later network framework among all cascaded dynamic pooling network frameworks, the input features obtained by the current dynamic pooling network framework can be the image features output by the previous dynamic pooling network framework.

[0081] B2. Perform dual-information dynamic pooling on the input features to obtain the image features output by the current dynamic pooling network framework.

[0082] Further, step B2, performing dual-information dynamic pooling on the input features to obtain the image features output by the current dynamic pooling network framework, includes:

[0083] B21. Obtain the target offset map corresponding to the input feature, wherein the target offset map is used to record the pooling sampling offset of each feature pixel of the input feature;

[0084] Further, step B21, obtaining the offset map corresponding to the input feature, includes:

[0085] B211. Perform global self-attention feature extraction on the input features to obtain a global offset map;

[0086] B212. Perform local convolution feature extraction on the input features to obtain a local offset map;

[0087] B213. The global offset map and the local offset map are stitched together and optimized to obtain the target offset map.

[0088] In this embodiment of the application, step B211 may involve capturing global contextual information of the input features using a self-attention module and determining a global offset map based on the global contextual information. The equivalent expression for this global offset map can be:

[0089] O g =Self-Att(X)

[0090] Among them, O g is the global offset map, Self-Att(·) is the self-attention mechanism function, and X is the input feature.

[0091] Understandably, step B212 can capture important local information of the input features through an n×n convolution, specifically through a 3×3 convolution in parallel with the self-attention module. Then, a local offset map is determined based on the local information. The equivalent expression for this local offset map can be:

[0092] O l =Conv 3×3 (X)

[0093] Among them, O l For local offset maps, Conv 3×3 (·) is a 3×3 convolutional layer without bias, and X is the input feature.

[0094] It is worth mentioning that after obtaining the global offset map and the local offset map respectively, step B213 can be to concatenate the global offset map and the local offset map along the channel dimension, and then input the concatenated offset map into a 1×1 convolutional layer for optimization to obtain the target offset map. The target offset map records the pooling sampling offset of each feature pixel of the input feature during pooling. The target offset maps corresponding to different dynamic pooling network frameworks may be the same or different.

[0095] It should be noted that the target offset maps corresponding to different dynamic pooling network frameworks in the embodiments of this application can make each feature pixel of the input feature offset a certain offset based on the pooling sampling offset recorded in the target offset map during the pooling process. This offset enables the visual scene recognition model to adaptively and dynamically adjust the sampling position of the input features at different feature levels, and can better adapt to objects with different shapes and scales. It can adaptively prioritize important object information, retain important spatial clues required for extracting robust features for visual scene recognition, improve the ability of visual scene recognition to distinguish similar visual scenes, and effectively improve the robustness and recognition effect of visual scene recognition.

[0096] B22. Based on the target offset map, the input features are dynamically pooled to obtain the image features, wherein the width of the image features is smaller than the width of the input features, and the height of the image features is smaller than the height of the input features.

[0097] In this embodiment, step B22 can be a pooling operation performed on the input features based on the sampling position determined by the target offset map. The specific pooling operation can be any one of average pooling, max pooling, etc., to obtain a higher-level output feature. The width of the output feature is smaller than the width of the input feature, and the height of the output feature is smaller than the height of the input feature. Specifically, for a certain feature pixel in the output feature, the equivalent expression for its dynamic pooling can be:

[0098]

[0099] Among them, Y (i′,j′) To output the pixel value of the feature at position (i′, j′); Ω (i′,j′) X is the pooling region corresponding to the feature pixel at position (i′,j′); (i,j) Δi, Δj is the pixel value of the feature pixel at position (i,j) in the input features; Δi, Δj is the pooling sampling offset of the feature pixel at position (i,j) in the input features.

[0100] It should be noted that the multi-layer dynamic pooling aggregation in this application embodiment utilizes global and local information to adapt to objects of various scales and shapes in the image. That is, by applying dynamic pooling aggregation multiple times, the visual scene recognition model can capture key spatial information and distinguish key spatial cues even when the objects and scenes change significantly.

[0101] Step 150: Based on the learnable prototype, perform prototype feature selection on the spliced ​​features to obtain globally optimized features;

[0102] In this embodiment of the application, the prototype feature selection can be based on a learnable prototype, focusing on learning important spatial cues of the spliced ​​features while filtering out redundant elements, thereby obtaining globally optimized features that can improve the robustness and discriminativeness of visual scene recognition.

[0103] Further, step 150, selecting prototype features from the spliced ​​features based on the learnable prototype to obtain globally optimized features, includes:

[0104] C1. Based on the learnable prototype, perform dynamic important element learning on the spliced ​​features to obtain a target feature matrix. The target feature matrix is ​​used to characterize the set of similarities between each learnable prototype and each feature element of the spliced ​​features.

[0105] Further, step C1, learning dynamic importance elements of the concatenated features based on the learnable prototype to obtain the target feature matrix, includes:

[0106] C11. Based on the learnable prototype, perform matrix multiplication on the concatenated features to obtain an intermediate feature matrix;

[0107] C12. Perform row normalization on the intermediate feature matrix to obtain the target feature matrix.

[0108] In the embodiments of this application, a learnable prototype can be a vector element in a set of learnable vectors. Each learnable element is used to capture a specific spatial pattern and dynamically learn how to identify important elements in the splicing features.

[0109] Understandably, step C11 could first involve obtaining the transpose of the concatenated features, and then performing matrix multiplication on the transpose of the concatenated features based on each learnable prototype to obtain an intermediate feature matrix. This intermediate feature matrix is ​​used to characterize the similarity between each learnable prototype and each feature space element of the concatenated features. Furthermore, after obtaining the intermediate feature matrix, the SoftMax function can be used to perform row-matrix normalization on each matrix element in the intermediate feature matrix to obtain the target feature matrix.

[0110] C2. Based on the target feature matrix, perform element aggregation on the spliced ​​features to obtain element-aggregated features;

[0111] C3. Perform feature optimization on the element aggregation features to obtain the global optimized features.

[0112] In this embodiment, the element aggregation in step C2 can be based on the obtained target feature matrix, performing matrix multiplication on the concatenated features to obtain the element-aggregated features. These element-aggregated features are the feature representations after aggregating the important feature space elements in the concatenated features. Step C3 can be to input the element-aggregated features into an optimized network structure consisting of fully connected layers, ReLU activation layers, and flattening layers to obtain the globally optimized features after feature optimization.

[0113] Step 160: Update the parameters of the initialized visual scene recognition model according to the global optimization features to obtain the trained visual scene recognition model.

[0114] In this embodiment, the visual scene recognition model first acquires a set of positive and negative samples corresponding to the training images of the visual scene during training. The target loss value is then determined using the global features and global optimization features of the sample images corresponding to these sets. This target loss value is then used to update the model's parameters, thereby obtaining the trained visual scene recognition model. Furthermore, this embodiment does not limit the loss function used to determine the target loss value; for example, it can be a multi-similarity loss (MS-Loss) function, which will not be elaborated upon here.

[0115] The following describes in detail, with reference to the accompanying drawings, a visual scene recognition system based on adaptive dynamic feature aggregation proposed according to an embodiment of this application.

[0116] Reference Figure 3 The visual scene recognition system based on adaptive dynamic feature aggregation proposed in this application includes:

[0117] The first processing unit 101 is used to acquire the visual scene image to be recognized;

[0118] The second processing unit 102 is used to input the visual scene image into a trained visual scene recognition model to perform scene recognition and obtain a visual scene recognition result.

[0119] The trained visual scene recognition model is obtained through the following steps:

[0120] Acquire visual scene training images and extract features from the visual scene training images to obtain local features;

[0121] The local features are aggregated by multi-layer dynamic pooling to obtain spliced ​​features, which are used to characterize the global image information and local image information corresponding to the training image of the visual scene;

[0122] Based on the learnable prototype, prototype feature selection is performed on the spliced ​​features to obtain globally optimized features;

[0123] Based on the global optimization features, the parameters of the initialized visual scene recognition model are updated to obtain the trained visual scene recognition model.

[0124] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0125] Reference Figure 4 This application also provides an electronic device, including:

[0126] At least one processor 201;

[0127] At least one memory 202 is used to store at least one program;

[0128] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the method embodiment described above.

[0129] Similarly, it can be understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0130] This application also provides a computer-readable storage medium storing a program executable by a processor 201, which, when executed by the processor 201, is used to implement the above-described method embodiments.

[0131] Similarly, the content of the above method embodiments is applicable to the present computer-readable storage medium embodiments. The specific functions implemented by the present computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0132] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0133] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0134] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0136] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0137] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0138] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0139] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0140] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A visual scene recognition method based on adaptive dynamic feature aggregation, characterized in that, include: Acquire the visual scene image to be identified; The visual scene image is input into a trained visual scene recognition model for scene recognition, and the visual scene recognition result is obtained. The trained visual scene recognition model is obtained through the following steps: Acquire visual scene training images and extract features from the visual scene training images to obtain local features; The local features are aggregated by multi-layer dynamic pooling to obtain spliced ​​features, which are used to characterize the global image information and local image information corresponding to the training image of the visual scene; Based on the learnable prototype, prototype feature selection is performed on the spliced ​​features to obtain globally optimized features; Based on the global optimization features, the parameters of the initialized visual scene recognition model are updated to obtain the trained visual scene recognition model. The step of performing prototype feature selection on the concatenated features based on the learnable prototype to obtain the globally optimized features includes: Based on the learnable prototype, dynamic importance element learning is performed on the spliced ​​features to obtain a target feature matrix. The target feature matrix is ​​used to characterize the set of similarities between each learnable prototype and each feature element of the spliced ​​features. Based on the target feature matrix, the spliced ​​features are element-aggregated to obtain element-aggregated features; The element aggregation features are optimized to obtain the global optimized features.

2. The method according to claim 1, characterized in that, The visual scene recognition model includes several cascaded dynamic pooling network frameworks. The process of performing multi-layer dynamic pooling aggregation on the local features to obtain concatenated features includes: The local features are input into the plurality of cascaded dynamic pooling network frameworks to obtain a plurality of image features, each of the image features corresponding to one of the dynamic pooling network frameworks; All the image features are flattened to obtain flattened features corresponding to different feature levels; All the flattened features are spliced ​​together to obtain the spliced ​​features.

3. The method according to claim 2, characterized in that, The dynamic pooling network framework is used to perform the following steps: Obtain input features, wherein the input features are the local features or the image features output by the previous dynamic pooling network framework; The input features are subjected to dual-information dynamic pooling to obtain the image features output by the current dynamic pooling network framework.

4. The method according to claim 3, characterized in that, The process of performing dual-information dynamic pooling on the input features to obtain the image features output by the current dynamic pooling network framework includes: Obtain a target offset map corresponding to the input feature, wherein the target offset map is used to record the pooling sampling offset of each feature pixel of the input feature; Based on the target offset map, the input features are dynamically pooled to obtain the image features, wherein the width of the image features is smaller than the width of the input features, and the height of the image features is smaller than the height of the input features.

5. The method according to claim 4, characterized in that, The step of obtaining the offset map corresponding to the input features includes: Global self-attention feature extraction is performed on the input features to obtain a global offset map; Local convolution feature extraction is performed on the input features to obtain a local offset map; The global offset map and the local offset map are stitched together and optimized to obtain the target offset map.

6. The method according to claim 1, characterized in that, The step of dynamically learning important elements of the concatenated features based on the learnable prototype to obtain the target feature matrix includes: Based on the learnable prototype, matrix multiplication is performed on the spliced ​​features to obtain an intermediate feature matrix; The intermediate feature matrix is ​​normalized by row operations to obtain the target feature matrix.

7. A visual scene recognition system based on adaptive dynamic feature aggregation, characterized in that, include: The first processing unit is used to acquire the visual scene image to be recognized; The second processing unit is used to input the visual scene image into the trained visual scene recognition model to perform scene recognition and obtain the visual scene recognition result. The trained visual scene recognition model is obtained through the following steps: Acquire visual scene training images and extract features from the visual scene training images to obtain local features; The local features are aggregated by multi-layer dynamic pooling to obtain spliced ​​features, which are used to characterize the global image information and local image information corresponding to the training image of the visual scene; Based on the learnable prototype, prototype feature selection is performed on the spliced ​​features to obtain globally optimized features; Based on the global optimization features, the parameters of the initialized visual scene recognition model are updated to obtain the trained visual scene recognition model. The step of performing prototype feature selection on the concatenated features based on the learnable prototype to obtain the globally optimized features includes: Based on the learnable prototype, dynamic importance element learning is performed on the spliced ​​features to obtain a target feature matrix. The target feature matrix is ​​used to characterize the set of similarities between each learnable prototype and each feature element of the spliced ​​features. Based on the target feature matrix, the spliced ​​features are element-aggregated to obtain element-aggregated features; The element aggregation features are optimized to obtain the global optimized features.

8. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the method as described in any one of claims 1-6.

Images