Visual scene recognition method based on adaptive dynamic feature aggregation

Abstract

The invention discloses a visual scene recognition method based on adaptive dynamic feature aggregation. The method comprises the following steps: acquiring a visual scene image to be recognized; inputting the visual scene image into a trained visual scene recognition model for scene recognition to obtain a visual scene recognition result; wherein the visual scene recognition model is obtained by training through the following steps: acquiring a visual scene training image, and performing feature extraction on the visual scene training image to obtain local features; performing multi-layer dynamic pooling aggregation on the local features to obtain splicing features; performing prototype feature selection on the splicing features according to a learnable prototype to obtain global optimization features; and according to the global optimization features, performing parameter updating on the initialized visual scene recognition model to obtain a trained visual scene recognition model. The visual scene recognition model provided by the method can effectively improve the robustness and recognition effect of visual scene recognition. The invention relates to the technical field of computer vision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual scene recognition method based on adaptive dynamic feature aggregation. Background Art

[0002] The goal of visual scene recognition is to find data images in the database that match the query image, thereby determining the visual scene of the query image.

[0003] At present, the existing technology usually aggregates the local features of the query image, and matches and identifies the global features obtained after feature aggregation with the global features of the data image to determine the visual scene of the query image. This method is easily affected by the shape and scale changes of the key objects in the query image. Specifically, when the shape and scale changes of the key objects are large, the robustness and recognition effect of the visual scene recognition are unsatisfactory.

[0004] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] To this end, an object of an embodiment of the present invention is to provide a visual scene recognition method based on adaptive dynamic feature aggregation, wherein the visual scene recognition model provided by the method can effectively improve the robustness and recognition effect of visual scene recognition.

[0007] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0008] In a first aspect, an embodiment of the present application provides a visual scene recognition method based on adaptive dynamic feature aggregation, comprising:

[0009] Acquire a visual scene image to be recognized;

[0010] Inputting the visual scene image into a trained visual scene recognition model to perform scene recognition and obtain a visual scene recognition result;

[0011] The trained visual scene recognition model is obtained by training through the following steps:

[0012] Acquire a visual scene training image, and perform feature extraction on the visual scene training image to obtain local features;

[0013] Performing multi-layer dynamic pooling aggregation on the local features to obtain splicing features, wherein the splicing features are used to characterize global image information and local image information corresponding to the visual scene training image;

[0014] According to the learnable prototype, prototype feature selection is performed on the splicing feature to obtain a global optimization feature;

[0015] According to the global optimization feature, the parameters of the initialized visual scene recognition model are updated to obtain the trained visual scene recognition model.

[0016] In addition, the method according to the above embodiment of the present application may also have the following additional technical features:

[0017] Furthermore, in one embodiment of the present application, the visual scene recognition model includes a plurality of cascaded dynamic pooling network frameworks, and the local features are subjected to multi-layer dynamic pooling aggregation to obtain splicing features, including:

[0018] Inputting the local features into the plurality of cascaded dynamic pooling network frameworks to obtain a plurality of image features, each of the image features corresponding to one of the dynamic pooling network frameworks;

[0019] Performing feature flattening on all the image features to obtain flattened features corresponding to different feature levels;

[0020] Feature splicing is performed on all the flattened features to obtain the spliced ​​features.

[0021] Further, in one embodiment of the present application, the dynamic pooling network framework is used to perform the following steps:

[0022] Obtaining input features, where the input features are the local features or image features output by a previous dynamic pooling network framework;

[0023] Perform dual-information dynamic pooling on the input features to obtain image features output by the current dynamic pooling network framework.

[0024] Further, in one embodiment of the present application, performing dual-information dynamic pooling on the input features to obtain image features output by the current dynamic pooling network framework includes:

[0025] Obtain a target offset map corresponding to the input feature, where the target offset map is used to record the pooled sampling offset of each feature pixel of the input feature;

[0026] According to the target offset map, the input feature is dynamically pooled to obtain the image feature, where a width of the image feature is smaller than a width of the input feature, and a height of the image feature is smaller than a height of the input feature.

[0027] Further, in one embodiment of the present application, obtaining the offset map corresponding to the input feature includes:

[0028] Performing global self-attention feature extraction on the input features to obtain a global offset map;

[0029] Performing local convolution feature extraction on the input features to obtain a local offset map;

[0030] The global offset map and the local offset map are spliced ​​and optimized to obtain the target offset map.

[0031] Further, in one embodiment of the present application, performing prototype feature selection on the splicing feature according to the learnable prototype to obtain the global optimization feature includes:

[0032] According to the learnable prototype, dynamic important element learning is performed on the splicing feature to obtain a target feature matrix, where the target feature matrix is ​​used to represent a set of similarities between each of the learnable prototypes and each feature element of the splicing feature;

[0033] According to the target feature matrix, performing element aggregation on the splicing features to obtain element aggregation features;

[0034] Feature optimization is performed on the element aggregation features to obtain the global optimization features.

[0035] Further, in the embodiment of the present application, the dynamic important element learning is performed on the splicing feature according to the learnable prototype to obtain a target feature matrix, including:

[0036] According to the learnable prototype, performing a matrix multiplication operation on the concatenated features to obtain an intermediate feature matrix;

[0037] The intermediate feature matrix is ​​row-normalized to obtain the target feature matrix.

[0038] In a second aspect, an embodiment of the present application provides a visual scene recognition system based on adaptive dynamic feature aggregation, including:

[0039] A first processing unit, used to obtain a visual scene image to be recognized;

[0040] A second processing unit is used to input the visual scene image into a trained visual scene recognition model to perform scene recognition and obtain a visual scene recognition result;

[0041] The trained visual scene recognition model is obtained by training through the following steps:

[0042] Acquire a visual scene training image, and perform feature extraction on the visual scene training image to obtain local features;

[0043] Performing multi-layer dynamic pooling aggregation on the local features to obtain splicing features, wherein the splicing features are used to characterize global image information and local image information corresponding to the visual scene training image;

[0044] According to the learnable prototype, prototype feature selection is performed on the splicing feature to obtain a global optimization feature;

[0045] According to the global optimization feature, the parameters of the initialized visual scene recognition model are updated to obtain the trained visual scene recognition model.

[0046] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0047] at least one processor;

[0048] at least one memory for storing at least one program;

[0049] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0050] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the above method when executed by the processor.

[0051] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0052] The embodiment of the present application discloses a visual scene recognition method based on adaptive dynamic feature aggregation, which method obtains a visual scene image to be recognized; inputs the visual scene image into a trained visual scene recognition model for scene recognition to obtain a visual scene recognition result; wherein the trained visual scene recognition model is trained by the following steps: obtaining a visual scene training image, and performing feature extraction on the visual scene training image to obtain local features; performing multi-layer dynamic pooling aggregation on the local features to obtain splicing features, and the splicing features are used to characterize global image information and local image information corresponding to the visual scene training image; performing prototype feature selection on the splicing features according to a learnable prototype to obtain a global optimization feature; and performing parameter update on the initialized visual scene recognition model according to the global optimization feature to obtain the trained visual scene recognition model. This method can capture global scene information and local object information at different feature levels by performing multi-layer dynamic pooling aggregation on the local features of visual scene training images, and dynamically adjust the aggregation area based on the captured global scene information and local object information. This method can adaptively capture key object features of different scales and geometric shapes when the object shape and scale change, which can effectively improve the robustness and recognition effect of visual scene recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the embodiments of the present application or the drawings of the related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A flowchart of a visual scene recognition method based on adaptive dynamic feature aggregation provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of a training process of a visual scene recognition model provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of the structural framework of a visual scene recognition system based on adaptive dynamic feature aggregation provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] At present, the existing technology usually aggregates the local features of the query image, and matches and identifies the global features obtained after feature aggregation with the global features of the data image to determine the visual scene of the query image. This method usually regards all aggregated areas in the query image as equally important, does not distinguish the importance of the aggregated areas, and cannot adaptively prioritize important object information. This results in unsatisfactory robustness and recognition effect of visual scene recognition when the shape and scale of key objects vary greatly.

[0061] In view of this, an embodiment of the present invention provides a visual scene recognition method based on adaptive dynamic feature aggregation, wherein the method can capture global scene information and local object information at different feature levels by performing multi-layer dynamic pooling aggregation on local features of visual scene training images, and dynamically adjust the aggregation area based on the captured global scene information and local object information, so that when the shape and scale of the object change, the method can still capture key object features of different scales and geometric shapes by adaptively shifting and focusing on the most important spatial area, which can effectively improve the robustness and recognition effect of visual scene recognition. In addition, the method is also based on prototype feature selection, which can eliminate redundant information in image features at different levels and highlight key spatial information, thereby further improving the robustness of visual scene recognition.

[0062] Reference Figure 1 and Figure 2 In an embodiment of the present application, a visual scene recognition method based on adaptive dynamic feature aggregation includes:

[0063] Step 110: obtaining a visual scene image to be recognized;

[0064] Step 120: input the visual scene image into a trained visual scene recognition model to perform scene recognition and obtain a visual scene recognition result;

[0065] In an embodiment of the present application, the visual scene image to be identified may be an image acquired by a camera device in an actual environment. After acquiring the visual scene image, the visual scene image may be input into a pre-trained visual scene recognition model for scene recognition, thereby obtaining a visual scene recognition result having object recognition information in a certain visual scene.

[0066] The trained visual scene recognition model is obtained by training through the following steps:

[0067] Step 130: acquiring a visual scene training image, and performing feature extraction on the visual scene training image to obtain local features;

[0068] In an embodiment of the present application, the visual scene training image may be a visual scene image in a visual scene training set publicly available on the Internet, or a historical visual scene image collected in an actual environment. There are many ways to obtain the visual scene training image, which will not be described in detail in this application.

[0069] Furthermore, the feature extraction in step 130 may firstly obtain a visual feature extractor, and then input the visual scene training image into the visual feature extractor for local feature extraction, so as to obtain local features recording local details of the visual scene training image.

[0070] Step 140: performing multi-layer dynamic pooling aggregation on the local features to obtain splicing features, where the splicing features are used to characterize global image information and local image information corresponding to the visual scene training image;

[0071] In an embodiment of the present application, the multi-layer dynamic pooling aggregation in step 140 is used to extract image features of different feature levels corresponding to the visual scene training image. Specifically, for a certain level of dynamic pooling aggregation, it can be through an adaptively dynamically changing aggregation pooling area to abstract the local features corresponding to the visual scene training image to extract image features with higher-level information. The dynamic pooling aggregation of the remaining levels is the same, and then the image features of all different feature levels are aggregated and spliced ​​to obtain the splicing features that represent the global image information and local image information corresponding to the visual scene training image.

[0072] In some embodiments, the visual scene recognition model includes a plurality of cascaded dynamic pooling network frameworks, and the step 140 of performing multi-layer dynamic pooling aggregation on the local features to obtain spliced ​​features includes:

[0073] A1. Inputting the local features into the plurality of cascaded dynamic pooling network frameworks to obtain a plurality of image features, each of the image features corresponding to one of the dynamic pooling network frameworks;

[0074] A2. Flatten all the image features to obtain flattened features corresponding to different feature levels;

[0075] A3. Perform feature splicing on all the flattened features to obtain the spliced ​​features.

[0076] In the embodiment of the present application, the specific number of dynamic pooling network frames can be set according to actual conditions, such as any one of 2, 4, 7, 8, etc. Step A1 can be to input the local features into several cascaded dynamic pooling network frames for dynamic pooling, so as to obtain an image feature output by each dynamic pooling network frame, and each image feature corresponds to a different feature level.

[0077] It can be understood that, for a certain image feature, the feature flattening in step A2 can be inputting the image feature into the flattening layer (Flatten) for flattening operation, so as to obtain the flattened feature in vector form, and the other image features can be simply derived by analogy in the same way. In addition, after obtaining the flattened features corresponding to different feature levels, the feature splicing in step A3 can be cascading splicing (Concat) operation on all the flattened features in vector form, so as to obtain the spliced ​​feature, which contains the global image information and local image information corresponding to the visual scene training image.

[0078] In some embodiments, the dynamic pooling network framework is used to perform the following steps:

[0079] B1. Obtaining input features, where the input features are the local features or image features output by a previous dynamic pooling network framework;

[0080] In an embodiment of the present application, if the current dynamic pooling network framework is the first network framework among all cascaded dynamic pooling network frameworks, the input features obtained by the current dynamic pooling network framework can be local features of the visual scene training image; or, if the current dynamic pooling network framework is the second or later network framework among all cascaded dynamic pooling network frameworks, the input features obtained by the current dynamic pooling network framework can be image features output by the previous dynamic pooling network framework.

[0081] B2. Perform dual-information dynamic pooling on the input features to obtain image features output by the current dynamic pooling network framework.

[0082] Furthermore, the step B2, performing dual-information dynamic pooling on the input features to obtain image features output by the current dynamic pooling network framework, includes:

[0083] B21, obtaining a target offset map corresponding to the input feature, wherein the target offset map is used to record the pooled sampling offset of each feature pixel point of the input feature;

[0084] Further, the step B21, obtaining a shift map corresponding to the input feature, includes:

[0085] B211, performing global self-attention feature extraction on the input features to obtain a global offset map;

[0086] B212, performing local convolution feature extraction on the input features to obtain a local offset map;

[0087] B213. Perform splicing optimization on the global offset map and the local offset map to obtain the target offset map.

[0088] In an embodiment of the present application, step B211 may be to capture the global context information of the input feature through a self-attention module, and determine a global offset map based on the global context information. The equivalent expression of the global offset map may be:

[0089] O g =Self-Att(X)

[0090] Among them, O g is the global offset map, Self-Att(·) is the self-attention mechanism function, and X is the input feature.

[0091] It can be understood that step B212 can capture important local information of the input feature through an n×n convolution, which can be specifically implemented through a 3×3 convolution in parallel with the self-attention module, and then determine the local offset map based on the local information. The equivalent expression of the local offset map can be:

[0092] O l =Conv 3×3 (X)

[0093] Among them, O l is the local offset map, Conv 3×3 (·) is a 3×3 convolutional layer without bias, and X is the input feature.

[0094] It is worth mentioning that after obtaining the global offset map and the local offset map respectively, step B213 can be to splice the global offset map and the local offset map along the channel dimension, and then input the spliced ​​offset map into a 1×1 convolutional layer for optimization, so as to obtain a target offset map, which records the pooling sampling offset of each feature pixel point of the input feature during pooling. The target offset maps corresponding to different dynamic pooling network frameworks may be the same or different.

[0095] It should be noted that the target offset map corresponding to the different dynamic pooling network frameworks in the embodiments of the present application can make each feature pixel of the input feature offset to a certain extent during the pooling process based on the pooling sampling offset recorded in the target offset map. This offset can enable the visual scene recognition model to adaptively and dynamically adjust the sampling positions of input features of different feature levels, and can better adapt to objects with different shapes and scales. It can adaptively prioritize important object information and retain important spatial cues required to extract robust features for visual scene recognition. It can improve the ability of visual scene recognition to distinguish similar visual scenes, and effectively improve the robustness and recognition effect of visual scene recognition.

[0096] B22. Dynamically pool the input features according to the target offset map to obtain the image features, wherein the width of the image features is smaller than the width of the input features, and the height of the image features is smaller than the height of the input features.

[0097] In the embodiment of the present application, step B22 may be to perform a pooling operation on the input feature based on the sampling position determined by the target offset map. The specific pooling operation may be any one of average pooling, maximum pooling, etc., so as to obtain a higher-level output feature, the width of the output feature is smaller than the width of the input feature, and the height of the output feature is smaller than the height of the input feature. Specifically, for a certain feature pixel point in the output feature, the equivalent expression of its dynamic pooling can be:

[0098]

[0099] Among them, Y (i′,j′) is the pixel value of the feature pixel at the position (i′, j′) of the output feature; Ω (i′,j′) is the pooling area corresponding to the feature pixel at position (i′, j′); (i,j) is the pixel value of the feature pixel at position (i, j) in the input feature; (Δi, Δj) is the pooled sampling offset of the feature pixel at position (i, j) of the input feature.

[0100] It should be noted that the multi-layer dynamic pooling aggregation in the embodiment of the present application adapts to objects of various scales and shapes in the image by utilizing global and local information. That is, by applying dynamic pooling aggregation multiple times, the visual scene recognition model can capture key spatial information and distinguish key spatial clues even when objects and scenes change significantly.

[0101] Step 150: Perform prototype feature selection on the splicing feature according to the learnable prototype to obtain a global optimization feature;

[0102] In an embodiment of the present application, the prototype feature selection in the embodiment of the present application can be based on a learnable prototype, focusing on learning important spatial clues of the splicing features, while filtering out redundant elements, thereby obtaining a global optimization feature that can improve the robustness and discriminability of visual scene recognition.

[0103] Furthermore, the step 150, performing prototype feature selection on the splicing feature according to the learnable prototype to obtain the global optimization feature, includes:

[0104] C1. According to the learnable prototype, dynamically learn important elements of the splicing feature to obtain a target feature matrix, wherein the target feature matrix is ​​used to represent a set of similarities between each of the learnable prototypes and each feature element of the splicing feature;

[0105] Furthermore, the step C1, based on the learnable prototype, dynamically learning important elements of the splicing features to obtain a target feature matrix, includes:

[0106] C11. According to the learnable prototype, perform a matrix multiplication operation on the concatenated features to obtain an intermediate feature matrix;

[0107] C12. Perform row normalization on the intermediate feature matrix to obtain the target feature matrix.

[0108] In an embodiment of the present application, the learnable prototype may be a vector element in a set of learnable vectors, each learnable element being used to capture a specific spatial pattern and dynamically learn how to identify important elements in splicing features.

[0109] It can be understood that step C11 can first obtain the transpose of the splicing feature, and then perform a matrix multiplication operation on the transpose of the splicing feature based on each learnable prototype to obtain an intermediate feature matrix, which is used to characterize the similarity between each learnable prototype and each feature space element of the splicing feature. In addition, after obtaining the intermediate feature matrix, each matrix element in the intermediate feature matrix can be row-normalized by the SoftMax function to obtain the target feature matrix.

[0110] C2. performing element aggregation on the splicing features according to the target feature matrix to obtain element aggregation features;

[0111] C3. Optimizing the element aggregation features to obtain the global optimization features.

[0112] In the embodiment of the present application, the element aggregation in step C2 can be based on the obtained target feature matrix, and matrix multiplication is performed on the spliced ​​features to obtain element aggregation features, and the element aggregation features are feature representations after the important feature space elements in the spliced ​​features are aggregated. Step C3 can be to input the element aggregation features into an optimized network structure composed of a fully connected layer, a ReLU activation layer, and a flattening layer to obtain a global optimization feature after feature optimization.

[0113] Step 160: Update the parameters of the initialized visual scene recognition model according to the global optimization feature to obtain the trained visual scene recognition model.

[0114] In an embodiment of the present application, the visual scene recognition model can first obtain a set of positive and negative samples corresponding to the visual scene training image during the training process, determine the target loss value through the global features and global optimization features of the sample images corresponding to the positive and negative sample sets, and then update the parameters of the model through the target loss value to obtain a trained visual scene recognition model. In addition, the embodiment of the present application does not limit the loss function used to determine the target loss value, for example, it can be a multi-similarity loss (Multi-Similarity Loss, MS-Loss) function, which will not be repeated in this application.

[0115] A visual scene recognition system based on adaptive dynamic feature aggregation proposed according to an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0116] Reference Figure 3 , a visual scene recognition system based on adaptive dynamic feature aggregation proposed in an embodiment of the present application includes:

[0117] A first processing unit 101 is used to obtain a visual scene image to be recognized;

[0118] The second processing unit 102 is used to input the visual scene image into a trained visual scene recognition model to perform scene recognition and obtain a visual scene recognition result;

[0119] The trained visual scene recognition model is obtained by training through the following steps:

[0120] Acquire a visual scene training image, and perform feature extraction on the visual scene training image to obtain local features;

[0121] Performing multi-layer dynamic pooling aggregation on the local features to obtain splicing features, wherein the splicing features are used to characterize global image information and local image information corresponding to the visual scene training image;

[0122] According to the learnable prototype, prototype feature selection is performed on the splicing feature to obtain a global optimization feature;

[0123] According to the global optimization feature, the parameters of the initialized visual scene recognition model are updated to obtain the trained visual scene recognition model.

[0124] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0125] Reference Figure 4 , the embodiment of the present application further provides an electronic device, including:

[0126] at least one processor 201;

[0127] At least one memory 202, used to store at least one program;

[0128] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiment.

[0129] Similarly, it can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0130] The embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 201 is stored. The program executable by the processor 201 is used to implement the above-mentioned method embodiment when executed by the processor 201.

[0131] Similarly, the contents of the above method embodiments are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0132] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0133] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0134] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0135] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0136] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0137] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0138] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0139] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0140] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A visual scene recognition method based on adaptive dynamic feature aggregation, characterized in that: include: Acquire a visual scene image to be recognized; Inputting the visual scene image into a trained visual scene recognition model to perform scene recognition and obtain a visual scene recognition result; The trained visual scene recognition model is obtained by training in the following steps: Acquire a visual scene training image, and perform feature extraction on the visual scene training image to obtain local features; Performing multi-layer dynamic pooling aggregation on the local features to obtain splicing features, wherein the splicing features are used to characterize global image information and local image information corresponding to the visual scene training image; According to the learnable prototype, prototype feature selection is performed on the splicing feature to obtain a global optimization feature; According to the global optimization feature, the parameters of the initialized visual scene recognition model are updated to obtain the trained visual scene recognition model.

2. The method according to claim 1, characterized in that The visual scene recognition model includes a plurality of cascaded dynamic pooling network frameworks, and the local features are subjected to multi-layer dynamic pooling aggregation to obtain splicing features, including: Inputting the local features into the plurality of cascaded dynamic pooling network frameworks to obtain a plurality of image features, each of the image features corresponding to one of the dynamic pooling network frameworks; Performing feature flattening on all the image features to obtain flattened features corresponding to different feature levels; Feature splicing is performed on all the flattened features to obtain the spliced ​​features.

3. The method according to claim 2, characterized in that The dynamic pooling network framework is used to perform the following steps: Obtaining input features, where the input features are the local features or image features output by a previous dynamic pooling network framework; Perform dual-information dynamic pooling on the input features to obtain image features output by the current dynamic pooling network framework.

4. The method according to claim 3, characterized in that The step of performing dual-information dynamic pooling on the input features to obtain image features output by the current dynamic pooling network framework includes: Obtain a target offset map corresponding to the input feature, where the target offset map is used to record the pooled sampling offset of each feature pixel of the input feature; According to the target offset map, the input feature is dynamically pooled to obtain the image feature, where a width of the image feature is smaller than a width of the input feature, and a height of the image feature is smaller than a height of the input feature.

5. The method according to claim 4, characterized in that The obtaining of the offset map corresponding to the input feature includes: Performing global self-attention feature extraction on the input features to obtain a global offset map; Performing local convolution feature extraction on the input features to obtain a local offset map; The global offset map and the local offset map are spliced ​​and optimized to obtain the target offset map.

6. The method according to claim 1, characterized in that The step of performing prototype feature selection on the splicing feature according to the learnable prototype to obtain the global optimization feature includes: According to the learnable prototype, dynamic important element learning is performed on the splicing feature to obtain a target feature matrix, where the target feature matrix is ​​used to represent a set of similarities between each of the learnable prototypes and each feature element of the splicing feature; According to the target feature matrix, performing element aggregation on the splicing features to obtain element aggregation features; Feature optimization is performed on the element aggregation features to obtain the global optimization features.

7. The method according to claim 6, characterized in that The step of performing dynamic important element learning on the splicing features according to the learnable prototype to obtain a target feature matrix includes: According to the learnable prototype, performing a matrix multiplication operation on the concatenated features to obtain an intermediate feature matrix; The intermediate feature matrix is ​​row-normalized to obtain the target feature matrix.

8. A visual scene recognition system based on adaptive dynamic feature aggregation, characterized in that: include: A first processing unit, used to obtain a visual scene image to be recognized; A second processing unit is used to input the visual scene image into a trained visual scene recognition model to perform scene recognition and obtain a visual scene recognition result; The trained visual scene recognition model is obtained by training in the following steps: Acquire a visual scene training image, and perform feature extraction on the visual scene training image to obtain local features; Performing multi-layer dynamic pooling aggregation on the local features to obtain splicing features, wherein the splicing features are used to characterize global image information and local image information corresponding to the visual scene training image; According to the learnable prototype, prototype feature selection is performed on the splicing feature to obtain a global optimization feature; According to the global optimization feature, the parameters of the initialized visual scene recognition model are updated to obtain the trained visual scene recognition model.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images