Scene representation method, device and electronic device based on multi-scale regional feature fusion

Abstract

The present invention provides a scene representation method, device and electronic device based on multi-scale regional feature fusion, in which multi-scale region division and feature mixing effectively capture the local salient information of the original scene image and improve the network's adaptability to complex scenes; the multi-head attention mechanism significantly improves the modeling ability of contextual relationships between regions, enabling the network to focus on salient areas related to scene recognition, that is, it can capture local salient areas and multi-scale features, thereby improving the adaptability and recognition performance of the scene recognition network in complex environments.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and artificial intelligence, and in particular to a scene representation method, device and electronic equipment based on multi-scale regional feature fusion. Background Art

[0002] Scene recognition is the core technology of many computer vision tasks. In complex real-world environments, the appearance of a scene may be affected by changes in lighting, viewing angle, and weather, which places high demands on the recognition ability of the model.

[0003] The traditional scene recognition process is generally as follows: extract the feature map corresponding to the original image through the backbone network, where each pixel in the feature map is a local feature, and then fuse all the local features in the feature map (i.e., global feature fusion) to obtain a scene descriptor for characterizing the scene corresponding to the original image. The above process ignores the saliency information in the local area, that is, the ability to capture the saliency information of the local area is weak, resulting in poor scene recognition accuracy when the obtained scene descriptor is used for subsequent scene recognition. For example, in the original image, there is only a building in the upper right corner, and the rest of the positions are the sky and the road. After the global feature fusion of the corresponding feature map, most of the features in the obtained scene descriptor are meaningless, and its ability to characterize the scene corresponding to the original image is poor.

[0004] The above-mentioned traditional scene representation methods have the following shortcomings: (1) they lack the ability to capture fine-grained local salient areas (such as the area of ​​the building in the upper right corner mentioned above); (2) they lack a multi-scale feature fusion mechanism and cannot effectively handle the complexity and diversity of the scene; (3) the global feature fusion method is prone to lose regional context information, affecting recognition accuracy.

[0005] In summary, traditional scene representation methods fail to capture local salient areas and multi-scale features, resulting in poor adaptability and recognition performance of scene recognition networks in complex environments. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a scene characterization method, device and electronic device based on multi-scale region feature fusion to alleviate the technical problem that traditional scene characterization methods fail to capture local salient areas and multi-scale features.

[0007] In a first aspect, an embodiment of the present invention provides a scene characterization method based on multi-scale regional feature fusion, comprising:

[0008] Using a pre-trained backbone network to extract features from an original scene image to obtain a feature map, and dividing the feature map into a plurality of sub-regions according to a preset division scale to obtain a plurality of sub-region feature maps;

[0009] Using a feature mixing module corresponding to each of the sub-region feature maps to extract saliency information of each of the sub-region feature maps, and performing dimension reduction aggregation on the saliency information of each of the sub-region feature maps to obtain a region-level feature descriptor corresponding to each of the sub-region feature maps;

[0010] A multi-head attention mechanism is used to dynamically weight and reorganize the region-level feature descriptors to obtain dynamically weighted and reorganized region-level feature descriptors corresponding to the sub-region feature maps;

[0011] The dynamically weighted reorganized regional-level feature descriptors are optimized and dimensionally reduced to obtain low-dimensional scene feature representation vectors.

[0012] Furthermore, the saliency information of each sub-region feature map is subjected to dimension reduction aggregation, including:

[0013] Using a dimensionality reduction linear projection layer to perform dimensionality reduction processing on the saliency information of each sub-region feature map to obtain the saliency information of each sub-region feature map after dimensionality reduction;

[0014] The aggregated linear projection layer is used to aggregate the saliency information after dimension reduction to obtain the region-level feature descriptors corresponding to the sub-region feature maps.

[0015] Furthermore, the dynamically weighted reorganized regional feature descriptors are optimized and dimensionally reduced, including:

[0016] Using a residual multi-layer perceptron to optimize each of the dynamically weighted reorganized region-level feature descriptors, to obtain an optimized region-level feature descriptor corresponding to each of the sub-region feature maps;

[0017] Performing vector flattening and normalization processing on each of the optimized region-level feature descriptors to obtain a global feature descriptor corresponding to the feature map;

[0018] The dimension reduction technology is used to reduce the dimension of the global feature descriptor to obtain the low-dimensional scene feature representation vector.

[0019] Furthermore, the region-level feature descriptor carries the spatial position information of its corresponding sub-region feature map.

[0020] Furthermore, the method further comprises:

[0021] Calculating the similarity between the low-dimensional scene feature representation vector and a preset low-dimensional scene feature representation vector in a scene recognition relationship, wherein the scene recognition relationship is a relationship between the preset low-dimensional scene feature representation vector and a target scene;

[0022] A target preset low-dimensional scene feature representation vector corresponding to the low-dimensional scene feature representation vector is determined according to the similarity, and then a target scene corresponding to the target preset low-dimensional scene feature representation vector is used as the scene corresponding to the original scene image.

[0023] Furthermore, the preset division scales include: a horizontal equal division scale, a vertical equal division scale, a quadrant division scale and a global division scale.

[0024] In a second aspect, an embodiment of the present invention provides a scene representation device based on multi-scale regional feature fusion, comprising:

[0025] A feature extraction and division unit, used to extract features from the original scene image using a pre-trained backbone network to obtain a feature map, and divide the feature map into multiple sub-regions according to a preset division scale to obtain multiple sub-region feature maps;

[0026] An extraction and dimensionality reduction aggregation unit, used to extract the saliency information of each sub-region feature map by using a feature mixing module corresponding to each sub-region feature map, and perform dimensionality reduction aggregation on the saliency information of each sub-region feature map to obtain a region-level feature descriptor corresponding to each sub-region feature map;

[0027] A dynamic weighted reorganization unit, used to dynamically weight the region-level feature descriptors using a multi-head attention mechanism to obtain dynamically weighted reorganized region-level feature descriptors corresponding to each sub-region feature map;

[0028] The optimization and dimensionality reduction unit is used to optimize and reduce the dimension of the region-level feature descriptor after the dynamic weighted reorganization to obtain a low-dimensional scene feature representation vector.

[0029] Furthermore, the extraction and dimension reduction aggregation unit is also used for:

[0030] Using a dimensionality reduction linear projection layer to perform dimensionality reduction processing on the saliency information of each sub-region feature map to obtain the saliency information of each sub-region feature map after dimensionality reduction;

[0031] The aggregated linear projection layer is used to aggregate the saliency information after dimension reduction to obtain the region-level feature descriptors corresponding to the sub-region feature maps.

[0032] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0033] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to execute any method described in the first aspect above.

[0034] In an embodiment of the present invention, a scene representation method based on multi-scale regional feature fusion is provided, including: using a pre-trained backbone network to extract features from an original scene image to obtain a feature map, and dividing the feature map into multiple sub-regions according to a preset division scale to obtain multiple sub-region feature maps; using a feature mixing module corresponding to each sub-region feature map to extract the saliency information of each sub-region feature map, and performing dimensionality reduction aggregation on the saliency information of each sub-region feature map to obtain a region-level feature descriptor corresponding to each sub-region feature map; using a multi-head attention mechanism to dynamically weighted reorganize the region-level feature descriptors to obtain dynamically weighted reorganized region-level feature descriptors corresponding to each sub-region feature map; optimizing and reducing the dimensionality of the dynamically weighted reorganized region-level feature descriptors to obtain a low-dimensional scene feature representation vector. From the above description, it can be seen that in the scene representation method based on multi-scale regional feature fusion of the present invention, multi-scale region division and feature mixing effectively capture the local salient information of the original scene image and improve the network's adaptability to complex scenes; the multi-head attention mechanism significantly improves the modeling ability of contextual relationships between regions, enabling the network to focus on salient areas related to scene recognition, that is, it can capture local salient areas and multi-scale features, thereby improving the adaptability and recognition performance of the scene recognition network in complex environments, and alleviating the technical problem of insufficient capture of local salient areas and multi-scale features by traditional scene representation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flowchart of a scene characterization method based on multi-scale regional feature fusion provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of obtaining a region-level feature descriptor provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of obtaining a low-dimensional scene feature representation vector from a region-level feature descriptor provided in an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a scene characterization device based on multi-scale regional feature fusion provided by an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Traditional scene representation methods fail to capture local salient areas and multi-scale features.

[0043] Based on this, in the scene representation method based on multi-scale regional feature fusion of the present invention, multi-scale region division and feature mixing effectively capture the local salient information of the original scene image and improve the network's adaptability to complex scenes; the multi-head attention mechanism significantly improves the modeling ability of contextual relationships between regions, enabling the network to focus on salient areas related to scene recognition, that is, it can capture local salient areas and multi-scale features, thereby improving the adaptability and recognition performance of the scene recognition network in complex environments.

[0044] To facilitate understanding of this embodiment, a scene representation method based on multi-scale regional feature fusion disclosed in an embodiment of the present invention is first introduced in detail.

[0045] Embodiment 1:

[0046] According to an embodiment of the present invention, an embodiment of a scene characterization method based on multi-scale regional feature fusion is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0047] Figure 1 is a flow chart of a scene characterization method based on multi-scale regional feature fusion according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0048] Step S102, using a pre-trained backbone network to extract features from the original scene image to obtain a feature map, and dividing the feature map into multiple sub-regions according to a preset division scale to obtain multiple sub-region feature maps;

[0049] Specifically, the above-mentioned preset division scales include: horizontal division scale, vertical division scale, quadrant division scale and global division scale, such as Figure 2 As shown, the obtained multiple sub-region feature maps include: upper sub-region feature map, lower sub-region feature map, left sub-region feature map, right sub-region feature map, upper left region feature map, upper right region feature map, lower left region feature map, lower right region feature map and the entire feature map (to obtain global context).

[0050] Step S104, using a feature mixing module corresponding to each sub-region feature map to extract saliency information of each sub-region feature map, and performing dimension reduction aggregation on the saliency information of each sub-region feature map to obtain a region-level feature descriptor corresponding to each sub-region feature map;

[0051] Specifically, each sub-region feature map is processed by an independent feature mixing module to obtain the saliency information of the corresponding sub-region feature map. Then, the saliency information of each sub-region feature map is aggregated by dimensionality reduction to obtain the region-level feature descriptor corresponding to each sub-region feature map, such as Figure 2 As shown in Figure 1, 9 region-level feature descriptors are obtained. The so-called salient information is the information that improves the performance of the current task network.

[0052] Step S106, using a multi-head attention mechanism to dynamically weighted reorganize the region-level feature descriptors to obtain dynamically weighted reorganized region-level feature descriptors corresponding to each sub-region feature map;

[0053] Specifically, Figure 3 As shown in FIG. 5 , the above-mentioned dynamic weighted reorganization process can capture the contextual relationship between regions. By means of dynamic weighting, the network can focus on the salient areas in the scene, thereby improving the distinguishability and robustness of the features.

[0054] Step S108, optimizing and reducing the dimension of the dynamically weighted reorganized region-level feature descriptors to obtain a low-dimensional scene feature representation vector.

[0055] From the above description, it can be seen that the above method of the present invention is a process of generating regional features (i.e., regional-level feature descriptors) from local features, and then generating global features (i.e., low-dimensional scene feature representation vectors) from regional features.

[0056] In an embodiment of the present invention, a scene representation method based on multi-scale regional feature fusion is provided, including: using a pre-trained backbone network to extract features from an original scene image to obtain a feature map, and dividing the feature map into multiple sub-regions according to a preset division scale to obtain multiple sub-region feature maps; using a feature mixing module corresponding to each sub-region feature map to extract the saliency information of each sub-region feature map, and performing dimensionality reduction aggregation on the saliency information of each sub-region feature map to obtain a region-level feature descriptor corresponding to each sub-region feature map; using a multi-head attention mechanism to dynamically weighted reorganize the region-level feature descriptors to obtain dynamically weighted reorganized region-level feature descriptors corresponding to each sub-region feature map; optimizing and reducing the dimensionality of the dynamically weighted reorganized region-level feature descriptors to obtain a low-dimensional scene feature representation vector. From the above description, it can be seen that in the scene representation method based on multi-scale regional feature fusion of the present invention, multi-scale region division and feature mixing effectively capture the local salient information of the original scene image and improve the network's adaptability to complex scenes; the multi-head attention mechanism significantly improves the modeling ability of contextual relationships between regions, enabling the network to focus on salient areas related to scene recognition, that is, it can capture local salient areas and multi-scale features, thereby improving the adaptability and recognition performance of the scene recognition network in complex environments, and alleviating the technical problem of insufficient capture of local salient areas and multi-scale features by traditional scene representation methods.

[0057] The above content briefly introduces the scene representation method based on multi-scale regional feature fusion of the present invention. The specific contents involved are described in detail below.

[0058] In an optional embodiment of the present invention, the saliency information of each sub-region feature map is subjected to dimension reduction aggregation, which specifically includes the following steps:

[0059] (1) Using a dimensionality reduction linear projection layer to reduce the dimensionality of the saliency information of each sub-region feature map, and obtain the saliency information of each sub-region feature map after dimensionality reduction;

[0060] (2) An aggregated linear projection layer is used to aggregate the saliency information after dimensionality reduction to obtain the region-level feature descriptor corresponding to each sub-region feature map.

[0061] In an optional embodiment of the present invention, the region-level feature descriptors after dynamic weighted reorganization are optimized and reduced in dimension, specifically comprising the following steps:

[0062] (1) Using a residual multi-layer perceptron to optimize each dynamically weighted reorganized region-level feature descriptor, and obtain an optimized region-level feature descriptor corresponding to each sub-region feature map;

[0063] (2) Flatten and normalize the optimized regional feature descriptors to obtain the global feature descriptor corresponding to the feature map;

[0064] (3) The dimensionality reduction technology is used to reduce the dimension of the global feature descriptor to obtain a low-dimensional scene feature representation vector for subsequent scene recognition tasks.

[0065] In an optional embodiment of the present invention, the region-level feature descriptor carries the spatial position information of its corresponding sub-region feature map.

[0066] Specifically, as shown in the above content, the above-mentioned spatial position information can be an upper sub-region feature map, a lower sub-region feature map, a left sub-region feature map, a right sub-region feature map, an upper left region feature map, an upper right region feature map, a lower left region feature map, a lower right region feature map and the entire feature map.

[0067] In an optional embodiment of the present invention, the method further comprises the following steps:

[0068] (1) calculating the similarity between the low-dimensional scene feature representation vector and a preset low-dimensional scene feature representation vector in a scene recognition relationship, wherein the scene recognition relationship is a relationship between the preset low-dimensional scene feature representation vector and the target scene;

[0069] (2) Determine a target preset low-dimensional scene feature representation vector corresponding to the low-dimensional scene feature representation vector based on the similarity, and then use the target scene corresponding to the target preset low-dimensional scene feature representation vector as the scene corresponding to the original scene image.

[0070] The method of the present invention is tested on public scene recognition benchmark datasets (such as Tokyo24 / 7, Nordland, Pitts30k, etc.), and the scene recognition performance of the present invention is significantly better than the existing methods. The experimental results show that:

[0071] In scenes with complex lighting and viewing angle changes, the accuracy of the present invention is improved by more than 10%; the dimension of the low-dimensional scene feature representation vector is significantly reduced (2048 dimensions), and the computing efficiency is improved by 30%.

[0072] The method of this law achieves the following technical effects:

[0073] 1. Fine-grained feature capture:

[0074] The multi-scale region division and processing method effectively captures the local saliency information of the image and improves the network's adaptability to complex scenes.

[0075] 2. Enhance contextual understanding:

[0076] The multi-head attention mechanism significantly improves the ability to model contextual relationships between regions, enabling the network to focus on salient regions related to scene recognition;

[0077] 3. Improved recognition performance:

[0078] In multiple scene recognition tasks, the present invention significantly outperforms traditional global feature aggregation methods, especially in recognition tasks in diverse and complex environments.

[0079] Embodiment 2:

[0080] An embodiment of the present invention also provides a scene characterization device based on multi-scale regional feature fusion. The scene characterization device based on multi-scale regional feature fusion is mainly used to execute the scene characterization method based on multi-scale regional feature fusion provided in Embodiment 1 of the present invention. The scene characterization device based on multi-scale regional feature fusion provided in the embodiment of the present invention is specifically introduced below.

[0081] Figure 4 is a schematic diagram of a scene representation device based on multi-scale regional feature fusion according to an embodiment of the present invention, such as Figure 4 As shown, the device mainly includes: a feature extraction and division unit 10, an extraction and dimension reduction aggregation unit 20, a dynamic weighted recombination unit 30, and an optimization dimension reduction unit 40, wherein:

[0082] A feature extraction and division unit, used to extract features from the original scene image using a pre-trained backbone network to obtain a feature map, and divide the feature map into multiple sub-regions according to a preset division scale to obtain multiple sub-region feature maps;

[0083] An extraction and dimensionality reduction aggregation unit, used to extract the saliency information of each sub-region feature map by using a feature mixing module corresponding to each sub-region feature map, and perform dimensionality reduction aggregation on the saliency information of each sub-region feature map to obtain a region-level feature descriptor corresponding to each sub-region feature map;

[0084] A dynamic weighted reorganization unit is used to dynamically weight the regional feature descriptors using a multi-head attention mechanism to obtain the dynamically weighted reorganized regional feature descriptors corresponding to the feature maps of each sub-region;

[0085] The optimized dimension reduction unit is used to optimize and reduce the dimension of the region-level feature descriptors after dynamic weighted reorganization to obtain a low-dimensional scene feature representation vector.

[0086] In an embodiment of the present invention, a scene representation device based on multi-scale regional feature fusion is provided, comprising: using a pre-trained backbone network to extract features from an original scene image to obtain a feature map, and dividing the feature map into multiple sub-regions according to a preset division scale to obtain multiple sub-region feature maps; using a feature mixing module corresponding to each sub-region feature map to extract the saliency information of each sub-region feature map, and performing dimensionality reduction aggregation on the saliency information of each sub-region feature map to obtain a region-level feature descriptor corresponding to each sub-region feature map; using a multi-head attention mechanism to dynamically weighted reorganize the region-level feature descriptors to obtain dynamically weighted reorganized region-level feature descriptors corresponding to each sub-region feature map; optimizing and reducing the dimensionality of the dynamically weighted reorganized region-level feature descriptors to obtain a low-dimensional scene feature representation vector. From the above description, it can be seen that in the scene representation device based on multi-scale regional feature fusion of the present invention, multi-scale region division and feature mixing effectively capture the local salient information of the original scene image and improve the network's adaptability to complex scenes; the multi-head attention mechanism significantly improves the modeling ability of contextual relationships between regions, enabling the network to focus on salient areas related to scene recognition, that is, it can capture local salient areas and multi-scale features, thereby improving the adaptability and recognition performance of the scene recognition network in complex environments, and alleviating the technical problem of insufficient capture of local salient areas and multi-scale features by traditional scene representation methods.

[0087] Optionally, the extraction and dimensionality reduction aggregation unit is also used to: use a dimensionality reduction linear projection layer to reduce the dimensionality of the saliency information of each sub-region feature map to obtain the saliency information of each sub-region feature map after dimensionality reduction; use an aggregation linear projection layer to aggregate the saliency information after dimensionality reduction to obtain a region-level feature descriptor corresponding to each sub-region feature map.

[0088] Optionally, the optimization and dimensionality reduction unit is also used to: use a residual multi-layer perceptron to optimize each dynamically weighted reorganized region-level feature descriptor to obtain an optimized region-level feature descriptor corresponding to each sub-region feature map; flatten and normalize each optimized region-level feature descriptor to obtain a global feature descriptor corresponding to the feature map; use dimensionality reduction technology to reduce the dimension of the global feature descriptor to obtain a low-dimensional scene feature representation vector.

[0089] Optionally, the region-level feature descriptor carries spatial location information of its corresponding sub-region feature map.

[0090] Optionally, the device is also used to: calculate the similarity between the low-dimensional scene feature representation vector and a preset low-dimensional scene feature representation vector in a scene recognition relationship, wherein the scene recognition relationship is the relationship between the preset low-dimensional scene feature representation vector and the target scene; determine the target preset low-dimensional scene feature representation vector corresponding to the low-dimensional scene feature representation vector based on the similarity, and then use the target scene corresponding to the target preset low-dimensional scene feature representation vector as the scene corresponding to the original scene image.

[0091] Optionally, the preset division scales include: a horizontal equal division scale, a vertical equal division scale, a quadrant division scale and a global division scale.

[0092] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0093] like Figure 5 As shown, an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the scene characterization method based on multi-scale regional feature fusion as described above.

[0094] Specifically, the above-mentioned memory 602 and processor 601 can be general-purpose memories and processors, which are not specifically limited here. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned scene characterization method based on multi-scale regional feature fusion.

[0095] The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 601. The above processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and completes the steps of the above method in combination with its hardware.

[0096] Corresponding to the above-mentioned scene characterization method based on multi-scale regional feature fusion, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above-mentioned scene characterization method based on multi-scale regional feature fusion.

[0097] The scene characterization device based on multi-scale regional feature fusion provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, for the parts not mentioned in the device embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0098] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0099] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0102] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the scene characterization method based on multi-scale regional feature fusion described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0103] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0104] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A scene representation method based on multi-scale regional feature fusion, characterized in that: include: Using a pre-trained backbone network to extract features from an original scene image to obtain a feature map, and dividing the feature map into a plurality of sub-regions according to a preset division scale to obtain a plurality of sub-region feature maps; Using a feature mixing module corresponding to each of the sub-region feature maps to extract saliency information of each of the sub-region feature maps, and performing dimension reduction aggregation on the saliency information of each of the sub-region feature maps to obtain a region-level feature descriptor corresponding to each of the sub-region feature maps; A multi-head attention mechanism is used to dynamically weight and reorganize the region-level feature descriptors to obtain dynamically weighted and reorganized region-level feature descriptors corresponding to the sub-region feature maps; The dynamically weighted reorganized regional-level feature descriptors are optimized and dimensionally reduced to obtain low-dimensional scene feature representation vectors.

2. The method according to claim 1, characterized in that The saliency information of each sub-region feature map is aggregated by dimensionality reduction, including: Using a dimensionality reduction linear projection layer to perform dimensionality reduction processing on the saliency information of each sub-region feature map to obtain the saliency information of each sub-region feature map after dimensionality reduction; The aggregated linear projection layer is used to aggregate the saliency information after dimension reduction to obtain the region-level feature descriptors corresponding to the sub-region feature maps.

3. The method according to claim 1, characterized in that The dynamically weighted reorganized regional feature descriptors are optimized and dimensionally reduced, including: Using a residual multi-layer perceptron to optimize each of the dynamically weighted reorganized region-level feature descriptors, to obtain an optimized region-level feature descriptor corresponding to each of the sub-region feature maps; Performing vector flattening and normalization processing on each of the optimized region-level feature descriptors to obtain a global feature descriptor corresponding to the feature map; The dimension reduction technology is used to reduce the dimension of the global feature descriptor to obtain the low-dimensional scene feature representation vector.

4. The method according to claim 1, characterized in that: The region-level feature descriptor carries the spatial position information of its corresponding sub-region feature map.

5. The method according to claim 1, characterized in that The method further comprises: Calculating the similarity between the low-dimensional scene feature representation vector and a preset low-dimensional scene feature representation vector in a scene recognition relationship, wherein the scene recognition relationship is a relationship between the preset low-dimensional scene feature representation vector and a target scene; A target preset low-dimensional scene feature representation vector corresponding to the low-dimensional scene feature representation vector is determined according to the similarity, and then a target scene corresponding to the target preset low-dimensional scene feature representation vector is used as the scene corresponding to the original scene image.

6. The method according to claim 1, characterized in that The preset division scales include: a horizontal equal division scale, a vertical equal division scale, a quadrant division scale and a global division scale.

7. A scene representation device based on multi-scale regional feature fusion, characterized in that: include: A feature extraction and division unit, used to extract features from the original scene image using a pre-trained backbone network to obtain a feature map, and divide the feature map into multiple sub-regions according to a preset division scale to obtain multiple sub-region feature maps; An extraction and dimensionality reduction aggregation unit, used to extract the saliency information of each sub-region feature map by using a feature mixing module corresponding to each sub-region feature map, and perform dimensionality reduction aggregation on the saliency information of each sub-region feature map to obtain a region-level feature descriptor corresponding to each sub-region feature map; A dynamic weighted reorganization unit, used to dynamically weight the region-level feature descriptors using a multi-head attention mechanism to obtain dynamically weighted reorganized region-level feature descriptors corresponding to each sub-region feature map; The optimization and dimensionality reduction unit is used to optimize and reduce the dimension of the region-level feature descriptor after the dynamic weighted reorganization to obtain a low-dimensional scene feature representation vector.

8. The device according to claim 7, characterized in that The extraction and dimension reduction aggregation unit is also used for: Using a dimensionality reduction linear projection layer to perform dimensionality reduction processing on the saliency information of each sub-region feature map to obtain the saliency information of each sub-region feature map after dimensionality reduction; The aggregated linear projection layer is used to aggregate the saliency information after dimension reduction to obtain the region-level feature descriptors corresponding to the sub-region feature maps.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to execute the method according to any one of claims 1 to 6.

Images