Image feature aggregation method and device, electronic equipment and storage medium

Abstract

The application relates to an image feature aggregation method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining image feature data of a picture to be aggregated; training the image feature data according to a detail module to obtain a first feature set, wherein the first feature set comprises feature subsets obtained by at least twice down-sampling of the image feature data by the detail module; training the image feature data according to a semantic module to obtain a second feature set, wherein the second feature set comprises feature subsets obtained by at least twice up-sampling of the image feature data by the semantic module; and aggregating and merging the feature subsets of the first feature set and the feature subsets of the second feature set based on an aggregation module to obtain target fusion features. The scheme provided by the application can improve the aggregation efficiency of multiple image features.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to methods, apparatus, electronic devices and storage media for image feature aggregation. Background Technology

[0002] In the field of intelligent driving, related technologies require the fusion of multiple image features through aggregation methods to facilitate further image analysis. Currently, image feature aggregation methods mainly include those based on the BiseNetv2 network structure. The original BiseNetv2 network structure consists of two branches: a detail branch and a semantic branch. The detail branch has a small receptive field, resulting in high-resolution extracted features. The semantic branch has a relatively large receptive field, resulting in low-resolution extracted features. However, the original BiseNetv2 network structure only fuses the 1 / 32 resolution features from the last layer of the semantic branch and the 1 / 8 resolution features from the last layer of the detail branch. Since the last layer of the BiseNetv2 network structure has the largest receptive field, BiseNetv2 only fuses features with the largest receptive field, failing to consider the smaller receptive field features in the shallower layers of the network. Therefore, BiseNetv2 performs well for large targets but poorly for small targets.

[0003] Therefore, existing image feature aggregation methods suffer from the problem of using only distance features and failing to achieve good aggregation results for features with different resolutions or sizes. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a method, apparatus, electronic device, and storage medium for aggregating image features, which can improve the aggregation efficiency of various image features.

[0005] The first aspect of this application provides a method for aggregating image features, comprising:

[0006] Obtain the image feature data of the image to be aggregated;

[0007] Based on the training image feature data of the detail module, a first type of feature set is obtained, which includes a subset of features obtained by downsampling the image feature data at least twice by the detail module;

[0008] Based on the training image feature data of the semantic module, a second type of feature set is obtained. The second type of feature set includes a subset of features obtained by the semantic module by upsampling the image feature data at least twice.

[0009] Based on the aggregation module, the feature subsets of the first type of feature set and the feature subsets of the second type of feature set are aggregated and merged to obtain the target fused features.

[0010] Optionally, the detail module includes a three-layer downsampling function, and the semantic module includes a three-layer upsampling function. The downsampling function includes a convolution function and an average pooling function, and at least the three downsampling functions include the same network parameters.

[0011] Optionally, the first feature set includes a first feature subset, a second feature subset, and a third feature subset. The first feature set is obtained by training image feature data based on detail modules, and includes:

[0012] Image feature data is input into the downsampling function preset by the detail module to output the first feature subset;

[0013] Input the first feature subset into the downsampling function preset by the detail module to output the second feature subset;

[0014] The second feature subset is input into the downsampling function preset by the detail module to output the third feature subset.

[0015] Optionally, the semantic module includes at least three layers of upsampling functions.

[0016] Optionally, the second feature set includes a fourth feature subset, a fifth feature subset, and a sixth feature subset. The second feature set is obtained by training image feature data based on the semantic module, and includes:

[0017] Image feature data is input into the upsampling function preset by the semantic module to output the fourth feature subset;

[0018] The fourth feature subset is input into the upsampling function preset by the semantic module to output the fifth feature subset;

[0019] The fifth feature subset is input into the upsampling function preset by the semantic module to output the sixth feature subset.

[0020] Optionally, based on the aggregation module, the feature subsets of the first type of feature set and the feature subsets of the second type of feature set are aggregated and merged to obtain the target fused features, including:

[0021] Obtain each feature subset from the first type of feature set and each feature subset from the second type of feature set;

[0022] The feature subsets in the first type of feature set and the feature subsets in the second type of feature set are aggregated in sequence according to their respective sampling order to obtain a candidate aggregated feature set;

[0023] Each candidate aggregation feature in the candidate aggregation feature set is input into a preset convolutional network to obtain the target fusion feature.

[0024] Optionally, each candidate aggregation feature in the candidate aggregation feature set is input into a preset convolutional network to obtain the target fusion features, including:

[0025] Each candidate aggregated feature in the candidate aggregated feature set is input into a preset convolutional network to obtain predicted features;

[0026] Based on the error between the predicted features and the candidate features, and the loss function of the preset convolutional network, the parameters of the preset convolutional network are adjusted to output the target fused features.

[0027] A second aspect of this application provides an image feature aggregation apparatus, comprising:

[0028] The acquisition module is used to acquire the image feature data of the images to be aggregated;

[0029] The first acquisition unit is used to train image feature data based on the detail module to obtain a first type of feature set, which includes a subset of features obtained by the detail module through multiple downsampling of the image feature data.

[0030] The second acquisition unit trains image feature data based on the semantic module to obtain a second type of feature set, which includes a subset of features obtained by the semantic module through multiple upsampling of image feature data.

[0031] The aggregation unit is used to aggregate and merge the first type of feature set and the second type of feature set based on the aggregation module to obtain the target fused feature.

[0032] A third aspect of this application provides an electronic device, comprising:

[0033] Processor; and

[0034] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0035] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0036] The technical solution provided in this application can include the following beneficial effects: by setting up detail modules and semantic modules, feature subsets of different resolution sizes are extracted from image feature data respectively. Then, according to the order in which the two modules extract features, the feature subsets of different resolution sizes are aggregated and fused, thereby improving the fusion degree between image features of different resolution sizes. The technical solution disclosed in this application fully fuses features of different sizes, enabling the network to obtain richer contextual information.

[0037] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0038] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0039] Figure 1 This is a schematic diagram illustrating the application environment of the image feature aggregation method shown in the embodiments of this application;

[0040] Figure 2 This is a flowchart illustrating the image feature aggregation method shown in an embodiment of this application;

[0041] Figure 3 This application embodiment shows a schematic diagram of the network structure of an image feature aggregation method;

[0042] Figure 4 This is a schematic diagram of the structure of the image feature aggregation device shown in the embodiments of this application;

[0043] Figure 5 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0044] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0045] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0046] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0047] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be subject to the following interpretations.

[0048] The BiseNetv2 network architecture (a dual-segmentation real-time semantic segmentation network) compensates for accuracy loss by fusing low-level detail information and high-level semantic information using a lightweight backbone network. The semantic branch utilizes a deep network composed of deep separable convolutional networks to extract high-level semantic information, thereby improving the model's inference speed. The detail branch uses a shallow convolutional network to preserve low-level detail information. In this application, detail modules represent semantic branches, and semantic modules represent detail branches. This application also improves the functions of the semantic branch and detail branch functions in the BiseNetv2 network architecture.

[0049] A Convolutional Neural Network (CNN) is a feedforward neural network that extracts features from the image layer by layer, starting directly from the pixel features at the bottom layer. It is the most commonly used implementation model for encoders and is responsible for encoding images into vectors.

[0050] Global average pooling is used to sample image data to generate averaged features.

[0051] Upsampling, in general, refers to resampling a digital signal. The resampling rate is compared to the original sampling rate used to obtain the digital signal (e.g., sampled from an analog signal). If the resampling rate is higher than the original rate, it's called upsampling; if it's lower, it's called downsampling. Upsampling is the inverse process of downsampling, also known as upsampling or interpolating. Downsampling, on the other hand, is the decimation of the signal. This application uses bilinear interpolation.

[0052] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0053] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0054] See Figure 1 The diagram shown illustrates the application architecture of the image feature aggregation method in this embodiment, including a server 100 and a terminal 200. The terminal 200 can be any intelligent device such as a smartphone, tablet, portable personal computer, or robot. Various applications can be installed on the terminal 200, and these applications can provide intelligent customer service.

[0055] For example, in this embodiment of the application, the user can input image feature data of the image to be aggregated through the terminal 200. Then, the server 100 can process the input image feature data of the image to be aggregated to obtain the target fusion feature, and return the target fusion feature to the terminal 200. After receiving the target fusion feature, the terminal 200 displays it. The image to be aggregated is generally a vehicle driving image collected during vehicle movement.

[0056] Terminal 200 and server 100 can be connected via the Internet to enable communication between them. Optionally, the Internet described above uses standard communication technologies and / or protocols. The Internet is typically the Internet, but can also be any network, including but not limited to any combination of Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, wired or wireless networks, private networks or virtual private networks. In some embodiments, technologies and / or formats including HyperText Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links.

[0057] In other embodiments, customized and / or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0058] The following analysis examines the solutions for image feature aggregation provided by relevant technologies.

[0059] In the field of intelligent driving, related technologies require the fusion of multiple image features through aggregation methods to facilitate further image analysis. Currently, image feature aggregation methods mainly include those based on the BiseNetv2 network structure. The original BiseNetv2 network structure consists of two branches: a detail branch and a semantic branch. The detail branch has a small receptive field, resulting in high-resolution extracted features. The semantic branch has a relatively large receptive field, resulting in low-resolution extracted features. However, the original BiseNetv2 network structure only fuses the 1 / 32 resolution features from the last layer of the semantic branch and the 1 / 8 resolution features from the last layer of the detail branch. Since the last layer of the BiseNetv2 network structure has the largest receptive field, BiseNetv2 only fuses features with the largest receptive field, failing to consider the smaller receptive field features in the shallower layers of the network. Therefore, BiseNetv2 performs well for large targets but poorly for small targets.

[0060] Therefore, existing image feature aggregation methods suffer from the problem of using only distance features and failing to achieve good aggregation results for features with different resolutions or sizes.

[0061] To address the aforementioned issues, this application provides an image feature aggregation method that can improve the aggregation efficiency of various image features.

[0062] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0063] Figure 2 This is a flowchart illustrating the image feature aggregation method shown in the embodiments of this application.

[0064] See Figure 2 ,include:

[0065] Step S201: Obtain the image feature data of the image to be aggregated;

[0066] Image features mainly include color features, texture features, shape features, and spatial relationship features. Color features are global features that describe the surface properties of objects in an image or image region. Texture features are also global features, describing the surface properties of objects in an image or image region. Shape features have two types of representation: contour features and region features. Contour features mainly target the outer boundaries of objects, while region features relate to the entire shape region. Spatial relationship features refer to the spatial positions or relative directions between multiple targets segmented in an image. These relationships can be categorized into connectivity / adjacency, overlap / intersection, and containment / enclosure relationships. In scenarios driven by massive amounts of image data, traditional image features used for image comparison, such as Scale-invariant feature transform (SIFT), color histogram, and histogram of oriented gradient (HOG), perform poorly when processing unseen or noisy images. Image features extracted by deep learning models based on metric learning, such as Sim CLR (AS imple Framework for Contrastive Learning of Visual Representations) and BYOL (Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning), are also effective.

[0067] In one embodiment, the first feature set is an image sample set including category labels, such as the ImageNet dataset, the COCO dataset, etc., and this application does not limit this. The second feature set is obtained by sampling from the image dataset to obtain an image sample set including triples. The sampled image dataset can be an image dataset including category labels or an image dataset without category labels, and this application does not limit this. For example, this application randomly selects an image sample from the image dataset without category labels as a reference image sample (anchor), and performs image enhancement processing on the reference image sample, such as adding Gaussian noise, salt and pepper noise, cropping, rotating, adding watermarks, tone transformation, vertical flipping, horizontal flipping, adding a black frame or one of the following.

[0068] Step S202: Train the image feature data based on the detail module to obtain the first type of feature set. The first type of feature set includes the feature subset obtained by downsampling the image feature data at least twice by the detail module.

[0069] In step S202, the detail module includes three downsampling functions, each comprising a convolutional function and an average pooling function. At least all three downsampling functions use the same network parameters. The detail module obtains detailed boundary features of the data through the convolutional channels of the convolutional function, and the average pooling function extracts these detailed features to enrich semantic boundary information. Each time downsampling is performed, the convolutional function and the average pooling function process the data simultaneously.

[0070] In one embodiment, the first feature set includes a first feature subset, a second feature subset, and a third feature subset. Obtaining the first feature set by training image feature data based on the detail module includes: inputting image feature data into a preset downsampling function of the detail module to output the first feature subset; inputting the first feature subset into a preset downsampling function of the detail module to output the second feature subset; and inputting the second feature subset into a preset downsampling function of the detail module to output the third feature subset.

[0071] The first feature subset is a 1 / 32 resolution dataset; the second feature subset is a 1 / 16 resolution dataset; and the third feature subset is a 1 / 8 resolution dataset.

[0072] In this embodiment, the time complexity of generating the first feature subset is HxWxC; the time complexity of generating the second feature subset is H / 2xW / 2xC for feature D1; and the time complexity of generating the third feature subset is H / 4xW / 4xC.

[0073] Step S203: Train the image feature data according to the semantic module to obtain the second type of feature set. The second type of feature set includes the feature subset obtained by the semantic module by upsampling the image feature data at least twice.

[0074] Step S203: The semantic module includes three upsampling functions, and the parameters of the upsampling functions are the same.

[0075] In one embodiment, the second feature set includes a fourth feature subset, a fifth feature subset, and a sixth feature subset. The second feature set is obtained by training image feature data using the semantic module, including: inputting image feature data into a preset upsampling function of the semantic module to output the fourth feature subset; inputting the fourth feature subset into a preset upsampling function of the semantic module to output the fifth feature subset; and inputting the fifth feature subset into a preset upsampling function of the semantic module to output the sixth feature subset. Wherein, the fourth feature subset is a 1 / 8 resolution dataset; the fifth feature subset is a 1 / 4 resolution dataset; and the sixth feature subset is a 1 / 2 resolution dataset.

[0076] Step S204: Based on the aggregation module, aggregate and merge the feature subsets of the first type of feature set and the feature subsets of the second type of feature set to obtain the target fused features.

[0077] Specifically, step S104 includes: obtaining each feature subset in the first type of feature set and each feature subset in the second type of feature set; sequentially aggregating each feature subset in the first type of feature set and each feature subset in the second type of feature set according to their respective sampling order to obtain a candidate aggregated feature set; and inputting each candidate aggregated feature in the candidate aggregated feature set into a preset convolutional network to obtain the target fusion feature.

[0078] In one embodiment, the step of inputting each candidate aggregated feature in the candidate aggregated feature set into a preset convolutional network to obtain the target fusion feature includes: inputting each candidate aggregated feature in the candidate aggregated feature set into a preset convolutional network to obtain a predicted feature; and adjusting the parameters of the preset convolutional network according to the error between the predicted feature and the candidate feature and the loss function of the preset convolutional network to output the target fusion feature.

[0079] Combination Figure 3Step S204 is explained as follows: For the detail module, a first feature subset Id is generated through a first downsampling, a second feature subset D1 through a second downsampling, and a third feature subset D2 through a third downsampling. For the semantic module, a fourth feature subset Iu is generated through a first upsampling, a fifth feature subset U1 through a second upsampling, and a sixth feature subset U2 through a third upsampling. The third feature subset D2 and the fourth feature subset Iu are then upsampled to generate a first candidate aggregate; the second feature subset D1, the fifth feature subset U1, and the first candidate aggregate are upsampled to generate a second candidate aggregate; and the first feature subset Id, the sixth feature subset U2, and the second candidate aggregate are upsampled to generate a candidate aggregate feature set O. The subsequent aggregate sets need to be further fused by a convolutional network.

[0080] In this embodiment, Fu represents the upsampling function, and the feature O obtained after fusion in step S024 can be expressed by the mathematical formula:

[0081] O = Fu(Iu + D2) + Fu(U1 + D1) + Fu(U2 + Id)

[0082] We then perform another convolution operation on the obtained feature O to obtain the final feature fusion result. Practice has shown that this network structure can effectively fuse features of different sizes, enabling the network to acquire richer contextual information.

[0083] In one embodiment, the original model includes detail modules and semantic modules, as well as shared network parameters that enable network models with shared network parameters in the first and second branch networks to remain synchronized during learning. These network parameters employ the same upsampling function.

[0084] The technical solution provided in this application can include the following beneficial effects: by setting up detail modules and semantic modules, feature subsets of different resolution sizes are extracted from image feature data respectively. Then, according to the order in which the two modules extract features, the feature subsets of different resolution sizes are aggregated and fused, thereby improving the fusion degree between image features of different resolution sizes. The technical solution disclosed in this application fully fuses features of different sizes, enabling the network to obtain richer contextual information.

[0085] Corresponding to the aforementioned application function implementation method embodiments, this application also provides an image feature aggregation device, electronic device, and corresponding embodiments.

[0086] Figure 4 This application embodiment illustrates an image feature aggregation device, comprising:

[0087] The acquisition module 401 is used to acquire the image feature data of the image to be aggregated;

[0088] The first acquisition unit 402 is used to train image feature data based on the detail module to obtain a first type of feature set, which includes a feature subset obtained by the detail module through multiple downsampling of the image feature data;

[0089] The second acquisition unit 403 trains image feature data based on the semantic module to obtain a second type of feature set, which includes a subset of features obtained by the semantic module through multiple upsampling of image feature data.

[0090] The aggregation unit 404 is used to aggregate and merge the first type of feature set and the second type of feature set based on the aggregation module to obtain the target fused feature.

[0091] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0092] In one embodiment, the detail module includes three downsampling functions, the semantic module includes three upsampling functions, the downsampling functions include convolution functions and average pooling functions, and at least the three downsampling functions include the same network parameters.

[0093] In one embodiment, the first feature set includes a first feature subset, a second feature subset, and a third feature subset. The first feature set is obtained by training image feature data based on detail modules, and includes:

[0094] Image feature data is input into the downsampling function preset by the detail module to output the first feature subset;

[0095] Input the first feature subset into the downsampling function preset by the detail module to output the second feature subset;

[0096] The second feature subset is input into the downsampling function preset by the detail module to output the third feature subset.

[0097] In one embodiment, the semantic module includes at least three layers of upsampling functions.

[0098] In one embodiment, the second feature set includes a fourth feature subset, a fifth feature subset, and a sixth feature subset. The second feature set is obtained by training image feature data based on the semantic module, and includes:

[0099] Image feature data is input into the upsampling function preset by the semantic module to output the fourth feature subset;

[0100] The fourth feature subset is input into the upsampling function preset by the semantic module to output the fifth feature subset;

[0101] The fifth feature subset is input into the upsampling function preset by the semantic module to output the sixth feature subset.

[0102] In one embodiment, based on the aggregation module, a subset of features from the first type of feature set and a subset of features from the second type of feature set are aggregated and merged to obtain the target fused features, including:

[0103] Obtain each feature subset from the first type of feature set and each feature subset from the second type of feature set;

[0104] The feature subsets in the first type of feature set and the feature subsets in the second type of feature set are aggregated in sequence according to their respective sampling order to obtain a candidate aggregated feature set;

[0105] Each candidate aggregation feature in the candidate aggregation feature set is input into a preset convolutional network to obtain the target fusion feature.

[0106] In one embodiment, each candidate aggregation feature in the candidate aggregation feature set is input into a preset convolutional network to obtain the target fusion feature, including:

[0107] Each candidate aggregated feature in the candidate aggregated feature set is input into a preset convolutional network to obtain predicted features;

[0108] Based on the error between the predicted features and the candidate features, and the loss function of the preset convolutional network, the parameters of the preset convolutional network are adjusted to output the target fused features.

[0109] Figure 5 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0110] See Figure 5 The electronic device 500 includes a memory 510 and a processor 520.

[0111] The processor 520 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0112] Memory 510 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 520 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 510 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 510 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0113] The memory 510 stores executable code, which, when processed by the processor 520, can cause the processor 520 to execute part or all of the methods described above.

[0114] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0115] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0116] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for aggregating image features, characterized in that, include: Obtain the image feature data of the image to be aggregated; The image feature data is trained based on the detail module to obtain a first type of feature set, which includes a subset of features obtained by the detail module by downsampling the image feature data at least twice; the first type of feature set includes a first feature subset, a second feature subset, and a third feature subset; The image feature data is trained using the semantic module to obtain a second type of feature set. The second type of feature set includes a subset of features obtained by the semantic module through at least two upsamplings of the image feature data. The second type of feature set includes a fourth feature subset, a fifth feature subset, and a sixth feature subset. Based on the aggregation module, the feature subsets of the first type of feature set and the feature subsets of the second type of feature set are aggregated and merged to obtain the target fused features; This includes: generating a first candidate aggregate by applying an upsampling function to the third and fourth feature subsets; generating a second candidate aggregate by applying an upsampling function to the second, fifth, and first candidate aggregates; generating a candidate aggregate feature set by applying an upsampling function to the first, sixth, and second candidate aggregates; and generating the target fusion feature based on the candidate aggregate feature set.

2. The method according to claim 1, characterized in that, The detail module includes a three-layer downsampling function, the semantic module includes a three-layer upsampling function, the downsampling function includes a convolution function and an average pooling function, and the three downsampling functions include the same network parameters.

3. The method according to claim 2, characterized in that, The process of training the image feature data based on the detail module to obtain a first type of feature set includes: The image feature data is input into the downsampling function preset by the detail module to output the first feature subset; The first feature subset is input into the downsampling function preset by the detail module to output the second feature subset; The second feature subset is input into the downsampling function preset by the detail module to output the third feature subset.

4. The method according to claim 1, characterized in that, The semantic module includes at least three layers of upsampling functions.

5. The method according to claim 4, characterized in that, The second type of feature set is obtained by training the image feature data according to the semantic module, including: The image feature data is input into the upsampling function preset by the semantic module to output the fourth feature subset; The fourth feature subset is input into the upsampling function preset by the semantic module to output the fifth feature subset; The fifth feature subset is input into the upsampling function preset by the semantic module to output the sixth feature subset.

6. The method according to claim 1, characterized in that, The aggregation module aggregates and merges the feature subsets of the first type of feature set and the feature subsets of the second type of feature set to obtain the target fused features, including: Obtain each feature subset from the first type of feature set and each feature subset from the second type of feature set; The feature subsets in the first type of feature set and the feature subsets in the second type of feature set are aggregated sequentially according to their respective sampling order to obtain a candidate aggregated feature set; Each candidate aggregation feature in the candidate aggregation feature set is input into a preset convolutional network to obtain the target fusion feature.

7. The method according to claim 6, characterized in that, The step of inputting each candidate aggregation feature in the candidate aggregation feature set into a preset convolutional network to obtain the target fusion feature includes: Each candidate aggregated feature in the candidate aggregated feature set is input into a preset convolutional network to obtain predicted features; Based on the error between the predicted features and the candidate aggregated features, and the loss function of the preset convolutional network, the parameters of the preset convolutional network are adjusted to output the target fused features.

8. An image feature aggregation device, characterized in that, include: The acquisition module is used to acquire the image feature data of the images to be aggregated; The first acquisition unit is used to train the image feature data based on the detail module to obtain a first type of feature set. The first type of feature set includes a subset of features obtained by the detail module through multiple downsampling of the image feature data. The first type of feature set includes a first feature subset, a second feature subset, and a third feature subset. The second acquisition unit trains the image feature data according to the semantic module to obtain a second type of feature set. The second type of feature set includes a feature subset obtained by the semantic module through multiple upsampling of the image feature data. The second type of feature set includes a fourth feature subset, a fifth feature subset, and a sixth feature subset. An aggregation unit is used to aggregate and merge the first type of feature set and the second type of feature set based on the aggregation module to obtain the target fused feature; This includes: generating a first candidate aggregate by applying an upsampling function to the third and fourth feature subsets; generating a second candidate aggregate by applying an upsampling function to the second, fifth, and first candidate aggregates; generating a candidate aggregate feature set by applying an upsampling function to the first, sixth, and second candidate aggregates; and generating the target fusion feature based on the candidate aggregate feature set.

9. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-7.

Images