Feature fusion method and device, computer device and computer readable storage medium

Abstract

Embodiments of the present application disclose a feature fusion method and device, computer equipment and a computer readable storage medium. The method comprises: obtaining a first global feature and a first local feature of first data; obtaining a second global feature and a second local feature of second data; inputting the first global feature, the first local feature, the second global feature and the second local feature into a trained feature fusion model to obtain a first fusion feature and a second fusion feature, the first fusion feature being obtained by fusing the first global feature and the second local feature, and the second fusion feature being obtained by fusing the second global feature and the first local feature; and performing feature fusion processing on the first fusion feature and the second fusion feature to obtain a third fusion feature. The embodiments of the present application can improve the accuracy of the fusion feature.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a feature fusion method, apparatus, computer device, and computer-readable storage medium. Background Technology

[0002] Pedestrian re-identification technology refers to the technology that uses computer vision to determine whether a specific pedestrian exists in an image or video sequence. Its application scenarios mainly include intelligent security, unmanned smart supermarkets, and human-computer interaction.

[0003] Among them, pedestrian re-identification with clothing template refers to searching the database for an image of the pedestrian wearing the specified clothing template, using a pedestrian image and a clothing template image.

[0004] Currently, for pedestrian images and clothing template images that need to be retrieved, the features of each image can be extracted first. Then, the features of the two images can be fused to obtain the fused features of the pedestrian "wearing" the clothing template. Finally, this fused feature can be used to search the database. The more accurate the fused feature, the higher the retrieval accuracy. Therefore, how to improve the accuracy of the fused feature is a concern for technical personnel. Summary of the Invention

[0005] This invention discloses a feature fusion method, apparatus, computer device, and computer-readable storage medium for improving the accuracy of fused features.

[0006] The first aspect discloses a feature fusion method, which can be applied to computer devices or modules (e.g., chips) within computer devices. The following description uses an application to a computer device as an example. The method may include:

[0007] Obtain the first global feature and the first local feature of the first data, which includes information about the first target;

[0008] Acquire second global features and second local features of second data, which includes information about the second target;

[0009] The first global feature, the first local feature, the second global feature, and the second local feature are input into the trained feature fusion model to obtain the first fusion feature and the second fusion feature. The first fusion feature is obtained by fusing the first global feature and the second local feature, and the second fusion feature is obtained by fusing the second global feature and the first local feature.

[0010] The first fusion feature and the second fusion feature are subjected to feature fusion processing to obtain a third fusion feature, which is used to describe the fusion features of the first target and the second target.

[0011] In this embodiment of the invention, the computer device can first acquire the global and local features of the first and second data. In the scenario of pedestrian re-identification with clothing templates, the first data can be a pedestrian image (i.e., an image including a target), and the second data can be a clothing template image. Then, the computer device can input the first global feature, the first local feature, the second global feature, and the second local feature into a trained feature fusion model to obtain the first fused feature and the second fused feature. The first fused feature is obtained based on the first global feature and the second local feature, and the second fused feature is obtained based on the second global feature and the first local feature. This symmetrical fusion method of global and local features allows for the discovery of more strongly correlated local features during the fusion process between the global features of the first data and the local features of the second data, as well as between the global features of the second data and the local features of the first data. (For example, in the problem of pedestrian re-identification with clothing templates, when fusing global features of the clothing template image and local features of the pedestrian image, the global features of the clothing template image are mainly fused with target-related features of the local features of the pedestrian image. This reduces the influence of target-irrelevant features (i.e., noise, such as patterns on clothing) in the local features of the pedestrian image, thereby improving the purity of the fused features.) This improves the accuracy of the fused features. Simultaneously, the symmetrical structure ensures that the features of the first data (i.e., global and local features) and the features of the second data are treated equally, without information loss, thus improving the performance of feature fusion. Finally, the computer can perform feature fusion processing on the first and second fused features to obtain a third fused feature. Since the third fused feature includes information from both the first and second fused features, its accuracy can be improved. In addition, the third fusion feature can be used for retrieval, which can improve retrieval accuracy.

[0012] As one possible implementation, the first global feature and the first local feature for acquiring the first data include:

[0013] The first global features and first local features of the first data are obtained through a convolutional neural network or a visual transformer (VIT) network.

[0014] As one possible implementation, the feature fusion model includes a first feature fusion unit and a second feature fusion unit. The first global feature, the first local feature, the second global feature, and the second local feature are input into the trained feature fusion model to obtain the first fused feature and the second fused feature, including:

[0015] The first global feature and the second local feature are input into the first feature fusion processor to obtain the first fused feature;

[0016] The second global feature and the first local feature are input into the second feature fusion processor to obtain the second fused feature.

[0017] As one possible implementation, the first feature fusion unit and the second feature fusion unit share parameters.

[0018] In this embodiment of the invention, the first feature fusion unit and the second feature fusion unit can share parameters, thereby reducing the number of parameters in the feature fusion model and improving the inference speed of the feature fusion model.

[0019] As one possible implementation, the first feature fusion unit and the second feature fusion unit are one or more Transformer networks.

[0020] In this embodiment of the invention, the self-attention mechanism in Transformer can help to more accurately discover more correlated local features during the fusion process of global features of the first data and local features of the second data, as well as during the fusion process of global features of the second data and local features of the first data, thereby further improving the accuracy of fused features.

[0021] As one possible implementation, the first fusion feature and the second fusion feature are subjected to feature fusion processing to obtain a third fusion feature, including:

[0022] The first fusion feature and the second fusion feature are concatenated to obtain the third fusion feature.

[0023] As one possible implementation, the method may further include:

[0024] Determine the similarity between the third fusion feature and the fusion features of the data in the database;

[0025] Data with similarity scores greater than a threshold are identified as matching data with the first data and the second data.

[0026] The second aspect discloses a feature fusion method, which can be applied to computer devices or modules (e.g., chips) within computer devices. The following description uses an application to a computer device as an example. The method may include:

[0027] Obtain the first global feature and the first local feature of the third data, wherein the first data includes information about the first target;

[0028] Acquire the second global feature and the second local feature of the fourth data, the second data including information about the second target;

[0029] The first global feature and the second local feature are fused using a first feature fusion processor to obtain the first fused feature.

[0030] The second feature is obtained by fusing the second global feature and the first local feature using a second feature fusion processor.

[0031] The first fused feature is classified using a first classifier to obtain the first predicted label;

[0032] The second fused feature is classified by the second classifier to obtain the second predicted label; the initial feature fusion model includes the first feature fusion unit, the second feature fusion unit, the first classifier, and the second classifier;

[0033] A first loss is determined based on the first predicted label and the first label; the first label is the category label of the third data.

[0034] The second loss is determined based on the second prediction label and the first label;

[0035] The parameters of the initial feature fusion model are optimized based on the first loss and the second loss to obtain the trained feature fusion model.

[0036] In this embodiment of the invention, the computer device can first acquire the global and local features of the third and fourth data. In the scenario of pedestrian re-identification with clothing templates, the third data can be a pedestrian image (i.e., an image including a certain target), and the fourth data can be an image of the clothing template. Then, the computer device can perform feature fusion processing on the first global features and the second local features using a first feature fusion processor to obtain a first fused feature, and can perform feature fusion processing on the second global features and the first local features using a second feature fusion processor to obtain a second fused feature. Next, the computer device can classify the first fused feature using a first classifier to obtain a first predicted label, and can classify the second fused feature using a second classifier to obtain a second predicted label. The initial feature fusion model includes the first feature fusion processor, the second feature fusion processor, the first classifier, and the second classifier. Subsequently, the computer device can determine a first loss based on the first predicted label and the first label, and can determine a second loss based on the second predicted label and the first label. The first label is the category label of the third data (e.g., if the third data is a pedestrian image including a certain target, then the label of the third data is the category label of that target). Finally, the computer equipment can optimize the parameters of the initial feature fusion model based on the first and second losses to obtain the trained feature fusion model. Specifically, under the supervision of the first label, to reduce the loss of the initial feature fusion model, during the optimization of model parameters, the first feature fusion unit is prompted to focus on more correlated local features when fusing global features of the third data and local features of the fourth data, and the second feature fusion unit is prompted to focus on more correlated local features when fusing global features of the fourth data and local features of the third data. (For example, in the problem of pedestrian re-identification with clothing templates, when fusing global features of the clothing template image and local features of the pedestrian image, the global features of the clothing template image are encouraged to be fused primarily with target-related features in the local features of the pedestrian image. This is because if the fused features are unrelated to the target (i.e., noise, such as patterns on clothing in the local features of the pedestrian image), the final loss will be relatively large.) This improves the accuracy of the fused features. Meanwhile, this symmetrical fusion method of global features + local features can ensure that the features of the third data (i.e., global features and local features) and the features of the fourth data are treated equally, without causing information loss, and can improve the performance of feature fusion.

[0037] As one possible implementation, the first feature fusion unit and the second feature fusion unit share parameters.

[0038] In this embodiment of the invention, the first feature fusion unit and the second feature fusion unit can share parameters, thereby reducing the number of parameters in the initial feature fusion model and thus improving the training speed and inference speed of the initial feature fusion model.

[0039] The third aspect discloses a feature fusion apparatus, which can be a computer device or a module (e.g., a chip) within the computer device. The apparatus may include:

[0040] The first acquisition unit is used to acquire the first global feature and the first local feature of the first data, the first data including information about the first target;

[0041] The second acquisition unit is used to acquire the second global feature and the second local feature of the second data, the second data including information about the second target;

[0042] The first processing unit is configured to input the first global feature, the first local feature, the second global feature, and the second local feature into a trained feature fusion model to obtain a first fusion feature and a second fusion feature. The first fusion feature is obtained by fusing the first global feature and the second local feature, and the second fusion feature is obtained by fusing the second global feature and the first local feature.

[0043] The second processing unit is used to perform feature fusion processing on the first fusion feature and the second fusion feature to obtain a third fusion feature, which is used to describe the fusion feature of the first target and the second target.

[0044] As one possible implementation, the first acquisition unit is specifically used to acquire the first global features and the first local features of the first data through a convolutional neural network or a visual transformer (VIT) network.

[0045] As one possible implementation, the feature fusion model includes a first feature fusion unit and a second feature fusion unit, wherein the first processing unit is specifically used for:

[0046] The first global feature and the second local feature are input into the first feature fusion processor to obtain the first fused feature;

[0047] The second global feature and the first local feature are input into the second feature fusion processor to obtain the second fused feature.

[0048] As one possible implementation, the first feature fusion unit and the second feature fusion unit share parameters.

[0049] As one possible implementation, the first feature fusion unit and the second feature fusion unit are one or more Transformer networks.

[0050] In one possible implementation, the second processing unit is specifically used to splice the first fusion feature and the second fusion feature to obtain the third fusion feature.

[0051] As one possible implementation, the device may further include:

[0052] A determining unit is used to determine the similarity between the third fusion feature and the fusion features of the data in the database;

[0053] The determining unit is also used to determine the data with similarity greater than a threshold as data that matches the first data and the second data.

[0054] The fourth aspect discloses a feature fusion apparatus, which can be a computer device or a module (e.g., a chip) within the computer device. The apparatus may include:

[0055] The first acquisition unit is used to acquire the first global feature and the first local feature of the third data, wherein the first data includes information about the first target;

[0056] The second acquisition unit is used to acquire the second global feature and the second local feature of the fourth data, wherein the second data includes information about the second target;

[0057] The first processing unit is configured to perform feature fusion processing on the first global feature and the second local feature through a first feature fusion processor to obtain a first fused feature;

[0058] The second processing unit is used to perform feature fusion processing on the second global feature and the first local feature through the second feature fusion processor to obtain the second fused feature;

[0059] The first classification unit is used to classify the first fused feature using a first classifier to obtain the first predicted label;

[0060] The second classification unit is used to classify the second fused feature through the second classifier to obtain the second predicted label; the initial feature fusion model includes the first feature fusion unit, the second feature fusion unit, the first classifier, and the second classifier;

[0061] The first determining unit is configured to determine a first loss based on the first predicted label and the first label; the first label is the category label of the third data.

[0062] The second determining unit is used to determine the second loss based on the second predicted label and the first label;

[0063] The optimization unit is used to optimize the parameters of the initial feature fusion model based on the first loss and the second loss, so as to obtain the trained feature fusion model.

[0064] As one possible implementation, the first feature fusion unit and the second feature fusion unit share parameters.

[0065] A fifth aspect discloses a computer device, comprising a processor and a memory. The memory stores a computer program, and the processor invokes the computer program. When the processor executes the computer program stored in the memory, it performs the feature fusion method disclosed in the first aspect or any embodiment of the first aspect.

[0066] A sixth aspect discloses a computer device, comprising a processor and a memory. The memory stores a computer program, and the processor invokes the computer program. When the processor executes the computer program stored in the memory, it performs the feature fusion method disclosed in the second aspect or any embodiment of the second aspect.

[0067] The seventh aspect discloses a computer-readable storage medium storing a computer program or computer instructions, which, when executed, implements the feature fusion method disclosed in the above aspects.

[0068] The eighth aspect discloses a chip including a processor for executing a program stored in a memory, which, when executed, causes the chip to perform the methods described above.

[0069] As one possible implementation, the memory is located outside the chip.

[0070] The ninth aspect discloses a computer program product comprising computer program code, which, when executed, causes the aforementioned feature fusion method to be performed. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating a feature fusion method disclosed in an embodiment of the present invention;

[0072] Figure 2 This is a schematic diagram of the network framework of a feature fusion model disclosed in an embodiment of the present invention;

[0073] Figure 3 This is a flowchart illustrating another feature fusion method disclosed in an embodiment of the present invention;

[0074] Figure 4 This is a schematic diagram of the structure of a feature fusion device disclosed in an embodiment of the present invention;

[0075] Figure 5 This is a schematic diagram of another feature fusion device disclosed in an embodiment of the present invention;

[0076] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0077] This invention discloses a feature fusion method, apparatus, computer device, and computer-readable storage medium to improve the accuracy of fused features. The technical solutions in this application will be clearly and completely described below with reference to the accompanying drawings.

[0078] Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The reference to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily indicate the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand, explicitly and implicitly, that the embodiments described herein can be combined with other embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects and are not used to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, it may include a series of steps or units, or optionally, steps or units not listed, or optionally other steps or units inherent to these processes, methods, products, or devices.

[0079] The accompanying drawings show only the portions relevant to this application, not all of them. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts depict operations (or steps) as sequential processes, many of these operations may be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations may be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process may correspond to a method, function, procedure, subroutine, subprogram, etc.

[0080] The terms “component,” “module,” “system,” “unit,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or distributed between two or more computers. Furthermore, these units can be executed from various computer-readable media on which various data structures are stored. Units can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from a second unit interacting with another unit between a local system, a distributed system, and / or a network; for example, the Internet interacting with other systems via signals).

[0081] To better understand the embodiments of the present invention, some terms and related technologies used in the embodiments of the present invention will be described below.

[0082] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0083] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0084] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0085] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, weakly supervised learning, and strongly supervised learning.

[0086] Transfer learning is the process of transferring the parameters of a pre-trained model to an untrained model to improve the training efficiency of the untrained model. Since most data or tasks are correlated, transfer learning allows learned model parameters (or knowledge acquired by the model) to be shared with other untrained models in some way, thereby accelerating and optimizing the model's learning efficiency without having to learn a model from scratch.

[0087] Feature fusion refers to combining multiple features into a single feature through some method. Traditional feature fusion methods mainly transform multiple feature vectors into a single feature vector through simple concatenation or addition, or through operations such as fully connected networks and convolutional neural networks. Furthermore, the fusion of multiple feature vectors can also be achieved using the self-attention structure in Transformers, which can combine multiple feature vectors into a single feature vector.

[0088] Person re-identification, also known as pedestrian re-identification, is a technique that uses computer vision to determine whether a specific pedestrian exists in an image or video sequence. Person re-identification is a sub-problem of image retrieval.

[0089] Currently, pedestrian re-identification technology has made significant progress, but the problem of pedestrians wearing different clothes still hinders the large-scale practical application of pedestrian re-identification systems. The traditional single-image (i.e., a single image containing a pedestrian) problem of pedestrian re-identification with different clothes is particularly challenging because the features unrelated to clothing provided by a single image are relatively weak, and clothing features can cause significant interference. In other words, the extracted features from a single image contain a lot of noise, few useful features, and low feature diversity. To address this issue, a new clothing-template-based pedestrian re-identification configuration has been proposed: pedestrian re-identification with clothing templates. Unlike traditional single-image re-identification, this method uses a pedestrian image and a clothing template image to search a database for images of the pedestrian wearing the specified clothing template. Since clothing constitutes the majority of the pedestrian image, using a clothing template can effectively improve the accuracy and diversity of extracted features compared to features from a single pedestrian image.

[0090] Currently, for pedestrian images and clothing template images to be retrieved, the first step is to extract the global features of each image using a convolutional neural network. These two features are then concatenated and fused using a fully connected network to obtain a fused feature representation of the pedestrian "wearing" the clothing template. This fused feature can then be used for retrieval in a database. Traditional feature fusion methods simply fuse two global features (global features from the pedestrian image and global features from the clothing template image), which results in the loss of important local information and low accuracy of the fused feature, leading to low retrieval precision. Therefore, how to better fuse the features of the pedestrian image and the clothing template image to improve the accuracy of the fused feature is a key concern for researchers.

[0091] To better understand the embodiments of the present invention, the applicable scenarios of the embodiments of the present invention will be described exemplarily below.

[0092] For example, the feature fusion method provided in this embodiment of the invention can be used to improve the accuracy of fused features and improve the retrieval accuracy of pedestrian re-identification with clothing templates.

[0093] In the context of building smart and safe cities, pedestrian re-identification technology with clothing templates can help quickly locate targets (such as finding lost individuals). It uses a query image containing a specific target (i.e., the object) and a clothing template image (i.e., an image containing only one item / set of clothing). It can search a database (including massive amounts of surveillance video data, image data, etc.) to find image data containing the target, and based on this image data, determine the time and location of the target's appearance in the surveillance video, thus determining the target's movement trajectory.

[0094] The pedestrian re-identification process using clothing templates involves two steps: establishing a target database and target querying. Establishing the target database first requires target detection and tracking on massive amounts of video data, extracting image data of each target to create the database. Target querying involves extracting the fusion features of the input clothing template image and the query image containing a specific target. These features are then compared with the fusion features of the image data in the target database (e.g., calculating similarity) to pinpoint the time and location of the target's appearance in the massive video dataset.

[0095] To address the problem of pedestrian re-identification with clothing templates, the feature fusion method provided in this invention, when fusing features from pedestrian images and clothing template images, first acquires global and local features of both the pedestrian and clothing template images. Then, it fuses the global features of the pedestrian image and the local features of the clothing template image to obtain a fused global feature, and vice versa. Finally, it concatenates the two fusion results (i.e., the two fused global features) to obtain the final fused feature. This symmetrical global + local feature fusion method allows for the discovery of more correlated local features during the fusion process, as well as between the global features of the pedestrian image and the local features of the clothing template image, and vice versa. Simultaneously, the symmetrical structure ensures that the features of the pedestrian image and the clothing template image are treated equally, preventing information loss and improving feature fusion performance, thereby effectively enhancing retrieval accuracy.

[0096] It should be understood that the feature fusion method provided in this embodiment of the invention can also be applied to other feature fusion tasks, such as other tasks requiring multimodal feature fusion, including fusion between text features and image features, fusion between audio features and text features, etc., and is not limited to the feature fusion task in the above-described scenario of pedestrian re-identification with clothing templates. Furthermore, the feature fusion method provided in this embodiment of the invention can be specifically applied to image retrieval (such as pedestrian re-identification with clothing templates), image and text retrieval, intelligent security, unmanned smart supermarkets, human-computer interaction, and other fields.

[0097] The feature fusion method provided in this invention can be executed by a computer device, which includes, but is not limited to, terminal devices or servers. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms. The terminal device can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, in-vehicle terminal, smart TV, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0098] Please see Figure 1 , Figure 1 This is a flowchart illustrating a feature fusion method disclosed in an embodiment of the present invention. Figure 1 As shown, the feature fusion method may include the following steps.

[0099] 101. Obtain the first global feature and the first local feature of the first data.

[0100] For problems such as pedestrian re-identification with clothing templates, the input data typically consists of two parts: an image of the clothing template and an image of the pedestrian containing the target. To accurately retrieve the corresponding target (i.e., the object) data from the database, feature fusion processing can be performed on the global and local features of the clothing template image and the pedestrian image to obtain a fused feature (i.e., the feature representation of the pedestrian after "changing" to the clothing template). This fused feature can then be used for retrieval. Therefore, computer equipment can first extract the global and local features of the clothing template image and the pedestrian image.

[0101] Specifically, the computer device can first acquire first data, then preprocess the first data, and then acquire the first global features and first local features of the first data. The first data can be image data, text data, or audio data; this embodiment of the invention does not limit the types of data. The first data may include information about a first target, and the second data may include information about a second target.

[0102] For the problem of pedestrian re-identification with clothing templates, the first data can be a pedestrian image (i.e., an image containing a certain target), and the second data can be a clothing template image. The first target included in the first data can be a pedestrian, and correspondingly, the first data can include pedestrian information; the second target included in the second data can be a clothing template, and correspondingly, the second data can include clothing template information.

[0103] The first and second data can be image data. When both the first and second data are image data, the computer device can preprocess the image data before extracting features. Image preprocessing can include image scaling, brightness enhancement, and deblurring, which can reduce noise interference and improve the quality of feature extraction.

[0104] Computer devices can use convolutional neural networks (CNNs) or vision transformer (VIT) networks to obtain the first global and first local features of the first dataset. The CNN and VIT networks are pre-trained networks that can extract relatively accurate global and local features. Furthermore, the larger the dataset on which the CNN and VIT networks are pre-trained, the more accurate the extracted global and local features, and the better the results.

[0105] It should be understood that computer devices can also use other feature extraction networks to extract global and local features of the first data, such as other Transformer-based networks (e.g., DeiT, transReID, etc.), which are not limited to this embodiment of the invention.

[0106] The input data for the VIT network is one-dimensional sequential data. Therefore, when the first data is image data, the computer device needs to convert the image data into one-dimensional sequential data first. Specifically, the computer device can slice the image into small image patches, and then forcibly convert each small image patch into one-dimensional sequential data. For example, the first data is an image with dimensions H*W*C, where H is the height of the image, W is the width of the image (in pixels), and C is the number of channels of the image, typically including red, green, and blue channels. The length and width of the image segment can be p, so a total of (H*W) / (p*p) small image blocks can be obtained. The size of each image block can be p*p*c (e.g., if the image size is 224*224*3, the length and width of the segment are 32, and 49 32*32*3 image blocks can be obtained). Then, the image blocks of size p*p*c can be flattened into one-dimensional sequence data (e.g., one-dimensional vectors), and then input into the VIT network for processing.

[0107] It should be understood that the first local feature obtained by the computer device can be one or more, and the specific number of local features is related to the structure of the feature extraction network, the size of the image data segmentation, etc.

[0108] It should be noted that the embodiments of the present invention do not limit the format of the above-mentioned image data, and it can be any image format, such as portable network graphics (PNG), bitmap images (BMP), joint photographic experts group (JPEG) (a most commonly used image file format), etc. Furthermore, the above-mentioned multiple images can be color images (RGB) with three color channels: red, green, and blue; the above-mentioned multiple images can also be grayscale images with a luminance channel (Y component). The embodiments of this application do not limit the type of image.

[0109] 102. Obtain the second global feature and the second local feature of the second data.

[0110] The second data can be image data, text data, or audio data; this embodiment of the invention does not limit the types of data. It should be understood that, for the problem of pedestrian re-identification with clothing templates, the first data can be a pedestrian image (i.e., an image containing a specific target), and the second data can be a clothing template image. For the image-text retrieval problem, the first data can be image data, and the second data can be text data. Therefore, the data types of the first and second data can differ in different application scenarios.

[0111] The second global feature and the second local feature of the computer device for acquiring the second data are similar to the first global feature and the first local feature of the computer device for acquiring the first data. Please refer to the relevant description in step 101, which will not be elaborated here.

[0112] It should be noted that the network model used by the computer device to acquire the first global feature and the first local feature of the first data can be a different model from the network model used to acquire the second global feature and the second local feature of the second data, or it can be a different model (i.e., it can share parameters).

[0113] It should be understood that the order of steps 101 and 102 can be interchanged, and this embodiment of the invention is not limited thereto.

[0114] 103. Input the first global feature, the first local feature, the second global feature, and the second local feature into the trained feature fusion model to obtain the first fused feature and the second fused feature.

[0115] After obtaining the first global features and first local features of the first data, and the second global features and second local features of the second data, the computer device can obtain the first fused feature and the second fused feature through a trained feature fusion model. The first fused feature can be obtained by fusing the first global features and the second local features, and the second fused feature can be obtained by fusing the second global features and the first local features.

[0116] Specifically, the trained feature fusion model can include a first feature fusion unit and a second feature fusion unit. The computer device can input a first global feature and a second local feature into the first feature fusion unit to obtain the first fused feature. Alternatively, the computer device can input a second global feature and a first local feature into the second feature fusion unit to obtain the second fused feature. Specifically, the computer device can first concatenate the first global feature and the second local feature (i.e., concatenate the first global feature and the second local feature into a single feature vector), and then input this vector into the first feature fusion unit for feature fusion processing to obtain the first fused feature. The computer device can also first concatenate the second global feature and the first local feature (i.e., concatenate the second global feature and the first local feature into a single feature vector), and then input this vector into the second feature fusion unit for feature fusion processing to obtain the second fused feature. This symmetrical fusion of global and local features (i.e., fusing the global features of the first data with the local features of the second data, and fusing the global features of the second data with the local features of the first data) can effectively uncover the correlation between the local and global features of different data, and can better fuse the features of the first data and the features of the second data (e.g., for pedestrian re-identification with clothing templates, it can better fuse the features of the pedestrian image and the features of the clothing template image). It should be understood that the computer device can also process the first global features and the second local features, as well as the second global features and the first local features, in other ways before inputting them into the first feature fusion processor; this embodiment of the invention is not limited thereto.

[0117] The first and second feature fusion units can be two independent feature fusion units, and do not need to share parameters. Alternatively, they can share parameters, meaning they can be the same feature fusion unit. When the first and second feature fusion units share parameters, the number of parameters in the feature fusion model can be reduced, thereby improving the inference speed. When the first and second feature fusion units do not share parameters, the overall model complexity can be increased, but the quality of feature fusion can be improved.

[0118] The first feature fusion unit and the second feature fusion unit can be one or more Transformer networks. The specific number of Transformer network layers used can be dynamically increased or decreased according to actual conditions (such as hardware conditions). In this embodiment of the invention, the self-attention mechanism in the Transformer can help to more accurately discover more correlated local features during the fusion process of global features of the first data and local features of the second data, as well as during the fusion process of global features of the second data and local features of the first data, thereby improving the accuracy of feature fusion.

[0119] It should be understood that the first feature fusion unit and the second feature fusion unit can also be fully connected networks or other feature fusion networks, and the embodiments of the present invention are not limited thereto.

[0120] 104. Perform feature fusion processing on the first and second fusion features to obtain the third fusion feature.

[0121] After obtaining the first and second fusion features, the computer device can perform feature fusion processing on the first and second fusion features to obtain a third fusion feature. The third fusion feature can be used to describe the fusion features of the first target and the second target. Specifically, for the problem of pedestrian re-identification with clothing templates, the first data can be a pedestrian image (i.e., an image containing a certain target), and the second data can be a clothing template image. The third fusion feature obtained by processing the first and second data can be used to describe the fusion features of the first target (i.e., the pedestrian) and the second target (clothing template). Correspondingly, the third fusion feature can be understood as the feature representation of the first target (i.e., the pedestrian) in the pedestrian image after "changing" to the second target (i.e., the clothing template) in the clothing template image.

[0122] The computer device can combine the first and second fusion features to obtain a third fusion feature. Alternatively, the computer device can add the first and second fusion features or perform a weighted sum to obtain the third fusion feature. For example, if the first fusion feature is [1.2, 2.4, 1.6, 1.9] and the second fusion feature is [1.3, 2.6, 1.3, 1.7], the third fusion feature obtained by combining the first and second fusion features could be [1.2, 2.4, 1.6, 1.9, 1.3, 2.6, 1.3, 1.7].

[0123] It should be noted that the computer device can also acquire the third global feature and the third local feature of the first data. The third global feature and the first global feature can be obtained by the computer device using different feature extraction models. These different models are independent of each other and have a weak correlation; therefore, the first global feature and the third global feature can be different. Similarly, the computer device can also acquire the fourth global feature and the fourth local feature of the second data. The fourth global feature and the second global feature can also be obtained by the computer device using different feature extraction models; therefore, the second global feature and the fourth global feature can be different. Subsequently, the computer device can obtain the fourth fusion feature based on the third global feature and the fourth local feature, and it can also obtain the fifth fusion feature based on the fourth global feature and the third local feature. Then, the computer device can perform feature fusion processing on the first fusion feature, the second fusion feature, the fourth fusion feature, and the fifth fusion feature to obtain the sixth fusion feature. The sixth fusion feature includes information from the first fusion feature, the second fusion feature, the fourth fusion feature, and the fifth fusion feature. Furthermore, since the first and third global features can differ, and the second and fourth global features can also differ, the first, second, fourth, and fifth fusion features can incorporate richer information from the original data (i.e., the first and second data). This makes the sixth fusion feature more accurate, robust, and discriminative. Similarly, computer devices can use more weakly correlated independent models (such as different convolutional neural network models, VIT models, etc.) to obtain multiple global and local features of the first and second data, making the final fusion feature more accurate and discriminative, thereby further improving the model's retrieval performance (e.g., retrieval accuracy).

[0124] In this embodiment of the invention, the number of global and local features of the first and second data extracted by the computer device is not limited; it is only necessary to satisfy the symmetrical fusion of global and local features.

[0125] After obtaining the third fusion feature, the computer device can determine the similarity between the third fusion feature and the fusion features of the data in the database. Then, data with similarity greater than a threshold can be identified as matching data with the first and second data. Similarity calculation can use methods such as street distance, Euclidean distance, Mahalanobis distance, and cosine distance, or multiple different types of distances can be calculated and then weighted for fusion (e.g., similarity can be measured using both Euclidean and cosine distances; the smaller the distance, the greater the similarity). The specific calculation method can be selected according to the actual situation. The threshold value can also be selected based on the actual situation. For example, when using Euclidean distance for similarity measurement, the threshold can be set to 0.9. Data with an Euclidean distance greater than 0.9 from the third fusion feature can then be identified as matching data with the first and second data. In the scenario of pedestrian re-identification with clothing templates, the first data can be a pedestrian image (i.e., an image containing a target, such as a full-body or half-body image of a target), and the second data can be a clothing template image (i.e., an image containing a specific garment). Next, the computer device can first obtain fused features from the pedestrian image and the clothing template image. These fused features can be understood as the feature representation of the target in the pedestrian image after "changing" to the clothing template. Then, based on a similarity threshold, it can determine the data that matches the pedestrian image and the clothing template image (i.e., the images matched in the database). This matched data can be understood as the data filtered by the threshold that is most likely to be the same target included in the pedestrian image, and the clothing worn by the target in the matched data is the clothing corresponding to the clothing template image (i.e., clothing that is the same as or similar to the clothing in the clothing template image).

[0126] It should be understood that computer devices can also sort all data in a database from highest to lowest similarity (the higher the similarity, the higher the ranking) and then present the sorted results.

[0127] It should be noted that the process of obtaining the third fusion feature from the data in the database should be the same as that of obtaining the third fusion feature from the computer device, and the model parameters used should also be the same. For example, in the scenario of pedestrian re-identification with clothing templates, in the process of establishing the target database, it is first necessary to perform target detection and tracking on massive amounts of video data, extract the image data of each target from the massive video data (i.e., pedestrian images including a certain target, which can be used as the first data mentioned above), and then for each pedestrian image, the clothing part in the image can be detected and extracted as the clothing template image (which can be used as the second data mentioned above). After that, the fusion feature can be obtained based on the pedestrian image and the clothing template image (i.e., feature fusion processing of the pedestrian image and the clothing template image is performed through the above processing steps), thereby creating the target database.

[0128] For example, in a scenario involving pedestrian re-identification while changing clothes using clothing templates, the overall network framework can be as follows: Figure 2 As shown in the diagram, the input consists of two parts: a pedestrian image and a clothing template image. The computer device can input the pedestrian image into a first feature extraction module for feature extraction, obtaining global and local features of the pedestrian image. Similarly, the computer device can input the clothing template image into a second feature extraction module for feature extraction, obtaining global and local features of the clothing template image. Both the first and second feature extraction modules can be VIT networks. Then, the computer device can use a dual fusion component to symmetrically exchange and fuse the global and local features of the pedestrian image and the clothing template image. That is, the global features of the pedestrian image and the local features of the clothing template image form a new set, and the global features of the clothing template image and the local features of the pedestrian image form another new set. These two new sets can then be fed into the feature fusion modules (i.e., the first and second feature fusion modules) for deep fusion, yielding fused local and global features (i.e., the fused global features of the pedestrian image and the fused global features of the clothing template image). The first feature fusion module and the second feature fusion module can be two-layer Transformer networks. The first feature fusion module and the second feature fusion module can share parameters or not.

[0129] It should be understood that during the overall network training phase, the global features of the fused pedestrian image and the global features of the clothing template image are respectively processed by a classification network (i.e., a first classifier and a second classifier, whose network structure can be a fully connected layer + softmax layer structure), and are supervised by the labels of the pedestrian images (i.e., the category labels of the pedestrian images). During the network inference phase, the global features of the fused pedestrian image and the global features of the clothing template image can be concatenated into a feature vector for retrieval in the database. The dual fusion part can make full use of the relationship between global and local features. For example, when the global features of the clothing template image and the local features of the pedestrian image are fused, under the supervision of the task label (i.e., the category label of the pedestrian image mentioned above), the global features of the clothing template image are forced to focus on the parts of the local features of the pedestrian image that are only related to the target (i.e., the global features of the clothing template image are forced to fuse the parts of the local features of the pedestrian image that are only related to the target). This can improve the purity of the fused features (e.g., the global features of the fused clothing template image do not contain the clothing-related local features of the original pedestrian image). Therefore, dual fusion can better fuse the features of the pedestrian image and the clothing template image, and can improve the accuracy of the fused features.

[0130] Table 1 shows the experimental results for same-domain cross-clothing retrieval, and Table 2 shows the experimental results for cross-domain cross-clothing retrieval. Same-domain means the training and test sets follow the independent and identically distributed hypothesis; cross-domain means the training and test sets are sampled from different distributions. mAP refers to mean average precision. Top-1 is a ratio. For M images to be retrieved, for each image, if the first retrieval result (i.e., the result with the highest similarity to the image to be retrieved) is correct, then the corresponding image can be retrieved correctly; otherwise, it cannot be retrieved correctly. Finally, the number of correctly retrieved images divided by the total number of images to be retrieved gives the top-1. Top-5 is similar to top-1. For M images to be retrieved, for each image, if the first 5 retrieval results include correct results, then the corresponding image can be retrieved correctly; otherwise, it cannot be retrieved correctly. Top-10 is similar to top-1 and top-5; refer to the above descriptions. Linear fusion refers to fusing the global features of a pedestrian image and the global features of a clothing template image using a fully connected network. Transformer fusion refers to fusing the features of a pedestrian image and the features of a clothing template image using a single Transformer network. Dual fusion, the feature fusion method provided in this invention, employs the aforementioned dual fusion structure. When fusing the features of a pedestrian image and a clothing template image, it can fuse the global features of the pedestrian image and the local features of the clothing template image in one step; it can also fuse the local features of the pedestrian image and the global features of the clothing template image in another step; then, the results of the two fusions can be concatenated to obtain the final fused features.

[0131] Linear fusion 46.0 48.5 63.9 71.7 Transformer fusion 49.6 51.2 65.9 74.0 Dual Fusion 54.2 55.9 70.5 77.9

[0132] Table 1

[0133]

[0134]

[0135] Table 2

[0136] The experimental results in Tables 1 and 2 show that the dual feature fusion method provided by this invention significantly improves retrieval performance; while using fully connected components (i.e., linear fusion) or Transformer fusion results in poorer retrieval performance. Compared to linear fusion, dual fusion improves mAP by approximately 8% and top-1 accuracy by approximately 10%, indicating that the dual fusion method disclosed in this invention effectively improves feature fusion performance and the accuracy of fused features, thereby significantly enhancing retrieval precision.

[0137] Please see Figure 3 , Figure 3 This is a flowchart illustrating another feature fusion method disclosed in an embodiment of the present invention. Figure 3 As shown, the feature fusion method may include the following steps.

[0138] 301. Obtain the first global feature and the first local feature of the third data.

[0139] Step 301 is similar to step 101, and you can refer to the relevant description of step 101, so it will not be repeated here.

[0140] 302. Obtain the second global feature and the second local feature of the fourth data.

[0141] Step 302 is similar to step 102, and you can refer to the relevant description of step 102, so it will not be repeated here.

[0142] 303. The first global feature and the second local feature are fused using the first feature fusion processor to obtain the first fused feature.

[0143] The step of the computer device performing feature fusion processing on the first global feature and the second local feature through the first feature fusion processor to obtain the first fused feature is similar to the relevant steps in step 103. Please refer to the relevant description in step 103 above, and it will not be repeated here.

[0144] 304. The second global feature and the first local feature are fused using a second feature fusion processor to obtain the second fused feature.

[0145] The step of the computer device performing feature fusion processing on the second global feature and the first local feature through the second feature fusion processor to obtain the second fused feature is similar to the relevant steps in step 103. Please refer to the relevant description in step 103 above, and it will not be repeated here.

[0146] The first feature fusion unit and the second feature fusion unit can share parameters.

[0147] 305. Classify the first fused features using the first classifier to obtain the first predicted label.

[0148] The computer device can classify the first fused features using a first classifier to obtain a first predicted label. The first classifier can be a network structure consisting of a fully connected layer and a softmax layer, or other network structures; this embodiment of the invention is not limited thereto.

[0149] 306. The second fused feature is classified using a second classifier to obtain the second predicted label.

[0150] The computer device can classify the second fused features using a second classifier to obtain a second predicted label. The second classifier can be a network structure consisting of a fully connected layer and a softmax layer, or other network structures; this embodiment of the invention is not limited thereto.

[0151] The initial feature fusion model includes a first feature fusion unit, a second feature fusion unit, a first classifier, and a second classifier.

[0152] 307. Determine the first loss based on the first prediction label and the first label.

[0153] The first label can be the category label of the third data. In the scenario of pedestrian re-identification with clothing templates, the third data can be the pedestrian image, and the first label can be the category label of the pedestrian image (i.e., the specific identifier of the target included in the pedestrian image).

[0154] The first loss can be the cross-entropy loss, the triplet loss, the loss obtained by linearly adding or weighting the cross-entropy and triplet losses, or other losses. This embodiment of the invention does not limit the loss.

[0155] 308. Determine the second loss based on the second prediction label and the first label.

[0156] The second loss can be cross-entropy loss, triplet loss, or the loss obtained by linearly or weightedly adding cross-entropy and triplet losses, or other losses. This embodiment of the invention does not limit the loss.

[0157] It should be understood that the order of the above steps 303, 304, 305, 306, 307, and 308 can be flexibly changed, as long as the order of steps 303, 305, and 307 and the order of steps 304, 306, and 308 remain unchanged. For example, steps 303, 305, and 307 can be executed first, followed by steps 304, 306, and 308.

[0158] 309. Optimize the parameters of the initial feature fusion model based on the first loss and the second loss to obtain the trained feature fusion model.

[0159] The computer device can optimize the parameters of the initial feature fusion model based on the first loss and the second loss to obtain a trained feature fusion model. This trained feature fusion model can then be used for feature fusion.

[0160] Understandably, during the iterative training of the model, the computer can stop training when the total loss value is less than the preset model error, thus obtaining a trained feature fusion model. Optionally, the computer can set a preset number of iterations for the initial feature fusion model and record the number of training iterations. When the number of training iterations equals the preset number of iterations, training of the initial feature fusion model can be stopped, and the model with the preset number of iterations is determined as the finally trained feature fusion model. Simultaneously, during model training, the total loss values ​​of multiple sets of different data can be obtained first, and then averaged to obtain an average total loss value. Subsequently, the model parameters can be optimized based on this average total loss value.

[0161] It should be noted that, in this embodiment of the invention, the computer device can also fine-tune the parameters of the feature extraction network using the total loss value, thereby further improving the overall feature fusion performance.

[0162] When the first and second feature fusion machines share parameters, during the backpropagation of loss, the two sets of losses can be averaged, and then the parameters of the feature fusion machine can be optimized using the obtained average loss.

[0163] For example, in a scenario involving pedestrian re-identification while changing clothes using clothing templates, the overall network framework can be as follows: Figure 2 As shown in the diagram, the input consists of two parts: a pedestrian image and a clothing template image. The computer device can obtain fused local and global features (i.e., global features of the fused pedestrian image and global features of the fused clothing template image) from these two images. Specific steps can be found in the description above. Then, the computer device obtains predicted labels (i.e., the first and second predicted labels mentioned above) using a first and a second classifier. Afterward, it calculates the loss (i.e., the first loss and the second loss) based on the category labels of the pedestrian image. Finally, the loss can be backpropagated, and the parameters of the classifier, the feature fusion module, and the feature extraction module are optimized using the stochastic gradient descent algorithm. With the parameters of the first and second feature fusion modules shared, the computer device can average the backpropagated loss and then optimize the parameters of the feature fusion module based on the obtained average loss.

[0164] It should be understood that relevant information (i.e., the same or similar information) and related descriptions in the different embodiments described above can be referenced interchangeably. Meanwhile, the above... Figure 3 The corresponding embodiments can be used as described above. Figure 1 The model training method of the corresponding embodiment can be used to obtain the above. Figure 1 The trained feature fusion model in the corresponding embodiment.

[0165] Please see Figure 4 , Figure 4 This is a schematic diagram of a feature fusion device disclosed in an embodiment of the present invention. The feature fusion device can be a computer device or a module within a computer device. Figure 4 As shown, the device may include:

[0166] The first acquisition unit 401 is used to acquire the first global feature and the first local feature of the first data, the first data including information of the first target;

[0167] The second acquisition unit 402 is used to acquire the second global features and the second local features of the second data, the second data including information about the second target;

[0168] The first processing unit 403 is used to input the first global feature, the first local feature, the second global feature and the second local feature into the trained feature fusion model to obtain the first fusion feature and the second fusion feature. The first fusion feature is obtained by fusing the first global feature and the second local feature, and the second fusion feature is obtained by fusing the second global feature and the first local feature.

[0169] The second processing unit 404 is used to perform feature fusion processing on the first fusion feature and the second fusion feature to obtain a third fusion feature, which is used to describe the fusion feature of the first target and the second target.

[0170] In one embodiment, the first acquisition unit 401 is specifically used to acquire the first global features and the first local features of the first data through a convolutional neural network or a visual transformer (VIT) network.

[0171] In one embodiment, the feature fusion model includes a first feature fusion unit and a second feature fusion unit, and the first processing unit 403 is specifically used for:

[0172] The first global feature and the second local feature are input into the first feature fusion processor to obtain the first fused feature;

[0173] The second global feature and the first local feature are input into the second feature fusion processor to obtain the second fused feature.

[0174] In one embodiment, the first feature fusion unit and the second feature fusion unit share parameters.

[0175] In one embodiment, the first feature fusionist and the second feature fusionist are one or more Transformer networks.

[0176] In one embodiment, the second processing unit 404 is specifically used to splice the first fusion feature and the second fusion feature to obtain the third fusion feature.

[0177] In one embodiment, the device may further include:

[0178] The determining unit 405 is used to determine the similarity between the third fusion feature and the fusion features of the data in the database;

[0179] The determining unit 405 is further configured to determine the data with similarity greater than a threshold as data that matches the first data and the second data.

[0180] For a more detailed description of the first acquisition unit 401, the second acquisition unit 402, the first processing unit 403, the second processing unit 404, and the determination unit 405, please refer directly to the above description. Figure 1 The relevant descriptions in the method embodiments shown are directly obtained and will not be repeated here.

[0181] Please see Figure 5 , Figure 5 This is a schematic diagram of another feature fusion device disclosed in an embodiment of the present invention. This feature fusion device can be a computer device or a module within a computer device. Figure 5 As shown, the device may include:

[0182] The first acquisition unit 501 is used to acquire the first global feature and the first local feature of the third data, wherein the first data includes information about the first target;

[0183] The second acquisition unit 502 is used to acquire the second global feature and the second local feature of the fourth data, wherein the second data includes information about the second target.

[0184] The first processing unit 503 is used to perform feature fusion processing on the first global feature and the second local feature through a first feature fusion processor to obtain a first fused feature;

[0185] The second processing unit 504 is used to perform feature fusion processing on the second global feature and the first local feature through the second feature fusion processor to obtain the second fused feature;

[0186] The first classification unit 505 is used to classify the first fused feature through the first classifier to obtain the first predicted label;

[0187] The second classification unit 506 is used to classify the second fused feature through the second classifier to obtain the second predicted label; the initial feature fusion model includes the first feature fusion unit, the second feature fusion unit, the first classifier, and the second classifier;

[0188] The first determining unit 507 is used to determine a first loss based on the first predicted label and the first label; the first label is the category label of the third data.

[0189] The second determining unit 508 is used to determine a second loss based on the second predicted label and the first label;

[0190] The optimization unit 509 is used to optimize the parameters of the initial feature fusion model based on the first loss and the second loss to obtain the trained feature fusion model.

[0191] In one embodiment, the first feature fusion unit and the second feature fusion unit share parameters.

[0192] For a more detailed description of the first acquisition unit 501, the second acquisition unit 502, the first processing unit 503, the second processing unit 504, the first classification unit 505, the second classification unit 506, the first determination unit 507, the second determination unit 508, and the optimization unit 509, please refer directly to the above description. Figure 3 The relevant descriptions in the method embodiments shown are directly obtained and will not be repeated here.

[0193] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 6 As shown, the computer device 600 may include: at least one processor 601, such as a CPU, at least one memory 605, and at least one communication bus 602. Optionally, the computer device 600 may also include at least one network interface 604 and a user interface 603. The communication bus 602 is used to enable communication between these components. The user interface 603 may include a display screen and a keyboard. The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 605 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 605 may also be at least one storage device located remotely from the aforementioned processor 601. Figure 6As shown, the memory 605, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0194] exist Figure 6 In the computer device 600 shown, the network interface 604 provides network communication functions; the user interface 603 is mainly used to provide an input interface for users.

[0195] In one embodiment, the processor 601 can be used to invoke a device control application stored in the memory 605, which can achieve the following:

[0196] Obtain the first global feature and the first local feature of the first data, which includes information about the first target;

[0197] Acquire second global features and second local features of second data, which includes information about the second target;

[0198] The first global feature, the first local feature, the second global feature, and the second local feature are input into the trained feature fusion model to obtain the first fusion feature and the second fusion feature. The first fusion feature is obtained by fusing the first global feature and the second local feature, and the second fusion feature is obtained by fusing the second global feature and the first local feature.

[0199] The first fusion feature and the second fusion feature are subjected to feature fusion processing to obtain a third fusion feature, which is used to describe the fusion features of the first target and the second target.

[0200] In one embodiment, the processor 601 can be used to invoke a device control application stored in the memory 605, which can achieve the following:

[0201] Obtain the first global feature and the first local feature of the third data, wherein the first data includes information about the first target;

[0202] Acquire the second global feature and the second local feature of the fourth data, the second data including information about the second target;

[0203] The first global feature and the second local feature are fused using a first feature fusion processor to obtain the first fused feature.

[0204] The second feature is obtained by fusing the second global feature and the first local feature using a second feature fusion processor.

[0205] The first fused feature is classified using a first classifier to obtain the first predicted label;

[0206] The second fused feature is classified by the second classifier to obtain the second predicted label; the initial feature fusion model includes the first feature fusion unit, the second feature fusion unit, the first classifier, and the second classifier;

[0207] A first loss is determined based on the first predicted label and the first label; the first label is the category label of the third data.

[0208] The second loss is determined based on the second prediction label and the first label;

[0209] The parameters of the initial feature fusion model are optimized based on the first loss and the second loss to obtain the trained feature fusion model.

[0210] It should be understood that the computer device 600 described in the embodiments of this application can be used to perform the above-described tasks. Figure 1 and Figure 3 The method executed by the computer device in the method embodiment can be referred to the relevant description, and will not be repeated here.

[0211] This invention also discloses a computer-readable storage medium storing instructions thereon, which, when executed, perform the methods described in the above method embodiments.

[0212] This invention also discloses a computer program product including instructions that, when executed, perform the methods described in the above method embodiments.

[0213] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. A feature fusion method for pedestrian re-identification in clothing changing scenarios with clothing templates, characterized in that, include: Obtain the first global feature and the first local feature of the first data, wherein the first data includes information about the first target, the first data is a pedestrian image, and the first target is a pedestrian; Obtain the second global feature and the second local feature of the second data, the second data including information of the second target, the first target being different from the second target, the second data being a clothing template image, and the second target being a clothing template; The first global feature, the first local feature, the second global feature, and the second local feature are input into the trained feature fusion model to obtain the first fusion feature and the second fusion feature. The first fusion feature is obtained by fusing the first global feature and the second local feature, and the second fusion feature is obtained by fusing the second global feature and the first local feature. The first fusion feature and the second fusion feature are subjected to feature fusion processing to obtain a third fusion feature, which is used to describe the fusion features of the first target and the second target; The third fusion feature is used for retrieval in the database.

2. The method according to claim 1, characterized in that, The first global feature and the first local feature of the first data to be acquired include: The first global features and the first local features of the first data are obtained through a convolutional neural network or a visual transformer (VIT) network.

3. The method according to claim 1, characterized in that, The feature fusion model includes a first feature fusion unit and a second feature fusion unit. The step of inputting the first global feature, the first local feature, the second global feature, and the second local feature into the trained feature fusion model to obtain the first fused feature and the second fused feature includes: The first global feature and the second local feature are input into the first feature fusion processor to obtain the first fused feature; The second global feature and the first local feature are input into the second feature fusion processor to obtain the second fused feature.

4. The method according to claim 1, characterized in that, The step of performing feature fusion processing on the first fusion feature and the second fusion feature to obtain the third fusion feature includes: The first fusion feature and the second fusion feature are combined to obtain the third fusion feature.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Determine the similarity between the third fusion feature and the fusion features of the data in the database; Data with similarity values ​​greater than a threshold are identified as data that matches the first data and the second data.

6. A feature fusion method for pedestrian re-identification with clothing templates, characterized in that, include: Acquire the first global feature and the first local feature of the third data, wherein the third data includes information about the first target, the third data is a pedestrian image, and the first target is a pedestrian; Acquire the second global feature and the second local feature of the fourth data, wherein the fourth data includes information about the second target, the first target is different from the second target, the fourth data is a clothing template image, and the second target is a clothing template; The first global feature and the second local feature are fused using a first feature fusion processor to obtain a first fused feature. The second global feature and the first local feature are fused using a second feature fusion processor to obtain the second fused feature. The first fused feature is classified using a first classifier to obtain a first predicted label; The second fused feature is classified by a second classifier to obtain a second predicted label; the initial feature fusion model includes the first feature fusion unit, the second feature fusion unit, the first classifier, and the second classifier; A first loss is determined based on the first predicted label and the first label; the first label is the category label of the third data. The second loss is determined based on the second predicted label and the first label; The parameters of the initial feature fusion model are optimized based on the first loss and the second loss to obtain the trained feature fusion model.

7. The method according to claim 6, characterized in that, The first feature fusion unit and the second feature fusion unit share parameters.

8. A feature fusion device for re-identifying pedestrians changing clothes with clothing templates, characterized in that, include: The first acquisition unit is used to acquire the first global feature and the first local feature of the first data, wherein the first data includes information about the first target, the first data is a pedestrian image, and the first target is a pedestrian. The second acquisition unit is used to acquire the second global feature and the second local feature of the second data. The second data includes information about the second target. The first target is different from the second target. The second data is a clothing template image, and the second target is a clothing template. The first processing unit is used to input the first global feature, the first local feature, the second global feature, and the second local feature into a trained feature fusion model to obtain a first fusion feature and a second fusion feature. The first fusion feature is obtained by fusing the first global feature and the second local feature, and the second fusion feature is obtained by fusing the second global feature and the first local feature. The second processing unit is used to perform feature fusion processing on the first fusion feature and the second fusion feature to obtain a third fusion feature, which is used to describe the fusion feature of the first target and the second target. The third fusion feature is used for retrieval in the database.

9. A computer device, characterized in that, include: Memory and processor; wherein: The memory is used to store computer programs, the computer programs including program instructions; The processor is used to invoke the program instructions to cause the computer device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or computer instructions that, when executed, implement the method as described in any one of claims 1-7.

Images