A cross-modal retrieval method and system for eliminating modal differences

Abstract

The present invention provides a cross-modal retrieval method and system for eliminating modal differences, including randomly shuffling RGB images and IR images, selecting a training batch as input; introducing intermediate modal data, and utilizing the intermediate modal data in combination with a dynamic sparse normalized pseudo-labeling strategy to enhance model generalization. The present invention proposes a progressive cross-modal target retrieval algorithm for eliminating modal differences, focusing the model's attention on the image itself rather than the ID association between image pairs. Intermediate modal data is then introduced to further prevent IR image information from being ignored during training. The intermediate modal dataset is generated by a non-conditional generative adversarial network, and a dynamic sparse normalized pseudo-labeling strategy is used to generate labels for the intermediate modal dataset, which can be used for cross-modal face retrieval and pedestrian retrieval. This method and system have significant performance improvements in cross-modal target retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of target retrieval, and in particular to a cross-modal retrieval method and system for eliminating modal differences. Background Art

[0002] The task of object retrieval (face or pedestrian) is to associate different images of the same identity captured by different surveillance cameras. At present, object retrieval technology under RGB images has made significant progress. However, with the rapid development of object retrieval systems, a complex and robust system should be able to handle various situations, such as cross-modal object retrieval. In recent years, cross-modal (RGB image-IR image) retrieval has attracted widespread attention. In order to associate images captured by different types of cameras (RGB and IR), the study of typical cross-modal problems has received widespread attention. The main challenge of the cross-modal problem is that the model can tolerate the modal differences between RGB images and IR images, and can learn the feature information of RGB images and IR images in a balanced way.

[0003] Among existing methods, backbone networks pre-trained on ImageNet (ResNet50 or DenseNet121) have been widely used and achieved competitive performance. Using backbone networks pre-trained on ImageNet is also popular in state-of-the-art traditional retrieval methods. However, compared to traditional retrieval tasks, backbone networks pre-trained on ImageNet perform poorly on cross-modal retrieval tasks. Current cross-modal retrieval tasks typically train the network by feeding RGB and IR images of the same identity as image pairs. However, the learned shared features cannot be used to obtain true features that are unbiased between the two modalities. Due to the significant differences in training data between the RGB and IR modalities, the learned shared features will contain more information from the RGB modality. For example, when two images from different modalities need to be classified as belonging to the same person ID, a backbone network pre-trained on ImageNet is likely to learn discriminative features from the familiar RGB modality. As a result, feature information from the unfamiliar IR modality will be overwhelmed by RGB information during training. Summary of the Invention

[0004] In order to solve the technical problems in the existing technology that the operating speed is limited due to the excessive number of model parameters and the intrinsic connection between different pedestrian attributes is not taken into account, which affects the pedestrian attribute recognition performance, the present invention proposes a cross-modal retrieval method and system that eliminates modal differences to solve the above technical problems.

[0005] According to one aspect of the present invention, a cross-modal retrieval method for eliminating modality differences includes:

[0006] S1: Randomly shuffle the RGB image and IR image and select the training batch as input;

[0007] S2: Introduce intermediate modal data and use it in combination with dynamic sparse normalized pseudo-labeling strategy to enhance model generalization.

[0008] In some specific embodiments, step S1 further includes: using a triplet loss function to improve feature distinguishability, which is specifically expressed as follows: Among them, A and B represent data of two different modes, RGB and IR respectively. represents the anchor point, whose image label is Represent positive samples and negative samples respectively, and their image labels are F d (·) represents the backbone network extracting image features, the model outputs d-dimensional features, and ε is a constant parameter.

[0009] In some specific embodiments, the intermediate modality data is generated using a non-conditional generative adversarial network (DCGAN), where half of the training batches of the DCGAN network come from RGB modality images and the other half come from IR modality images.

[0010] In some specific embodiments, a dynamic sparse normalized pseudo-labeling strategy is used to assign labels to the intermediate modality data, and the label distribution is as follows: where β k Represents the possibility of each intermediate modal data being associated with all target categories of different modalities, and N represents a set that is randomly sparsely distributed to 0.

[0011] In some specific embodiments, the set N is obtained by setting a dynamic sparsity coefficient α, and the number of sparsified labels is α×K.

[0012] In some specific embodiments, the features of the last layer of the model are subjected to softmax to predict each target category and obtain the probability that the target belongs to the corresponding category.

[0013] In some specific embodiments, in step S2, a cross entropy loss function is used in combination with a dynamic sparse normalized pseudo-labeling strategy: λ and μ are manually set parameters, λ is set to 0.2, and μ is defined as 2 / (1+K).

[0014] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, any one of the above methods is implemented.

[0015] According to a third aspect of the present invention, a cross-modal retrieval system for eliminating modality differences is proposed, the system comprising:

[0016] Input unit: configured to randomly shuffle RGB images and IR images and select training batches as input;

[0017] Model processing unit: Introduces intermediate modal data and uses the intermediate modal data in combination with a dynamic sparse normalized pseudo-labeling strategy to enhance model generalization.

[0018] In some specific embodiments, a triplet loss function is used in the input unit to improve feature distinguishability. The specific expression is as follows: Among them, A and B represent data of two different modes, RGB and IR respectively. represents the anchor point, whose image label is Represent positive samples and negative samples respectively, and their image labels are F d (·) represents the backbone network extracting image features, the model outputs d-dimensional features, and ε is a constant parameter.

[0019] In some specific embodiments, the intermediate modality data is generated using a non-conditional generative adversarial network (DCGAN), where half of the training batches of the DCGAN network come from RGB modality images and the other half come from IR modality images.

[0020] In some specific embodiments, a dynamic sparse normalized pseudo-labeling strategy is used to assign labels to the intermediate modality data, and the label distribution is as follows: where β k It represents the possibility of each intermediate modal data being associated with all target categories of different modalities. N represents a set that is randomly sparsely distributed to 0. The set N is obtained by setting the dynamic sparsity coefficient α. The number of sparse labels is α×K.

[0021] In some specific embodiments, the model processing unit adopts a cross entropy loss function combined with a dynamic sparse normalized pseudo-labeling strategy: λ and μ are manually set parameters, λ is set to 0.2, and μ is defined as 2 / (1+K).

[0022] This paper proposes a cross-modal retrieval method and system that eliminates modality differences, which can be used for cross-modal face and pedestrian retrieval. First, the model focuses on the image itself, rather than the ID associations between image pairs. Next, intermediate modality data is introduced to further prevent IR image information from being ignored during training. The intermediate modality dataset is generated using an unconditional generative adversarial network, and a dynamic sparse normalized pseudo-labeling strategy is used to generate labels for the intermediate modality dataset. The proposed method significantly improves cross-modal object retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many of the intended advantages of the embodiments will be readily apparent as they become better understood by reference to the following detailed description. Other features, objects, and advantages of the present application will become more apparent upon reading the detailed description of the non-limiting embodiments made with reference to the following drawings:

[0024] Figure 1 is a flowchart of a cross-modal retrieval method for eliminating modality differences according to an embodiment of the present application;

[0025] Figure 2 This is a framework diagram of a cross-modal retrieval algorithm for progressively eliminating modal differences according to a specific embodiment of the present application;

[0026] Figure 3 This is a framework diagram of a cross-modal retrieval system for eliminating modality differences according to an embodiment of the present application;

[0027] Figure 4 It is a structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0029] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] According to a cross-modal retrieval method for eliminating modality differences according to an embodiment of the present application, Figure 1 FIG. 1 shows a flow chart of a cross-modal retrieval method for eliminating modality differences according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0031] S101: Randomly shuffle the RGB and IR images and select a training batch as input. Focus the model on the image itself rather than the ID associations between image pairs. In addition to the cross-entropy loss function, a triplet loss function is also used to improve feature discriminability.

[0032] S102: Introducing intermediate modal data and utilizing it in conjunction with a dynamic sparse normalized pseudo-labeling strategy to enhance model generalization. This introduces intermediate modal data to further prevent IR image information from being ignored during training. The intermediate modal dataset is generated using an unconditional generative adversarial network, and a dynamic sparse normalized pseudo-labeling strategy is used to generate labels for the intermediate modal dataset.

[0033] In a specific embodiment, Figure 2 A cross-modal retrieval algorithm framework diagram for eliminating modality differences according to a specific embodiment of the present application is shown, as shown in FIG. Figure 2 As shown in Figure 1, the algorithm is mainly divided into two stages. The first stage is used to learn image feature information between different modalities. The second stage uses the intermediate state data set to balance the backbone network's learning of different modal information and enhance the generalization of the model. The specific steps of the algorithm include:

[0034] Step S1: Unlike other conventional cross-modal algorithms, this application does not use RGB-IR image pairs as image input. Instead, it randomly shuffles the RGB and IR images and selects a training batch as input. In addition to using the cross-entropy loss function, a triplet loss function is also used to improve the distinguishability of features. Its expression is as follows:

[0035]

[0036] Among them, A and B represent data of different modalities, namely, RGB and IR, respectively. represents the anchor point, whose image label is Is a positive sample (negative sample), its image label is F d (·) represents the backbone network extracting image features, the model outputs d-dimensional features, and ε is a constant parameter.

[0037] Step S2: To further balance the model's learning of data from different modalities, this application introduces intermediate modality data. This intermediate modality data is generated using a non-conditional generative adversarial network (DCGAN). For each DCGAN training batch, half of the training batch is from RGB images and the other half from IR images. This ensures that the generated intermediate modality data balances information from both modalities.

[0038] Step S3: Since DCGAN is a non-conditional generative adversarial network, a dynamic sparse normalized pseudo-label strategy is used to assign labels to the intermediate modal data. The label distribution is as follows:

[0039]

[0040] where β krepresents the probability of each intermediate modal data being associated with all target categories of different modalities, and N represents a set that is randomly sparsified to 0. The set is obtained by setting the dynamic sparsity coefficient α, and the number of sparsified labels is α×K.

[0041] Step S4: The second stage is to use the intermediate modal data combined with the dynamic sparse normalization pseudo-label strategy to enhance the generalization of the model. The last layer of the model is passed through softmax to predict which category each target belongs to and obtain p(X c ), that is, predict the probability that Xc belongs to category C.

[0042] Step S5: The cross entropy loss function in the second stage combined with the dynamic sparse normalization pseudo-labeling strategy becomes:

[0043]

[0044] λ is an artificially set parameter, which is set to 0.2, and μ is also an artificially set parameter, which is defined as 2 / (1+K).

[0045] Existing cross-modal retrieval methods typically use a pair of images with the same identity ID (i.e., a cross-modal image pair with the same identity) and feed them into a two-stream backbone network. The backbone learns shared features across modalities to tolerate the modal differences between RGB and IR images. However, due to the significant differences between the modalities and the fact that the initial parameters loaded into the backbone network are pre-trained on RGB images from ImageNet, the model tends to learn RGB information and ignores IR image information. To ensure that the model balances the considerations of both RGB and IR images, this paper proposes a cross-modal object retrieval algorithm that progressively eliminates modal differences, building on existing network models. This algorithm can be used for cross-modal face and person retrieval. First, the model focuses on the image itself, rather than the ID association between the image pair. Next, intermediate modal data is introduced to further prevent IR image information from being ignored during training. The intermediate modal dataset is generated using an unconditional generative adversarial network, and a dynamic sparse normalized pseudo-labeling strategy is used to generate labels for the intermediate modal dataset. The proposed method achieves significant performance improvements in cross-modal object retrieval.

[0046] Continue to refer Figure 3 , Figure 3 A framework diagram of a cross-modal retrieval system for eliminating modal differences according to an embodiment of the present application is shown. The system specifically includes an input unit 301 and a model processing unit 302. Input unit 301 is configured to randomly shuffle RGB and IR images and select a training batch as input. Model processing unit 302 introduces intermediate modal data and utilizes this data in conjunction with a dynamic sparse normalized pseudo-labeling strategy to enhance model generalization.

[0047] Reference below Figure 4 , which shows a structural diagram of a computer system 400 suitable for implementing an electronic device of an embodiment of the present application. Figure 4 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0048] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the system 400 are also stored in the RAM 403. The CPU 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0049] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including a liquid crystal display (LCD) and speakers; a storage section 408 including a hard disk; and a communication section 409 including a network interface card such as a LAN card or a modem. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 410 as needed, so that computer programs read therefrom can be installed into the storage section 408 as needed.

[0050] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0051] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0052] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0053] The modules described in the embodiments of the present application may be implemented by software or hardware.

[0054] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: includes randomly shuffling RGB images and IR images, selecting a training batch as input; introducing intermediate modal data, and utilizing the intermediate modal data in combination with a dynamic sparse normalized pseudo-labeling strategy to enhance model generalization.

[0055] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A cross-modal retrieval method for eliminating modality differences, characterized in that: include: S1: Randomly shuffle the RGB image and IR image, select the training batch as input, and use the triplet loss function to improve feature distinguishability. The specific expression is as follows: Among them, A and B represent data of two different modes, RGB and IR respectively. represents the anchor point, whose image label is Represent positive samples and negative samples respectively, and their image labels are F d (·) represents the backbone network extracting image features, the model outputs d-dimensional features, and ε is a constant parameter; S2: Introduce intermediate modal data. The intermediate modal data is generated using the unconditional generative adversarial network (DCGAN). Half of the DCGAN network training batch comes from RGB modality images and the other half comes from IR modality images. The intermediate modal data is combined with a dynamic sparse normalized pseudo-labeling strategy to enhance model generalization. The dynamic sparse normalized pseudo-labeling strategy is used to assign labels to the intermediate modal data. The label distribution is as follows: where β k It represents the possibility of each intermediate modal data being associated with all target categories of different modalities. N represents a set that is randomly sparsely distributed to 0. The set N is obtained by setting the dynamic sparsity coefficient α. The number of sparse labels is α×K.

2. The cross-modal retrieval method for eliminating modality differences according to claim 1, characterized in that: The last layer of features of the model is passed through softmax to predict each target category and obtain the probability that the target belongs to the corresponding category.

3. The cross-modal retrieval method for eliminating modality differences according to claim 1, characterized in that: In step S2, a cross entropy loss function is used in combination with a dynamic sparse normalized pseudo-label strategy: λ and μ are manually set parameters, λ is set to 0.2, and μ is defined as 2 / (1+K).

4. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 3 is implemented.

5. A cross-modal retrieval system that eliminates modality differences, characterized in that: The system comprises: Input unit: This unit is configured to randomly shuffle the RGB image and the IR image, select a training batch as input, and use a triplet loss function to improve feature distinguishability. The specific expression is as follows: Among them, A and B represent data of two different modes, RGB and IR respectively. represents the anchor point, whose image label is Represent positive samples and negative samples respectively, and their image labels are F d (·) represents the backbone network extracting image features, the model outputs d-dimensional features, and ε is a constant parameter; Model processing unit: Introducing intermediate modal data, the intermediate modal data is generated using the unconditional generative adversarial network (DCGAN). Half of the DCGAN network training batch comes from RGB modality images and the other half comes from IR modality images. The intermediate modal data is combined with a dynamic sparse normalized pseudo-labeling strategy to enhance model generalization. The dynamic sparse normalized pseudo-labeling strategy is used to assign labels to the intermediate modal data. The label distribution is as follows: where β k Represents the possibility of each intermediate modal data being associated with all target categories of different modalities. N represents a set randomly sparsely set to 0. The set N is obtained by setting the dynamic sparsity coefficient α. The number of sparse labels is α×K. The cross entropy loss function is combined with the dynamic sparse normalized pseudo-label strategy in the model processing unit: λ and μ are manually set parameters, λ is set to 0.2, and μ is defined as 2 / (1+K).

6. The cross-modal retrieval system for eliminating modality differences according to claim 5, characterized in that: The model processing unit adopts the cross entropy loss function combined with the dynamic sparse normalized pseudo labeling strategy: λ and μ are manually set parameters, λ is set to 0.2, and μ is defined as 2 / (1+K).

Images