Global and local feature fusion pseudo label refining method based on attention mechanism

Abstract

The invention relates to the technical field of image retrieval, and particularly provides a global and local feature fusion pseudo tag refining method based on an attention mechanism. The method comprises the steps of obtaining a preprocessed training data set; inputting the preprocessed training data set into a convolutional neural network containing an attention mechanism to obtain global features, local features and prediction vectors corresponding to the local features; extracting clusters according to the obtained global features, and distributing pseudo labels; calculating a dynamic weight coefficient according to the global features and the local features; refining a local feature pseudo tag by using a dynamic weight coefficient, and training the network by using local feature loss to obtain a locally trained network; according to the method, the global feature pseudo labels are refined by using the integrated weights and the prediction vectors of the local features, the network after local training is trained by using the global feature loss to refine the pseudo labels and obtain the refined pseudo labels, and the method improves the recognition accuracy, reduces the calculation cost and ensures the performance of the network.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and in particular to a pseudo-label refining method based on global and local feature fusion of an attention mechanism. Background Art

[0002] With the rapid development of artificial intelligence technology, the task of person re-identification in image retrieval technology has attracted much attention. By giving a single picture, using the pedestrian re-identification system for retrieval, it can quickly and accurately identify the person. The image library is usually composed of photos taken by multiple cameras.

[0003] Traditional person re-identification mainly relies on supervised learning methods, which require a large amount of labeled data for training. However, obtaining large-scale labeled datasets is not only time-consuming and laborious, but also in practical applications, the cross-domain problem greatly limits the generalization ability of the model. Therefore, person re-identification technology based on unsupervised learning came into being. Unsupervised learning can effectively train models by exploring the intrinsic structure, feature distribution and potential correlation of pedestrian identities of data in the absence of manual annotations. This technology not only reduces the dependence on labeled data, but also improves the robustness and generalization ability in cross-domain scenarios, which greatly promotes the development of the field of person re-identification.

[0004] In most existing unsupervised training methods, the predictions of auxiliary networks are usually used as refined labels of the target network to address the inherent noise problem in pseudo-labels generated by clustering, and the label noise is reduced through model integration in a peer teaching manner. However, the use of auxiliary networks not only requires high computational costs, but also the labels optimized by these methods only consider global features and ignore fine-grained clues, resulting in insufficient performance. Summary of the invention

[0005] In view of this, the present invention provides a pseudo-label refining method based on global and local feature fusion of attention mechanism to improve recognition accuracy, reduce computational cost and ensure network performance.

[0006] In a first aspect, the present invention provides a pseudo-label refining method based on global and local feature fusion of an attention mechanism, the method comprising: Step 1: Obtain pedestrian image datasets from different cameras, perform preprocessing, and obtain the preprocessed training dataset; Step 2: Input the preprocessed training data set into the convolutional neural network with attention mechanism to obtain the global features, local features and the prediction vectors corresponding to the local features; Step 3: Extract clusters based on the acquired global features and assign pseudo labels; Step 4: Calculate the dynamic weight coefficient based on the global features and local features; Step 5: Use the dynamic weight coefficient to refine the local feature pseudo-label, and use the local feature loss to train the network to obtain the locally trained network; Step 6: Use the integrated weights and prediction vectors of the local features to refine the global feature pseudo-labels, and use the global feature loss to train the locally trained network to refine the pseudo-labels to obtain refined pseudo-labels.

[0007] Optionally, step 1 includes: In form, represents the unlabeled training dataset, where is an image, is the number of images; the feature map is represented as , where C, H and W are the channel size, height and width of the feature map respectively; firstly, the collected pedestrian image dataset is preprocessed, and the preprocessing includes: horizontal flipping, padding, and random cropping.

[0008] Optionally, the step 2 includes: In the attention mechanism, for the feature map First, perform global pooling, one-dimensional convolution, flattening and transposition, calculate attention weights, build attention matrix, feature weighting, convolution and activation function operations to obtain the weighted feature map ; The convolutional neural network consists of multiple blocks, each of which contains a convolution layer, a pooling layer, and an activation function layer. The attention mechanism is added to the back of the first block and the back of the last block of the convolutional neural network to form a convolutional neural network with an attention mechanism; after passing through the convolutional neural network with an attention mechanism, local features and global features are obtained; the initial data set images are horizontally split and then passed through the convolutional neural network to obtain local features, and the initial data set images are not horizontally split and directly passed through the convolutional neural network to obtain global features, which are finally expressed as and , and then the prediction vector corresponding to the local feature is obtained through the classifier .

[0009] Optionally, the step 3 includes: For the global feature set Perform DBSCAN clustering and assign pseudo labels to clusters; The pseudo label is recorded as , which is a one-hot encoding with hard assignment of K clusters.

[0010] Optionally, step 4 includes: According to local features and global features, the features are sorted based on the similarity between feature vectors, and the feature index list of the first k nearest neighbors is returned, which are represented as , ; Calculate the dynamic weight coefficients of global features and local features, where the dynamic weight coefficient calculation formula is: ; in, and The global features are and local features The index set of the first k samples in the calculated ranking list, and Represents the number of set elements, and finally obtains the weight, expressed as .

[0011] Optionally, step 5 includes: Using dynamic weight coefficients to refine local feature labels, given an image Pseudo-labels , local features The specific formula for label smoothing is: ; where u is a uniform vector, is the weight, which is used to dynamically adjust the weight of each part; the local feature loss formula is specifically expressed as: ; in and They are cross entropy and KL divergence, and the two terms are balanced by the dynamic coefficient; the former pushes the model's prediction results towards the true label close to, and reach a high confidence level, the latter term forces the prediction results to converge to a uniform vector.

[0012] Optionally, step 6 includes: The integrated weights and prediction vectors of local features are used to refine the global feature pseudo-labels. The specific formula is: ,in and Local features The integrated weights and prediction vector of Is a weight parameter used to control the ratio between pseudo labels and integrated predictions; the formula for global feature loss is: ; Use local features to refine global feature pseudo-labels to obtain refined pseudo-labels.

[0013] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the pseudo-label refining method based on global and local feature fusion based on the attention mechanism in the first aspect or any possible implementation of the first aspect.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to execute the pseudo-label refinement method for global and local feature fusion based on the attention mechanism in the first aspect or any possible implementation of the first aspect.

[0015] In the technical solution provided by the present invention, the method includes acquiring pedestrian image data sets from different cameras, performing preprocessing, and obtaining a preprocessed training data set; inputting the preprocessed training data set into a convolutional neural network containing an attention mechanism to obtain global features and local features and prediction vectors corresponding to the local features; extracting clusters according to the acquired global features and assigning pseudo labels; calculating dynamic weight coefficients according to the global features and local features; refining local feature pseudo labels using the dynamic weight coefficients, and training the network using local feature loss to obtain a locally trained network; refining global feature pseudo labels using the integrated weights and prediction vectors of the local features, and training the locally trained network using the global feature loss to refine the pseudo labels and obtain refined pseudo labels. The method improves the recognition accuracy, reduces the calculation cost, and ensures the performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of a pseudo-label refining method based on global and local feature fusion of an attention mechanism provided in an embodiment of the present invention; Figure 2 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0021] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0022] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0023] Figure 1 A flowchart of a pseudo-label refining method based on global and local feature fusion of an attention mechanism provided in an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes: Step 1: Obtain pedestrian image datasets from different cameras, perform preprocessing, and obtain the preprocessed training dataset.

[0024] In the embodiment of the present invention, step 1 includes: In form, represents the unlabeled training dataset, where is an image, is the number of images; the feature map is represented as , where C, H and W are the channel size, height and width of the feature map respectively; firstly, the collected pedestrian image dataset is preprocessed, and the preprocessing includes: horizontal flipping, padding, and random cropping.

[0025] Step 2: Input the preprocessed training data set into the convolutional neural network with attention mechanism to obtain the global features, local features and the prediction vectors corresponding to the local features.

[0026] In the embodiment of the present invention, step 2 includes: In the attention mechanism, for the feature map First, perform global pooling, one-dimensional convolution, flattening and transposition, calculate attention weights, build attention matrix, feature weighting, convolution and activation function operations to obtain the weighted feature map ; The convolutional neural network consists of multiple blocks, each of which contains a convolution layer, a pooling layer, and an activation function layer. The attention mechanism is added to the back of the first block and the back of the last block of the convolutional neural network to form a convolutional neural network with an attention mechanism; after passing through the convolutional neural network with an attention mechanism, local features and global features are obtained; the initial data set images are horizontally split and then passed through the convolutional neural network to obtain local features, and the initial data set images are not horizontally split and directly passed through the convolutional neural network to obtain global features, which are finally expressed as and , and then the prediction vector corresponding to the local feature is obtained through the classifier .

[0027] Step 3: Extract clusters based on the obtained global features and assign pseudo labels.

[0028] In the embodiment of the present invention, step 3 includes: For the global feature set Perform DBSCAN clustering and assign pseudo labels to clusters; The pseudo label is recorded as , which is a one-hot encoding with hard assignment of K clusters.

[0029] In the embodiment of the present invention, the core idea of ​​DBSCAN clustering is to divide the area with sufficiently high density into clusters, and randomly select points by setting the scanning radius and the minimum number of points. The points within the scanning radius are regarded as core points. If they are core points, they are expanded into clusters, otherwise they are skipped until all points are processed.

[0030] Step 4: Calculate the dynamic weight coefficient based on the global features and local features.

[0031] In the embodiment of the present invention, step 4 includes: According to local features and global features, the features are sorted based on the similarity between feature vectors, and the feature index list of the first k nearest neighbors is returned, which are represented as , ; Calculate the dynamic weight coefficients of global features and local features, where the dynamic weight coefficient calculation formula is: ; in, and The global features are and local features The index set of the first k samples in the calculated ranking list, and Represents the number of set elements, and finally obtains the weight, expressed as .

[0032] Step 5: Use the dynamic weight coefficient to refine the local feature pseudo-label, and use the local feature loss to train the network to obtain the locally trained network.

[0033] In the embodiment of the present invention, step 5 includes: Using dynamic weight coefficients to refine local feature labels, given an image Pseudo-labels , local features The specific formula for label smoothing is: ; where u is a uniform vector, is the weight, which is used to dynamically adjust the weight of each part; the local feature loss formula is specifically expressed as: ; in and They are cross entropy and KL divergence, and the two terms are balanced by the dynamic coefficient; the former pushes the model's prediction results towards the true label close to, and reach a high confidence level, the latter term forces the prediction results to converge to a uniform vector.

[0034] Step 6: Use the integrated weights and prediction vectors of the local features to refine the global feature pseudo-labels, and use the global feature loss to train the locally trained network to refine the pseudo-labels to obtain refined pseudo-labels.

[0035] In the embodiment of the present invention, step 6 includes: The integrated weights and prediction vectors of local features are used to refine the global feature pseudo-labels. The specific formula is: ,in and Local features The integrated weights and prediction vector of Is a weight parameter used to control the ratio between pseudo labels and integrated predictions; the formula for global feature loss is: ; Use local features to refine global feature pseudo-labels to obtain refined pseudo-labels.

[0036] In the embodiment of the present invention, local features are used to refine global feature pseudo labels. Global features can learn rich and fine-grained information from local prediction vectors to improve recognition capabilities. Moreover, this is a self-learning method that does not require an additional network and has high computational efficiency.

[0037] The present invention introduces the attention mechanism to integrate global features, local features, and information in attention to comprehensively capture the key information in the image and improve model performance.

[0038] Each step of the embodiment of the present invention may be performed by an electronic device, which includes but is not limited to a mobile phone, a tablet computer, a portable PC, a desktop computer, etc.

[0039] In the technical solution provided by the present invention, the method includes acquiring pedestrian image data sets from different cameras, performing preprocessing, and obtaining a preprocessed training data set; inputting the preprocessed training data set into a convolutional neural network containing an attention mechanism to obtain global features and local features and prediction vectors corresponding to the local features; extracting clusters according to the acquired global features and assigning pseudo labels; calculating dynamic weight coefficients according to the global features and local features; refining local feature pseudo labels using the dynamic weight coefficients, and training the network using local feature loss to obtain a locally trained network; refining global feature pseudo labels using the integrated weights and prediction vectors of the local features, and training the locally trained network using the global feature loss to refine the pseudo labels and obtain refined pseudo labels. The method improves the recognition accuracy, reduces the calculation cost, and ensures the performance of the network.

[0040] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute an embodiment of the pseudo-label refining method based on global and local feature fusion based on the attention mechanism.

[0041] Figure 2 A schematic diagram of an electronic device provided by an embodiment of the present invention, such as Figure 2 As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, the pseudo-label refining method for global and local feature fusion based on the attention mechanism in the embodiment is implemented. To avoid repetition, they are not described one by one here.

[0042] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will appreciate that Figure 2 It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0043] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0044] The memory 212 may be an internal storage unit of the electronic device 21, such as a hard disk or memory of the electronic device 21. The memory 212 may also be an external storage device of the electronic device 21, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 21. Further, the memory 212 may also include both an internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 may also be used to temporarily store data that has been output or is to be output.

[0045] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A pseudo-label refinement method based on global and local feature fusion based on attention mechanism, characterized in that: The method comprises: Step 1: Obtain pedestrian image datasets from different cameras, perform preprocessing, and obtain the preprocessed training dataset; Step 2: Input the preprocessed training data set into the convolutional neural network with attention mechanism to obtain the global features, local features and the prediction vectors corresponding to the local features; Step 3: Extract clusters based on the acquired global features and assign pseudo labels; Step 4: Calculate the dynamic weight coefficient based on the global features and local features; Step 5: Use the dynamic weight coefficient to refine the local feature pseudo-label, and use the local feature loss to train the network to obtain the locally trained network; Step 6: Use the integrated weights and prediction vectors of the local features to refine the global feature pseudo-labels, and use the global feature loss to train the locally trained network to refine the pseudo-labels to obtain refined pseudo-labels.

2. The method according to claim 1, characterized in that The step 1 comprises: In form, represents the unlabeled training dataset, where is an image, is the number of images; the feature map is represented as , where C, H and W are the channel size, height and width of the feature map respectively; firstly, the collected pedestrian image dataset is preprocessed, and the preprocessing includes: horizontal flipping, padding, and random cropping.

3. The method according to claim 1, characterized in that: The step 2 comprises: In the attention mechanism, for the feature map First, perform global pooling, one-dimensional convolution, flattening and transposition, calculate attention weights, build attention matrix, feature weighting, convolution and activation function operations to obtain the weighted feature map ; The convolutional neural network consists of multiple blocks, each of which contains a convolution layer, a pooling layer, and an activation function layer. The attention mechanism is added to the back of the first block and the back of the last block of the convolutional neural network to form a convolutional neural network with an attention mechanism; after passing through the convolutional neural network with an attention mechanism, local features and global features are obtained; the initial data set images are horizontally split and then passed through the convolutional neural network to obtain local features, and the initial data set images are not horizontally split and directly passed through the convolutional neural network to obtain global features, which are finally expressed as and , and then the prediction vector corresponding to the local feature is obtained through the classifier .

4. The method according to claim 1, characterized in that: The step 3 comprises: For the global feature set Perform DBSCAN clustering and assign pseudo labels to clusters; The pseudo label is recorded as , which is a one-hot encoding with hard assignment of K clusters.

5. The method according to claim 1, characterized in that The step 4 comprises: According to local features and global features, the features are sorted based on the similarity between feature vectors, and the feature index list of the first k nearest neighbors is returned, which are represented as , ; Calculate the dynamic weight coefficients of global features and local features, where the dynamic weight coefficient calculation formula is: ； in, and The global features are and local features The index set of the first k samples in the calculated ranking list, and Represents the number of set elements, and finally obtains the weight, expressed as .

6. The method according to claim 1, characterized in that The step 5 comprises: Using dynamic weight coefficients to refine local feature labels, given an image Pseudo-labels , local features The specific formula for label smoothing is: ; where u is a uniform vector, is the weight, which is used to dynamically adjust the weight of each part; the local feature loss formula is specifically expressed as: ； in and They are cross entropy and KL divergence, and the two terms are balanced by the dynamic coefficient; the former pushes the model's prediction results towards the true label close to, and reach a high confidence level, the latter term forces the prediction results to converge to a uniform vector.

7. The method according to claim 1, characterized in that The step 6 comprises: The integrated weights and prediction vectors of local features are used to refine the global feature pseudo-labels. The specific formula is: ,in and Local features The integrated weights and prediction vector of Is a weight parameter used to control the ratio between pseudo labels and integrated predictions; the formula for global feature loss is: ; Use local features to refine global feature pseudo-labels to obtain refined pseudo-labels.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is run, the computer-readable storage medium is controlled to execute the pseudo-label refining method for global and local feature fusion based on the attention mechanism as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: one or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to perform the pseudo-label refining method based on global and local feature fusion based on the attention mechanism as described in any one of claims 1 to 7.

Images