Training method of image encoder, image processing method, electronic device, storage medium and program product
By training an image encoder and using a combination loss function and clothing text word optimization model, the matching difficulty caused by clothing changes in long-term person re-identification was solved, improving recognition accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-19
- Publication Date
- 2026-03-31
AI Technical Summary
Existing long-term person re-identification methods struggle to accurately match individuals after they have changed clothes, and exhibit low robustness and recognition accuracy, especially under poor visual conditions.
By training an image encoder and iteratively optimizing model parameters using a combined loss function, clothing text symbols are generated. This reduces the visual symbol differences for the same person in different clothing and increases the differences between visual symbols and clothing text symbols, thus introducing a text modality to reduce reliance on visual information.
It improves the accuracy and robustness of person re-identification, reduces sensitivity to changes in image detection boxes and shooting angle, and enhances the recognition ability under different clothing.
Smart Images
Figure CN120766199B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a training method for an image encoder, an image processing method, an electronic device, a storage medium, and a program product. Background Technology
[0002] In applications such as security monitoring, intelligent transportation, and healthcare monitoring, there is a need for cross-camera person matching. While related technologies have made significant progress in short-term tasks, in long-term monitoring, changes in clothing can make it difficult for algorithms to accurately match people. Therefore, person re-identification technologies for clothing-changing situations have emerged, aiming to improve long-term person re-identification performance by learning and extracting features unrelated to clothing. Current long-term person re-identification methods are mainly divided into two categories: unimodal and multimodal. Unimodal methods extract clothing-independent features through specially designed learning methods and loss functions, while multimodal methods use additional information such as body segmentation maps and contour maps to assist in feature extraction. However, these methods rely heavily on visual information and are easily affected by camera angle and detection errors. Under poor visibility conditions, person re-identification exhibits poor robustness and low accuracy. Summary of the Invention
[0003] This application provides a training method for an image encoder, an image processing method, an electronic device, a storage medium, and a program product to alleviate or solve one or more technical problems existing in the prior art.
[0004] In a first aspect, embodiments of this application provide a method for training an image encoder, comprising:
[0005] The training images are input into the image encoder, which extracts visual symbols to represent image features.
[0006] Generate clothing text symbols for training images; the clothing text symbols are used to describe the clothing features of people in the training images;
[0007] The model parameters of the image encoder are iteratively trained using a combined loss function; the combined loss function is used to make the visual word differences of the image encoder for multiple training images of the same person smaller, and to make the visual word differences of the clothing text words in the same training image larger.
[0008] In some implementations, the iterative training of the image encoder model parameters using a combined loss function includes:
[0009] Obtain the pre-constructed combined loss function; the combined loss function includes a weighted sum of the following functions: cross-entropy function, orthogonality function, and contrast function; the cross-entropy function is used to represent the difference between the classification prediction result of the visual word and the person label of the training image; the orthogonality function represents the similarity between the visual word and the clothing text word; the contrast function represents the difference between different enhanced visual words, where any enhanced visual word is obtained by randomly enhancing the visual word using the clothing text word;
[0010] Calculate the loss value of each function in the combined loss function;
[0011] The optimized values of the image encoder model parameters are calculated based on the weighted sum of the loss values of each function, and the image encoder is optimized based on the optimized values.
[0012] In some implementations, for the cross-entropy function, calculating the loss value of each function in the combined loss function includes:
[0013] The training image is input into a classification network to obtain the classification prediction result; the classification prediction result is used to represent the probability of classifying the training image as the person label corresponding to the training image;
[0014] The loss value of the cross-entropy function is calculated based on the probability.
[0015] In some implementations, for the orthogonal function, calculating the loss value of each function in the combined loss function includes:
[0016] Using a multimodal self-attention neural network structure, the visual symbols for clothing that correspond to the text symbols for clothing are searched among the visual symbols;
[0017] The loss value of the orthogonal function is determined based on the similarity between the visual symbols of the clothing and the visual symbols.
[0018] In some implementations, for the comparison function, calculating the loss value of each function in the combined loss function includes:
[0019] Obtain two different weight values to get the first weight value and the second weight value;
[0020] Search among the visual symbols for clothing visual symbols that correspond to the clothing text symbols;
[0021] The loss value of the comparison function is obtained based on the following formula: (visual symbol - first weight value × visual symbol of clothing) - (visual symbol - second weight value × visual symbol of clothing).
[0022] In some implementations, generating clothing text symbols for the training images includes:
[0023] Large models are used to generate clothing description text to describe the clothing of people in the training images;
[0024] Input the clothing description text into a text encoder to obtain the clothing text tokens.
[0025] Secondly, embodiments of this application provide an image processing method, including:
[0026] The target image is input into the image encoder, wherein the image encoder is trained using the image encoder training method provided in the embodiments of this application;
[0027] The output of the image encoder is obtained to extract the image region containing human features from the target image.
[0028] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.
[0029] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.
[0030] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the embodiments of this application.
[0031] Based on the image encoder training method provided in the above embodiments, this application has at least the following beneficial effects or advantages:
[0032] This paper addresses the generation of clothing text symbols from training images to describe a person's attire. These clothing text symbols are used as enhancement features to identify clothing-independent features within the image symbols of the training images. By introducing text modalities, the reliance on visual information is reduced. Specifically, by designing a combined loss function, the differences in visual symbols extracted by the image encoder from multiple training images of the same person are reduced, while the differences between visual symbols and clothing text symbols are increased. This improves the accuracy of the image encoder in recognizing the same person under different clothing conditions, reduces sensitivity to changes in image detection boxes and shooting perspective, and enhances the accuracy and robustness of person re-identification.
[0033] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description
[0034] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.
[0035] Figure 1 A flowchart illustrating a training method for an image encoder provided in an embodiment of this application is shown;
[0036] Figure 2 Another flowchart of a training method for an image encoder provided in an embodiment of this application is shown;
[0037] Figure 3 A flowchart of an image processing method provided in an embodiment of this application is shown;
[0038] Figure 4 A schematic diagram of a training apparatus for an image encoder provided in an embodiment of this application is shown;
[0039] Figure 5 A schematic diagram of an image processing apparatus provided in an embodiment of this application is shown;
[0040] Figure 6 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation
[0041] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.
[0042] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.
[0043] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0044] See Figure 1 The flowchart shown is a training method for an image encoder, which specifically includes steps 101 to 103.
[0045] Step 101: Input the training image into the image encoder and extract visual symbols to represent image features through the image encoder;
[0046] Step 102: Generate clothing text symbols for the training images; clothing text symbols are used to describe the clothing features of people in the training images;
[0047] Step 103: Iteratively train the model parameters of the image encoder using the combined loss function; the combined loss function is used to make the visual word differences of the image encoder for multiple training images of the same person smaller, and to make the differences between the visual words and clothing text words of the same training image larger.
[0048] The image encoder training method provided in this application is used to train an image encoder so that the trained image encoder can extract human feature regions that do not depend on clothing as much as possible, thereby more accurately identifying people. It can be applied to human re-identification technology to improve the accuracy and robustness of human re-identification.
[0049] During the training of the image encoder, training images are input into the image encoder to obtain visual symbols extracted by the image encoder. Visual symbols represent image features and are features extracted by the image encoder based on the image. For example, visual symbols can be data formats such as vectors or matrices. In one example, the image encoder can use the VisionTransformer ViT-B / 16 model as the base model for training. It is understood that other models can also be used as the base model, and this application does not limit this. For example, the image encoder can be a model based on architectures such as Convolutional Neural Networks (CNN) or Transformers. For instance, using ResNet as the image encoder, it extracts features from the image through operations such as convolutional layers and pooling layers to obtain visual symbols used to represent image features. These visual symbols can be high-level semantic feature vectors of the image.
[0050] Different implementation methods can be designed to generate clothing text symbols that describe the clothing features of people in the training images.
[0051] In one example, a neural network model can be trained using sample images labeled with text tags of clothing, enabling the trained neural network model to generate clothing text symbols based on the images.
[0052] In another example, training images can be fed into a large model, which generates clothing description text to describe the clothing of people in the training images. The clothing description text is then input into a text encoder to obtain clothing text symbols.
[0053] Optionally, the text symbols can be in data formats such as vectors or matrices. For example, the text encoder can be a Transformer model or a BERT model, and the model parameters can be pre-trained using CLIP text image. It is understood that for the pre-trained text encoder, the model parameters of the text encoder can be frozen during the training of the image encoder to reduce the impact of changes in the text encoder model on the training accuracy.
[0054] In some implementations, prompts can be input into the large model to help it understand the need to describe clothing. For example, "Please describe the clothing of the person in the image," and the large model will generate a statement describing the clothing of the person in the image. Optionally, the large model can be a multimodal large model, or a large model finely tuned based on a large model in related technologies (such as GPT-3, GPT-4, etc.). This application does not limit the specific implementation method.
[0055] Incorporating clothing text symbols into the loss function allows the information carried by these symbols to be used as reference information for training the image encoder. This decoupling of image features through textual cues from the clothing text symbols reduces reliance on visual information. In essence, during iterative training, the goal is to minimize the loss function. The backpropagation algorithm calculates the gradient based on the combined loss function, iteratively updating the image encoder's model parameters to optimize the model in the desired direction: reducing the differences between visual symbols for the same person and increasing the differences between visual symbols and clothing text symbols.
[0056] When designing the sub-functions in the combined loss function, at least one of the following aspects should be considered: 1. The difference in visual symbols generated by the image encoder for the same person; 2. The similarity between the visual symbols extracted from the same training image and the clothing text symbols. This combined loss function design can increase the accuracy of the image encoder in recognizing the same person under different clothing conditions, reduce the sensitivity to changes in the image detection box and shooting angle, and improve the accuracy and robustness of person re-identification. By training with the combined loss function, both the consistency of the image's own features and the relevance to the clothing text description can be considered simultaneously, making the image encoder more targeted in capturing image features. This results in better performance for subsequent tasks involving clothing recognition and image retrieval, improving the model's ability to understand and distinguish person images.
[0057] In some implementations, the model parameters of the image encoder are iteratively trained using a combined loss function, including: obtaining a pre-constructed combined loss function; the combined loss function includes a weighted sum of the following functions: a cross-entropy function, an orthogonal function, and a contrastive function; the cross-entropy function is used to represent the difference between the classification prediction result of the visual word and the person label of the training image; the orthogonal function represents the similarity between the visual word and the clothing text word; the contrastive function represents the difference between different enhanced visual words, where any enhanced visual word is obtained by randomly enhancing the visual word using the clothing text word; calculating the loss value of each function in the combined loss function respectively; calculating the optimized value of the image encoder model parameters based on the weighted sum of the loss values of each function, and optimizing the image encoder based on the optimized value.
[0058] The combined loss function is composed of multiple loss functions with different functions, used to guide the model to integrate multiple learning objectives. A multimodal self-attention neural network structure is a neural network structure capable of processing multimodal data (such as visual and textual data) and capturing their interrelationships. Optionally, the combined loss function is constructed by adding the cross-entropy function, orthogonal function, and contrastive function with certain weights. These weights can be determined empirically or through hyperparameter search. For example, the weights of the cross-entropy function, orthogonal function, and contrastive function can be set to 0.4, 0.3, and 0.3 respectively. For the cross-entropy function, the training image is input into a classification network (which can be a network constructed with additional fully connected layers, etc.) to obtain the probability distribution of the training image belonging to each preset person label. Then, the cross-entropy is calculated with the true labels to obtain the loss value. Regarding the orthogonal function, in the multimodal self-attention neural network structure, the mechanism of its self-attention layer, pre-propagation layer, and cross-attention layer is used to find the clothing visual symbols that correspond to the clothing text symbols in the visual symbols, and the similarity between the two is calculated as the loss value of the orthogonal function. The contrastive function calculates the corresponding loss value based on different weights and the found clothing visual symbols according to the given formula. The gradient of the image encoder model parameters is calculated based on the total loss after weighted summation of the loss values of each function, and the model parameters are updated and optimized using an optimization algorithm (such as the Adam optimization algorithm).
[0059] This implementation comprehensively considers classification accuracy (cross-entropy), the distinguishability of visual and textual features (orthogonal function), and the differences in features under different enhancement conditions (contrast function), making the image encoder more comprehensive and accurate in feature extraction. It can better cope with the learning task of human features in complex and diverse image scenes, and improve the robustness and generalization ability of the model in practical applications.
[0060] In some implementations, for the cross-entropy function, the loss values of each function in the combined loss function are calculated separately, including: inputting the training image into the classification network to obtain the classification prediction result; the classification prediction result is used to represent the probability of classifying the training image as the person label corresponding to the training image; and calculating the loss value of the cross-entropy function based on the probability.
[0061] The aforementioned classification network is a neural network model used to classify input feature data into preset categories. It performs classification prediction by learning the mapping relationship between features and categories. For example, the classification network can contain multiple fully connected layers and activation functions. After processing by the classification network, an output vector is obtained, representing the probability distribution of the training image belonging to each different person label. For instance, with 10 person labels, the output vector dimension is 10, and each element corresponds to the probability of belonging to the corresponding category. Then, using the cross-entropy formula, the predicted probability distribution is compared with the true labels (usually in one-hot encoded form) to calculate the cross-entropy loss value, reflecting the difference in the accuracy of the classification prediction.
[0062] The cross-entropy function in the combined loss function can explicitly supervise and optimize the performance of the image encoder on the person classification task, enabling the visual symbols extracted by the image encoder to better reflect the identity characteristics of the person, and playing a key role in improving the accuracy of person recognition and other related tasks.
[0063] In some implementations, for the orthogonal function, the loss value of each function in the combined loss function is calculated separately, including: using a multimodal self-attention neural network structure to find clothing visual symbols corresponding to clothing text symbols in visual symbols; and determining the loss value of the orthogonal function based on the similarity between clothing visual symbols and other visual symbols. In some implementations, the multimodal self-attention neural network structure includes a self-attention layer, a forward layer, and a cross-attention layer.
[0064] Orthogonal functions aim to ensure that visual symbols and clothing text symbols are as orthogonal as possible in the feature space, thereby reducing their similarity and allowing the visual symbols extracted by the image encoder to focus more on human features rather than clothing features. An exemplary calculation of the orthogonal function loss value is as follows: First, the training image is input into the image encoder to extract visual symbols. Next, a multimodal self-attention neural network structure is used, which includes a self-attention layer, a forward layer, and a cross-attention layer. In the self-attention layer, text symbols interact with each other to determine the relevance of each symbol to other symbols; the forward layer performs a non-linear transformation on the symbols; the cross-attention layer uses the clothing text symbols as a query vector, incorporating a key vector and a value vector generated from the visual symbols, focusing the value vector on features related to the clothing description, thus obtaining clothing visual symbols that match the clothing text symbols. By finding the clothing visual symbols corresponding to the clothing text symbols, the orthogonal function is used to calculate the similarity between the clothing text symbols and the visual symbols, and finally, the loss value of the orthogonal function is determined based on the similarity.
[0065] By using a multimodal self-attention neural network structure, clothing visual symbols corresponding to clothing text symbols are retrieved from visual symbols. Then, orthogonal functions are used to orthogonalize the features of clothing visual symbols and visual symbols, making them more independent in the feature space, reducing similarity, and improving the performance of the image encoder in recognizing the features of the person themselves.
[0066] In some implementations, for the contrast function, the loss value of each function in the combined loss function is calculated separately, including: obtaining two different weight values to obtain a first weight value and a second weight value; searching for the clothing visual symbol corresponding to the clothing text symbol in the visual symbols; and obtaining the loss value of the contrast function based on the following formula: (visual symbol - first weight value × clothing visual symbol) - (visual symbol - second weight value × clothing visual symbol).
[0067] Regarding the calculation method of the contrast function loss value, the specific execution steps of an example are as follows:
[0068] First, a first weight value and a second weight value are randomly selected from a preset weight range. These two weight values are used for subsequent weighting operations on clothing visual symbols. Then, the aforementioned multimodal self-attention neural network structure can be used to search for clothing visual symbols corresponding to clothing text symbols among the visual symbols. Specifically, the multimodal self-attention neural network structure includes a self-attention layer, a forward layer, and a cross-attention layer. The self-attention layer is responsible for capturing the internal relationships of visual symbols, the forward layer performs nonlinear transformations on them, and the cross-attention layer introduces clothing text symbols, enabling visual symbols to focus on features related to clothing descriptions. Next, the loss value of the contrast function is calculated according to the formula (visual symbol - first weight value × clothing visual symbol) - (visual symbol - second weight value × clothing visual symbol). This formula measures the difference between different enhanced visual symbols by comparing the differences between combinations of visual symbols and clothing visual symbols under two different weights. By gradually reducing the contrast function loss value during iterative training, the variation between visual symbols and clothing visual symbols can be reduced. This makes the visual symbols extracted by the image encoder more inclined to represent human features rather than clothing features, reducing the image encoder's dependence on clothing features and enhancing the accuracy of correct recognition in images of the same person wearing different clothes.
[0069] The implementation steps for generating clothing text symbols from training images are as follows: First, a large model is used to generate clothing description text to describe the clothing of people in the training images. Specifically, the training images are input into the large model, which generates a text description based on the image content. This text accurately reflects the clothing features of the people in the image. For example, the large model can describe detailed information such as the type, color, and style of the clothing worn by the people. Then, the generated clothing description text is input into a text encoder, which converts this text into clothing text symbols. The text encoder is typically a deep learning-based model, such as the Transformer-based BERT model, which encodes the semantic information in the text into vector form. These vectors are the clothing text symbols, which can be used in the subsequent image encoder training process.
[0070] This implementation method offers significant advantages and beneficial effects. First, the large model possesses powerful language generation capabilities, enabling the generation of high-quality, semantically rich text descriptions of clothing. This provides accurate and detailed textual information for subsequent image encoder training, helping to improve the model's understanding and representation of clothing features. Second, by converting clothing description text into clothing text symbols, effective fusion of visual and textual features can be achieved. This allows the image encoder to combine textual features when learning image features, enhancing its semantic understanding of clothing features in images, thus making the model more accurate in handling tasks related to clothing. For example, in image retrieval tasks, it can more accurately find corresponding images based on clothing descriptions; in clothing changing tasks, it can better understand the features of target clothing and generate more suitable changing images. Furthermore, this method of fusing visual and textual features can also improve the model's generalization ability, enabling it to better utilize existing textual knowledge for feature representation and understanding when faced with new and unseen images, thereby improving the model's stability and reliability in practical applications.
[0071] The training method of an image encoder, as specifically exemplified in this application, is described below. Figure 2 This is a schematic diagram of the training method for the example image encoder.
[0072] refer to Figure 2 After acquiring an image, the image is input to the image encoder and the multimodal large model respectively.
[0073] An image encoder can generate multiple text tokens to represent image features. The text tokens generated by the image encoder include multiple small image patches (e.g., 16*16 pixels) formed due to image segmentation. The first text token, CLS201, is equivalent to a global representation used to aggregate information from the entire input sequence and is used for downstream classification tasks.
[0074] The image is input into a multimodal large-scale model, which performs visual question answering using a multimodal large-scale language model. The resulting text describes clothing: "A man wearing a black shirt with white writing on it and a pair of black pants." This text is then input into a pre-trained text encoder. While the image encoder's parameters are updated during iterative training, the text encoder's parameters are frozen during the iterative training of the image encoder. The text encoder generates multiple text tokens describing clothing features based on a unidirectional Transformer model. Each text token's representation depends only on the preceding text tokens, and the last text token in the sequence, EOS202, aggregates the information from the entire input text.
[0075] A cross-modal feature extractor, namely a multimodal self-attention neural network structure, is designed. Specifically, the extractor includes a self-attention layer, a forward propagation layer, and a cross-attention layer. Based on clothing text symbols as query vector Q, and generating key vector K and value vector V based on visual symbols, the visual features related to clothing text symbols in the visual symbol f(I) are extracted through the three-layer structure of self-attention, forward propagation, and cross-attention to obtain the clothing visual symbol f(T,I).
[0076] The combined loss function consists of a weighted sum of the following three loss functions:
[0077] Cross-entropy function L id The visual symbols and person identification labels are input into a cross-entropy function. Through function optimization, the visual symbols extracted from different photos of the same person are made as consistent as possible. After training based on the cross-entropy function, the visual symbols extracted by the image encoder strengthen the features representing the person's identity and weaken the features representing clothing.
[0078] Orthogonal function L ortThe visual features of clothing f(T,I) and visual symbols f(I) are input into an orthogonal function. The output orthogonal function loss value is the cosine similarity between the two. By optimizing the orthogonal function, the similarity between f(T,I) and f(I) can be reduced, making them as orthogonal as possible.
[0079] Comparison function L con The visual feature f(T,I) is calculated by subtracting two random weights from the visual symbol f(I), namely: f(I)-α1*f(T,I) and f(I)-α2*f(T,I), where α is a random number. α is randomly selected twice to obtain α1 and α2, resulting in two positive samples with the same randomly enhanced feature. This brings the distance between f(I)-α1*f(T,I) and f(I)-α2*f(T,I) closer, thereby weakening the clothing feature and strengthening the feature representing the person's identity.
[0080] The combination of three loss functions removes unwanted clothing-related features in the aligned visual language space, thereby driving the image encoder to retain only clothing-independent features in the extracted visual symbols. Finally, only the image encoder trained with feature decoupling needs to be retained, ensuring the efficiency of the model during deployment.
[0081] This embodiment has been experimentally verified to demonstrate strong capabilities in learning features that remove clothing interference, and exhibits high accuracy on multiple widely used benchmark datasets, surpassing existing methods for re-identifying people who have changed clothes. Furthermore, this embodiment also demonstrates good cross-domain generalization ability, offering the advantage of broad applicability.
[0082] This application also provides an image processing method, see embodiments thereof. Figure 3 The method includes the following steps:
[0083] Step 301: Input the target image into the image encoder, wherein the image encoder is trained using the training method of the image encoder provided in the embodiments of this application;
[0084] Step 302: Obtain the output of the image encoder to get the image region containing human features extracted from the target image.
[0085] Optionally, the image processing method provided in this application embodiment also provides an optional example, utilizing the output of an image encoder to identify people in a target image, thereby improving the accuracy and robustness of person re-identification. It is understood that the image processing method provided in this application embodiment can be applied to different fields, such as security monitoring, intelligent transportation, and medical health monitoring.
[0086] This application also provides an image encoder training device, which is a device corresponding to the image encoder training method provided in this application. It can be used to implement the image encoder training method provided in this application. Therefore, for the parts of the image encoder training device provided in this application that are not described in detail, please refer to the image encoder training method provided in this application. The same content will not be repeated.
[0087] refer to Figure 4 The image encoder training device provided in this application embodiment includes the following modules:
[0088] The input module 401 is used to input the training image into the image encoder, and the image encoder extracts visual symbols to represent image features;
[0089] The generation module 402 is used to generate clothing text symbols for the training images; the clothing text symbols are used to describe the clothing features of the people in the training images.
[0090] Training module 403 is used to iteratively train the model parameters of the image encoder using a combined loss function. The combined loss function is used to make the visual word differences of the image encoder for multiple training images of the same person smaller, and to make the differences between the visual words and clothing text words of the same training image larger.
[0091] The image encoder training device provided in this application embodiment is used to train the image encoder so that the trained image encoder can extract human feature regions that do not depend on clothing as much as possible, thereby more accurately identifying people. It can be applied to human re-identification technology to improve the accuracy and robustness of human re-identification.
[0092] This application also provides an image processing apparatus, which is a device corresponding to the image processing method provided in this application. It can be used to implement the image processing method provided in this application. Therefore, for the parts of the image processing apparatus provided in this application that are not described in detail, please refer to the image processing method provided in this application. The same content will not be repeated.
[0093] refer to Figure 5 The image processing apparatus provided in this application includes the following modules:
[0094] The input module 501 is used to input the target image into the image encoder, wherein the image encoder is trained using the training method of the image encoder provided in the embodiments of this application;
[0095] The acquisition module 502 acquires the output of the image encoder and obtains the image region containing human features extracted from the target image.
[0096] Figure 6 This is a block diagram of an electronic device used to implement embodiments of this application. Figure 6 As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the method described in the above embodiments. The number of memories 601 and processors 602 can be one or more. In a specific implementation, the electronic device may also include a communication interface 603 for communicating with external devices and exchanging data.
[0097] In practical implementation, if the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0098] Optionally, in a specific implementation, if the memory 601, processor 602 and communication interface 603 are integrated on a single chip, the memory 601, processor 602 and communication interface 603 can communicate with each other through an internal interface.
[0099] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.
[0100] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.
[0101] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.
[0102] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.
[0103] It should be understood that the aforementioned processor can be a CPU (Central Processing Unit), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.
[0104] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).
[0105] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.
[0106] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.
[0107] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.
[0108] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.
[0109] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).
[0110] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.
[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.
[0112] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A training method of an image encoder, characterized by, The method comprises the following steps: inputting a training image into an image encoder, and extracting visual words representing image features through the image encoder; generating clothing text words for the training image; the clothing text words are used to describe the clothing features of a person in the training image; iteratively training model parameters of the image encoder by using a combined loss function; the combined loss function is used to make the visual words of multiple training images of the same person generated by the image encoder less different, and make the visual words of the same training image and the clothing text words more different; the combined loss function comprises a weighted sum of a cross-entropy function, an orthogonality function and a contrast function; the cross-entropy function is used to represent the difference between the classification prediction result of the visual words and the person label of the training image; the orthogonality function represents the similarity between the visual words and the clothing text words; the contrast function represents the difference between different enhanced visual words, any of which is obtained by randomly enhancing the visual words by using the clothing text words; in the calculation step of the orthogonality function loss value, the clothing text words are used as query vectors by using a cross-attention layer, and are fused into key vectors and value vectors generated according to the visual words, clothing visual words matching the clothing text words are searched in the visual words, and the clothing visual words and the visual words are subjected to feature orthogonalization by using the orthogonality function.
2. The method of claim 1, wherein, The iteratively training model parameters of the image encoder by using the combined loss function comprises: obtaining the combined loss function constructed in advance; calculating the loss value of each function in the combined loss function respectively; calculating the optimization value of the image encoder model parameters according to the weighted sum of the loss values of each function, and optimizing the image encoder according to the optimization value.
3. The method of claim 2, wherein, For the cross-entropy function, the calculating the loss value of each function in the combined loss function respectively comprises: inputting the training image into a classification network to obtain the classification prediction result; the classification prediction result is used to represent the probability of classifying the training image into the person label corresponding to the training image; calculating the loss value of the cross-entropy function according to the probability.
4. The method of claim 2, wherein, For the orthogonality function, the calculating the loss value of each function in the combined loss function respectively comprises: finding the clothing visual words corresponding to the clothing text words in the visual words by using a multi-modal self-attention neural network structure; determining the loss value of the orthogonality function according to the similarity between the clothing visual words and the visual words.
5. The method of claim 2, wherein, For the contrast function, the calculating the loss value of each function in the combined loss function respectively comprises: obtaining two different weight values to obtain a first weight value and a second weight value; finding the clothing visual words corresponding to the clothing text words in the visual words; obtaining the loss value of the contrast function based on the following formula: (visual words-the first weight value×clothing visual words)-(visual words-the second weight value×clothing visual words).
6. The method of claim 1, wherein, The clothes text tokens are generated for the training image, comprising: generating clothes description text for describing the clothes of the character in the training image by using a large model; inputting the clothes description text into a text encoder to obtain the clothes text tokens.
7. An image processing method characterized by, comprising: inputting a target image into an image encoder, wherein the image encoder is trained by using the training method of the image encoder in any one of claims 1-6; obtaining the output of the image encoder to obtain the image region including the character features extracted from the target image. 8.An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of claims 1-7 when executing the computer program. 9.A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by a processor, implements the method of any one of claims 1-7. 10.A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any one of claims 1-7.