Vehicle local feature encoder training method and device, electronic equipment, storage medium and program product
Through local area detection and high-quality data set construction, the target visual encoder is trained, which solves the training problems of vehicle local feature encoder and the problem of insufficient data sets in the prior art, and realizes high-precision vehicle local feature retrieval.
Patent Information
- Application Number
- CN202411893919.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-05-09
AI Technical Summary
When the prior art realizes efficient vehicle local feature encoding, it is difficult to train a visual encoder with high descriptive power and discriminative power, and the lack of high-quality training data sets, resulting in insufficient accuracy in vehicle local feature retrieval.
By acquiring the vehicle image, local area detection and obtaining the area information, acquiring the area mask image and mask label based on the area information, fine-tuning the pre-trained visual encoder, building a high-quality training data set, and training the target visual encoder to obtain the vehicle local feature encoder.
It realizes high-precision training of vehicle local feature encoder, which can accurately distinguish subtle local feature differences and improves the retrieval accuracy of vehicle local features.
Smart Images
Figure CN119963968A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a vehicle local feature encoder training method, device, electronic device, storage medium and program product. Background Art
[0002] At present, vehicle local feature coding technology has important application value in the fields of intelligent transportation, vehicle monitoring and public safety. Unlike traditional vehicle re-identification (Re-Identification, ReID) technology, which mainly focuses on the overall characteristics of the vehicle, vehicle local feature coding focuses on specific parts of the vehicle, such as vehicle logos, annual inspection labels, pendants, sunroofs, etc. This feature extraction and retrieval based on specific parts has important practical value for identifying similar models, modified vehicles, etc. However, current technologies face the following major challenges in achieving efficient vehicle local feature coding: First, it is difficult to train a visual encoder with high descriptive and discriminative power. An ideal encoder should be able to extract feature vectors with rich information from the local area of the vehicle, accurately capture subtle differences, and have good robustness in complex scenes such as ambient lighting, perspective changes, and occlusion. Although existing models perform well in global feature extraction, they are still insufficient in the representation and differentiation of fine-grained local features, making it difficult to meet the needs of high-precision local feature extraction and matching.
[0003] Secondly, the lack of high-quality training datasets is a major obstacle to the training of vehicle local feature encoders. Existing vehicle ReID datasets only provide overall level annotation information, which results in the model being unable to obtain regional level supervision signals during training, limiting its performance in local feature extraction and differentiation. As a result, the current retrieval of vehicle local features is not accurate enough. Summary of the invention
[0004] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a vehicle local feature encoder training method, device, electronic device, storage medium and program product to improve the retrieval accuracy of vehicle local features.
[0005] A vehicle local feature encoder training method according to an embodiment of the first aspect of the present application includes: Acquire at least one vehicle image, perform local area detection on each of the vehicle images, and obtain area information of the local area of the corresponding vehicle image; the area information includes the area position and the area part type; Based on the area information, respectively obtain area mask images and mask labels from corresponding vehicle images; Based on the mask images of each region and their mask labels, the target visual encoder is trained to obtain a vehicle local feature encoder; wherein the target visual encoder is obtained by adding a multi-layer perceptron to the pre-trained visual encoder; the pre-trained visual encoder is used for local area detection of the vehicle.
[0006] According to one embodiment of the present application, the training of the target visual encoder based on each of the region mask images and their mask labels includes: Performing data screening on each of the region mask images to obtain at least one first region mask image; Performing data enhancement on each of the first region mask images to obtain a second region mask image; The target visual encoder is trained based on each of the second region mask images and their mask labels.
[0007] According to an embodiment of the present application, the step of performing data screening on each of the region mask images to obtain at least one first region mask image includes: Eliminating the regional mask images whose resolution is less than a preset resolution threshold from among the regional mask images, to obtain at least one third regional mask image; Determining negative samples and anchor point samples in each of the third region mask images; Determine the feature similarity between each negative sample and the anchor sample; The negative samples whose feature similarity is greater than a preset similarity threshold are adjusted, and the remaining third region mask image is used as the first region mask image.
[0008] According to one embodiment of the present application, the training of the target visual encoder based on each of the second region mask images and their mask labels includes: Resizing and formatting each of the second region mask images respectively to obtain a target region mask image; Based on the target area mask images and their mask labels, a low-rank adaptive technique is applied to train the target visual encoder.
[0009] According to one embodiment of the present application, the loss function of the target visual encoder is a triple loss function based on cosine similarity.
[0010] According to an embodiment of the present application, based on each of the area information, respectively obtaining the area mask image and the mask label from the corresponding vehicle image includes: For each vehicle image, the vehicle image and its region information are input into the image segmentation model to obtain the region mask image and mask label output by the image segmentation model.
[0011] A vehicle local feature encoder training device according to an embodiment of the second aspect of the present application includes: A detection module, used to obtain at least one vehicle image, perform local area detection on each of the vehicle images, and obtain area information of the local area of the corresponding vehicle image; the area information includes the area position and the area part type; An acquisition module, used for acquiring a regional mask image and a mask label from the corresponding vehicle image based on each of the regional information; A training module is used to train a target visual encoder based on each of the area mask images and their mask labels to obtain a vehicle local feature encoder; wherein the target visual encoder is obtained by adding a multi-layer perceptron to a pre-trained visual encoder; the pre-trained visual encoder is used for local area detection of the vehicle.
[0012] According to an electronic device of an embodiment of the third aspect of the present application, the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the vehicle local feature encoder training method described in any one of the above methods is implemented.
[0013] According to the storage medium of the fourth aspect embodiment of the present application, the storage medium is a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned vehicle local feature encoder training methods.
[0014] According to the computer program product of the fifth aspect of the present application, the computer program includes a computer program, which, when executed by a processor, implements any of the above-mentioned vehicle local feature encoder training methods.
[0015] The above one or more technical solutions in the embodiments of the present application have at least the following technical effects: By acquiring at least one vehicle image, performing local area detection on each vehicle image, and obtaining the regional information of the local area of the corresponding vehicle image, wherein the regional information includes the regional position and the regional part type; and further based on each regional information, obtaining the regional mask image and the mask label from the corresponding vehicle image; and, adding a multi-layer perceptron to the pre-trained visual encoder for local area detection of the vehicle to obtain the target visual encoder, thereby training the target visual encoder based on each regional mask image and its mask label to obtain the vehicle local feature encoder. By constructing high-quality training data including the local area mask image of the vehicle and its mask label, and training the target visual encoder obtained by fine-tuning the pre-trained visual encoder with the high-quality training data, a feature representation with both descriptive power and discriminative power can be obtained, so that the trained vehicle local feature encoder can accurately distinguish subtle local feature differences, meet the needs of high-precision local feature extraction and matching, and thus improve the retrieval accuracy of vehicle local features.
[0016] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 This is one of the flow charts of the vehicle local feature encoder training method provided in the embodiment of the present application.
[0019] Figure 2 This is the second flow chart of the vehicle local feature encoder training method provided in the embodiment of the present application.
[0020] Figure 3 This is the third flow chart of the vehicle local feature encoder training method provided in the embodiment of the present application.
[0021] Figure 4 It is a schematic diagram of the data set construction process in the vehicle local feature encoder training method provided in an embodiment of the present application.
[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION
[0023] The following is a further detailed description of the implementation of the present application in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but cannot be used to limit the scope of the present application.
[0024] In the description of the embodiments of the present application, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the embodiments of the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.
[0025] In the description of the embodiments of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present application can be understood according to specific circumstances.
[0026] In the embodiments of the present application, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.
[0027] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0028] The present application proposes a vehicle local feature encoder training method, device, electronic device, storage medium and program product.
[0029] Figure 1 is one of the flow charts of the vehicle local feature encoder training method provided in the embodiment of the present application, such as Figure 1 As shown, the vehicle local feature encoder training method includes: Step 110, obtaining at least one vehicle image, performing local area detection on each vehicle image, and obtaining area information of the local area of the corresponding vehicle image; the area information includes the area position and the area part type.
[0030] Step 120 , based on the information of each region, respectively obtain a region mask image and a mask label from the corresponding vehicle image.
[0031] Step 130, based on the mask images of each region and their mask labels, the target visual encoder is trained to obtain a vehicle local feature encoder; wherein the target visual encoder is obtained by adding a multi-layer perceptron to the pre-trained visual encoder; the pre-trained visual encoder is used for local area detection of the vehicle.
[0032] It should be noted that the execution subject of the vehicle local feature encoder training method provided in the embodiment of the present application can be a computer device, and the computer device can be, for example, a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It should be noted that the data required to be obtained in this application are all obtained through formal channels after authorization by relevant users.
[0033] A vehicle local feature encoder training device may be provided or connected in the computer device of the present application, thereby controlling the vehicle local feature encoder training device to execute the vehicle local feature encoder training method of the present application.
[0034] Figure 2 This is a second flow chart of the vehicle local feature encoder training method provided in the embodiment of the present application, such as Figure 2 As shown, the training process of the present application may include training data construction and model training.
[0035] Specifically, some vehicle images containing a small amount of background can be collected in advance from a vehicle ReID dataset (such as Veri-Wild). The number of vehicle images is not specifically limited in this application and can be set and adjusted according to actual needs. VERI-Wild is a large-scale vehicle re-identification dataset.
[0036] Furthermore, the vehicle region detection model can be used to detect the main significant local regions in the vehicle image, and output the regional positions (specifically the detection frame positions) and corresponding regional part types of significant local regions of the vehicle, including but not limited to annual inspection labels, pendants, tissue boxes, ornaments, sunroofs, luggage racks, front faces, and vehicle logos, as region information. The region information is given in the form of cropped vehicle images and text files, and the text files contain region labels (i.e., region part types) and detection frame position information.
[0037] Thus, the area information corresponding to each vehicle image is obtained.
[0038] Furthermore, according to the information of each region and its corresponding vehicle image, the target mask in the region can be automatically segmented for each region detection frame position to obtain a region mask image of the corresponding vehicle image and a mask label determined according to the region label.
[0039] For the spray-painted, painted and other significant local areas that may appear on the headlights, grilles, hoods, etc., manual annotation is used to obtain their regional mask images and mask labels.
[0040] All data is organized and stored in a hierarchy of the same brand, model, year, and vehicle identification (ID). Images and their labels are organized and stored by brand, model, year, vehicle ID, etc. to ensure a clear hierarchy of data for subsequent retrieval and processing.
[0041] In this way, we can ensure sample diversity and balance, avoid model bias caused by uneven data distribution, cover as many brands and models as possible, increase data diversity, enable the model to learn the characteristics of different categories and attributes, and improve generalization ability.
[0042] Furthermore, LLaVA 1.5 is used to generate the body color label, and the existing information is integrated to generate a text file containing the corresponding vehicle information in the format of "brand-model-year-color". Among them, LLaVA 1.5 is a multimodal large model.
[0043] In this way, a high-quality dataset containing rich vehicle local features and label information can be obtained.
[0044] In this application, the constructed data set can also be divided into positive samples, negative samples and anchor samples.
[0045] To ensure the model's discriminative ability, the construction of positive and negative sample pairs follows specific logical rules. According to the image category and vehicle ID or brand information, a triplet (anchor sample, positive sample, negative sample) is constructed. The anchor sample (Anchor) is the image corresponding to the current index selected from the dataset and has undergone preprocessing transformation. The selection rules for positive and negative samples (Negative) construct different positive and negative sample pairs according to the category classID and vehicle ID (or brand information).
[0046] The specific rules are as follows: Category 24 (car logo): Get the brand information corresponding to the current anchor sample. Positive sample selection: Find samples of the same brand and category 24 in the dataset. If there are multiple samples, randomly select one as the positive sample; if there are no samples of the same brand, select samples of different brands. Negative sample selection: Randomly select one from samples of different brands as the negative sample. If there are no samples of different brands, the negative sample is set as the positive sample by default; The categories are 12 (annual inspection label), 13 (pendant), 14 (tissue box), and 15 (ornament): Positive sample selection: Find samples with the same vehicle ID and the same category as the anchor sample, and randomly select one of them. If no conditions are met, select samples with different vehicle IDs but the same category. Negative sample selection: Randomly select one from the samples with different vehicle IDs as a negative sample; The categories are 17 (sunroof), 18 (luggage rack), and 20 (front face): Positive sample selection: Find samples with the same brand, year, and model information as the anchor sample and randomly select one. If no sample meets the conditions, select samples of different brands, years, and models. Negative sample selection: Randomly select one sample from samples of different brands, years, or models as a negative sample.
[0047] These rules ensure that positive samples have high similarity with anchor samples, while negative samples have large differences with anchor samples, thereby enhancing the model's ability to distinguish similar and dissimilar samples.
[0048] Furthermore, data quality enhancement can be performed on the dataset consisting of the mask images of each region and their mask labels to ensure the effectiveness and robustness of model training.
[0049] The present application can pre-acquire a pre-trained visual encoder for local area detection of a vehicle, and selectively freeze the pre-trained weights of the visual encoder during the fine-tuning process, that is, do not update it, maintain its original feature extraction capability, add a certain number (e.g., 1-2 layers) of multilayer perceptrons (MLP) on its top layer as a fine-tuning layer, and use the obtained visual encoder as the target visual encoder.
[0050] Among them, the visual encoder in this application can be a universal feature encoder AM-RADIO.
[0051] Specifically, in order to give full play to the potential of the AM-RADIO model, avoid overfitting and reduce the consumption of computing resources, the present application selectively freezes the pre-trained weights of the AM-RADIO model during the fine-tuning process, that is, does not update it, maintains its original feature extraction capability, and adds 1-2 layers of multi-layer perceptrons at the top layer of the model as fine-tuning layers.
[0052] Specifically, the newly added MLP layer includes one or more fully connected layers, accompanied by activation functions (such as ReLU function) and regularization layers (such as Dropout). During training, only the parameters of these newly added layers are updated, while the parameters of other layers remain unchanged, thereby reducing the number of training parameters and the risk of overfitting. It not only maintains the basic feature representation ability of the pre-trained model, but also enhances the adaptability of the model to specific downstream tasks. ReLU is one of the commonly used activation functions in neural networks, and Dropout is a commonly used regularization technique used to prevent neural networks from overfitting.
[0053] Furthermore, the AM-RADIO model, which is fine-tuned as the target visual encoder, can be trained using the data set with enhanced data quality. After training, a vehicle local feature encoder with the ability to recognize and retrieve local areas of vehicles in detection images is obtained.
[0054] After obtaining the vehicle local feature encoder, the vehicle image is input into the vehicle local feature encoder, and the vehicle local feature encoder can be used to identify and retrieve significant local areas of the vehicle in the vehicle image, for example, it can be used to identify similar models, modified vehicles, etc.
[0055] According to the vehicle local feature encoder training method of the embodiment of the present application, by acquiring at least one vehicle image, local area detection is performed on each vehicle image respectively, and the regional information of the local area of the corresponding vehicle image is obtained, wherein the regional information includes the regional position and the regional part type; and further based on each regional information, the regional mask image and the mask label are obtained from the corresponding vehicle image respectively; and, a multi-layer perceptron is added to the pre-trained visual encoder for local area detection of the vehicle in advance to obtain a target visual encoder, thereby training the target visual encoder based on each regional mask image and its mask label to obtain a vehicle local feature encoder. By constructing high-quality training data including the local area mask image of the vehicle and its mask label, and training the target visual encoder obtained by fine-tuning the pre-trained visual encoder with high-quality training data, a feature representation with both descriptive power and discriminative power can be obtained, so that the trained vehicle local feature encoder can accurately distinguish subtle local feature differences, which can meet the needs of high-precision local feature extraction and matching, thereby improving the retrieval accuracy of vehicle local features.
[0056] Based on the above embodiment, based on each region information, a region mask image and a mask label are obtained from the corresponding vehicle image respectively, including: For each vehicle image, the vehicle image and its region information are input into the image segmentation model to obtain the region mask image and mask label output by the image segmentation model.
[0057] Specifically, the present application can obtain or construct an image segmentation model, for example, SAM (SegmentAnything Model) can be obtained.
[0058] Therefore, the present application can input each vehicle image and its corresponding area information into SAM respectively. SAM can automatically segment the target mask in the area for each detection frame position, and obtain mask images and mask labels of significant areas including but not limited to annual inspection labels, pendants, ornaments, vehicle logos, etc.
[0059] It should be noted that for the spray-painted, painted and other significant local areas that may appear on the headlights, center grille, hood, etc., manual annotation is used to obtain their regional mask images and mask labels.
[0060] This application uses SAM to automatically segment the target mask, which not only improves the efficiency and accuracy of data processing, but also provides high-quality data support for subsequent data analysis and model training.
[0061] Based on the above embodiment, the target visual encoder is trained based on the mask images of each region and their mask labels, including: Performing data screening on each region mask image to obtain at least one first region mask image; Performing data enhancement on each first region mask image to obtain a second region mask image; The target visual encoder is trained based on each second region mask image and its mask label.
[0062] Specifically, the present application may first filter the region mask images with lower resolutions among the region mask images to obtain at least one third region mask image.
[0063] Furthermore, the false negative samples in each third region mask image are processed, and the remaining region mask images are respectively used as the first region mask images.
[0064] Furthermore, data enhancement can be performed on each first region mask image to simulate different shooting conditions and environmental changes, improve the adaptability of the model to various situations, and use the enhanced first region mask images as second region mask images.
[0065] Specifically, the present application can perform data enhancement on each first region mask image, including random cropping, horizontal flipping, color jittering, random noise addition, etc. In order to enhance the robustness of the model to occlusion and blur, occlusion simulation (such as adding random occlusion blocks) or Gaussian blur processing can also be performed on some images.
[0066] Furthermore, in order to ensure the reliability of the data, quality assessment and cleaning can also be carried out, including manual review and consistency check. For samples suspected of poor quality or incorrect labels, manual review is performed to correct inconsistent labels or remove abnormal samples.
[0067] In this way, a high-quality, balanced and diverse training dataset was constructed, laying a solid foundation for efficient training and performance improvement of the model.
[0068] Furthermore, the fine-tuned AM-RADIO model is trained using a data set consisting of each second region mask image and its mask label, and after the training is completed, a vehicle local feature encoder having the ability to recognize and retrieve local regions of vehicles in detection images is obtained.
[0069] By performing image filtering, this application can ensure the quality of the training data set and help improve the accuracy of model recognition; by performing data enhancement, the adaptability of the model to different situations can be increased, thereby improving the accuracy of model recognition.
[0070] Based on the above embodiment, data screening is performed on each region mask image to obtain at least one first region mask image, including: Eliminate the regional mask images whose resolution is less than a preset resolution threshold in each regional mask image to obtain at least one third regional mask image; Determine negative samples and anchor point samples in each third region mask image; Determine the feature similarity between each negative sample and the anchor sample; The negative samples whose feature similarity is greater than a preset similarity threshold are adjusted, and the remaining third region mask image is used as the first region mask image.
[0071] Specifically, in order to improve data quality, regional mask images can be filtered by image resolution during data screening. A minimum requirement for image resolution is set, for example, the width and height of the image are not less than 224 pixels. Regional mask images with resolutions lower than this threshold are removed from the data set because they may have poor feature extraction effects due to image blur or high noise, thereby obtaining a data set containing at least one third regional mask image and its mask label.
[0072] Furthermore, the pre-trained AM-RADIO model can be used to extract features from all image samples in the dataset (i.e., the third region mask image) to obtain a high-dimensional feature representation of each sample. .
[0073] Furthermore, for each image marked as a negative sample , calculate its corresponding anchor point sample The feature similarity is measured by cosine similarity, and the formula is: ; in, represents the vector dot product, Represents the bi-norm of a vector. If the calculated similarity is higher than the set threshold τ (for example, 0.8), it is considered that the negative sample is too close to the anchor sample in the feature space and may not be a true negative sample, that is, there is a "false negative sample". This may be caused by incorrect image annotation, confusion of sample categories, or other reasons. For the identified "false negative samples", manual review is performed, and after confirmation, they are removed from the negative sample set or re-annotated as positive samples, low-quality data is filtered to ensure the accuracy of the training data, and the remaining third region mask images are used as the first region mask images.
[0074] This application can ensure the quality of the training data set by performing image filtering, and construct a high-quality, balanced and diverse training data set, which lays a solid foundation for efficient training and performance improvement of the model and helps to improve the accuracy of model recognition.
[0075] Based on the above embodiment, the target visual encoder is trained based on each second region mask image and its mask label, including: Resizing and formatting the second region mask images respectively to obtain a target region mask image; Based on the mask images of each target area and its mask labels, the target visual encoder is trained using low-rank adaptive technology.
[0076] Specifically, the present application can perform standardization processing on each second region mask image, including adjusting the image size and converting it into a tensor format to adapt to the input requirements of the model.
[0077] We can further construct a dataset using the Triplet_dataset class based on the adjusted target region mask image and its mask label, and divide the data into a training set (80%) and an evaluation set (20%).
[0078] Furthermore, load the pre-trained fine-tuned AM-RADIO model and use the model weights of the specified version (e.g., radio_v2.1). Depending on the computing environment, set the model to run on multiple graphics processing units (GPUs) to fully utilize hardware resources.
[0079] The loss function uses a triplet loss function based on cosine similarity to measure the distance difference between the anchor sample and the positive and negative samples. The formula of the loss function is: ; in, and They represent the cosine distance between the anchor sample and the positive sample, and between the anchor sample and the negative sample, respectively. is the preset margin hyperparameter. The cosine distance d is calculated as: ; in, and is the feature vector of the corresponding sample, is the dot product operation, is the norm of the vector.
[0080] During the training process, this application applies low-rank adaptation (LoRA) technology to further improve training efficiency, and achieves efficient parameter adaptation effect by adding low-rank components to the weight matrix of the pre-trained model.
[0081] The specific approach is to introduce a low-rank matrix during fine-tuning. , decompose it into the product of two low-rank matrices ,in , ,and , only these low-rank matrices are optimized during training and , while keeping the original weight matrix unchanged, thereby reducing the number of parameters that need to be updated and significantly improving training efficiency.
[0082] This application also refers to the MASA method. We adopt a multi-scale feature fusion strategy to construct a multi-scale feature pyramid. Based on the AM-RADIO model, we extract feature maps of different levels to form a multi-scale feature pyramid structure. Shallow features are used to capture low-level texture and edge information, and deep features are used to capture high-level semantic information. Feature maps of different scales are aligned to the same spatial size, and then multi-scale features are fused through feature concatenation, weighted summation or attention mechanism to enhance local feature representation, so that the model can focus on local details and global context information at the same time, and improve the recognition and retrieval capabilities of local areas of vehicles.
[0083] MASA (Matching Anything by Segmenting Anything) is a novel method that leverages rich instance-level shape and appearance information from SAM (Segment Anything Model) to learn generalizable instance associations from unlabeled images.
[0084] In terms of preventing overfitting, this application introduces Dropout regularization in the newly added MLP layer, which reduces the risk of overfitting the model to the training data by randomly discarding neurons, and sets weight decay (L2 regularization) to constrain the model parameters to promote generalization ability. An early stopping strategy can also be adopted, that is, training is stopped when the performance of the validation set no longer improves in several consecutive training cycles, to prevent the model from continuing to fit the training set and reducing the generalization performance.
[0085] For the optimizer and learning rate strategy, the group parameter optimization method is adopted to set different learning rates for different parameter groups, such as using a higher learning rate for the newly added fine-tuning layer and a lower learning rate for the LoRA part, and using the AdamW optimizer, which improves the processing of weight decay based on Adam and is more suitable for the training of deep models such as Transformer. The learning rate scheduler uses Cosine Annealing or Warmup strategy to dynamically adjust the learning rate, thereby improving the stability and efficiency of training. Mixed precision training and distributed training have also been applied. The former uses half-precision floating point numbers (FP16) for training to reduce video memory usage and accelerate training, while the latter uses distributed data parallel strategies in a multi-GPU environment to increase training speed and support larger batch sizes. The gradual thawing strategy can gradually thaw some pre-trained layers according to the performance of the validation set in the later stage of training, and fine-tune with a small learning rate to further optimize the adaptability and performance of the model. The performance evaluation and tuning of the model runs through the entire training process. It is evaluated regularly on the validation set, including indicators such as accuracy, recall, mean Average Precision (mAP), and hyperparameters such as learning rate, batch size, and regularization coefficient are adjusted according to the results to optimize the model performance.
[0086] AdamW optimizer is an optimization algorithm widely used in deep learning. Transformer is a deep learning model architecture based on self-attention mechanism. Cosine annealing is a learning rate scheduling strategy. Warmup strategy is used in learning rate scheduling to gradually increase the learning rate at the beginning of training to avoid using too high a learning rate at the beginning, which may cause unstable model training.
[0087] These comprehensive model fine-tuning strategies fully leverage the pre-training advantages of the AM-RADIO model and make fine adjustments to the vehicle local feature encoding task, enabling the model to achieve significant performance improvements on specific tasks and reduce the consumption of training resources.
[0088] Figure 3 FIG. 3 is a flow chart of the vehicle local feature encoder training method provided in the embodiment of the present application. Figure 3 As shown, the overall process of this application includes the following steps 1-5.
[0089] Step 1: Data collection and preprocessing: Obtain images from vehicle re-identification (ReID) datasets (such as Veri-Wild), ensure image quality, and minimize background interference. Perform preliminary preprocessing such as resizing and format conversion on the images; Step 2: Dataset construction: Use vehicle area detector to detect salient areas of the vehicle (such as annual inspection labels, accessories, etc.) and output the detection frame position. Use SAM model to automatically segment the detection area; manually annotate complex areas. The data is organized by brand, model, year, etc., and LLAVA 1.5 is used to generate vehicle color labels; Step 3: Data quality enhancement: Use the AM-RADIO model to filter the data, calculate feature similarity, and remove "false negative samples". Set the image resolution threshold to remove samples below the standard. Perform data enhancement (such as random cropping, color jitter, etc.) to improve data diversity; Step 4: Model fine-tuning strategy: Load the pre-trained AM-RADIO model, freeze the original weights, and add 1-2 layers of MLP for fine-tuning. Use LoRA technology to fine-tune low-rank parameters. Introduce multi-scale feature fusion to improve local area expression capabilities; Step 5: Training process: Construct triplets of anchors, positive samples, and negative samples. Use triplet loss to optimize the model, reduce the distance between similar samples, and expand the distance between different samples. Use AdamW optimizer for training, and dynamically adjust the learning rate. Evaluate the model on the validation set, adjust the hyperparameters, and save the best model weights.
[0090] Figure 4 : is a schematic diagram of a data set construction process in a vehicle local feature encoder training method provided in an embodiment of the present application, such as Figure 4 As shown, the vehicle local feature encoder training method of the present application includes the following steps 1-6.
[0091] Step 1: Initial data collection: Obtain vehicle images from vehicle ReID datasets such as Veri-Wild, ensuring image quality and diversity; Step 2: Significant area detection: Use the vehicle area detector to process each vehicle image. Detect the significant local areas of the vehicle, such as annual inspection labels, pendants, tissue boxes, ornaments, etc. Step 3: Region segmentation and labeling: Automatic segmentation, using the SAM model to automatically segment the detected area and obtain the target mask. Manual labeling, for complex areas that cannot be automatically segmented (such as headlights, grilles, spraying, etc.), manual labeling; Step 4: Data organization and storage: Organize and store images and their labels by brand, model, year, vehicle ID, etc.; ensure that the data hierarchy is clear for subsequent retrieval and processing; Step 5: Label generation and integration: Use LLAVA 1.5 to generate vehicle color labels. Integrate all label information (region labels, detection box positions, colors, etc.) into a text file in the format of: brand-model-year-color; Step 6: Data quality enhancement: Filter low-quality samples and remove image samples with resolution below the set threshold. Remove samples with incorrect labels or poor quality. Perform data enhancement operations on images to improve the model's adaptability to different environments. Perform consistency checks to verify the consistency of labels and attribute information to ensure that the data is accurate and reliable.
[0092] The vehicle local feature encoder training device provided in the present application is described below. The vehicle local feature encoder training device described below and the vehicle local feature encoder training method described above can be referenced to each other.
[0093] Furthermore, the present application also provides a vehicle local feature encoder training device.
[0094] The vehicle local feature encoder training device comprises: A detection module, used to obtain at least one vehicle image, perform local area detection on each of the vehicle images, and obtain area information of the local area of the corresponding vehicle image; the area information includes the area position and the area part type; An acquisition module, used for acquiring a regional mask image and a mask label from the corresponding vehicle image based on each of the regional information; A training module is used to train a target visual encoder based on each of the area mask images and their mask labels to obtain a vehicle local feature encoder; wherein the target visual encoder is obtained by adding a multi-layer perceptron to a pre-trained visual encoder; the pre-trained visual encoder is used for local area detection of the vehicle.
[0095] The vehicle local feature encoder training device of the present application obtains at least one vehicle image, performs local area detection on each vehicle image, obtains the regional information of the local area of the corresponding vehicle image, wherein the regional information includes the regional position and the regional part type; and further obtains the regional mask image and the mask label from the corresponding vehicle image based on each regional information; and, adds a multi-layer perceptron to the pre-trained visual encoder for local area detection of the vehicle to obtain the target visual encoder, thereby training the target visual encoder based on each regional mask image and its mask label to obtain the vehicle local feature encoder. By constructing high-quality training data including the local area mask image of the vehicle and its mask label, and training the target visual encoder obtained by fine-tuning the pre-trained visual encoder with the high-quality training data, a feature representation with both descriptive power and discriminative power can be obtained, so that the trained vehicle local feature encoder can accurately distinguish subtle local feature differences, meet the needs of high-precision local feature extraction and matching, and thus improve the retrieval accuracy of vehicle local features.
[0096] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the following method: acquiring at least one vehicle image, performing local area detection on each of the vehicle images, and obtaining the area information of the local area of the corresponding vehicle image; the area information includes the area position and the area part type; Based on the area information, respectively obtain area mask images and mask labels from corresponding vehicle images; Based on the mask images of each region and their mask labels, the target visual encoder is trained to obtain a vehicle local feature encoder; wherein the target visual encoder is obtained by adding a multi-layer perceptron to the pre-trained visual encoder; the pre-trained visual encoder is used for local area detection of the vehicle.
[0097] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0098] In another aspect, the embodiment of the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method provided by the above embodiments is implemented, for example, including: obtaining at least one vehicle image, performing local area detection on each of the vehicle images, and obtaining area information of the local area of the corresponding vehicle image; the area information includes the area position and the area part type; Based on the area information, respectively obtain area mask images and mask labels from corresponding vehicle images; Based on the mask images of each region and their mask labels, the target visual encoder is trained to obtain a vehicle local feature encoder; wherein the target visual encoder is obtained by adding a multi-layer perceptron to the pre-trained visual encoder; the pre-trained visual encoder is used for local area detection of the vehicle.
[0099] In another aspect, the present application also provides a computer program product having a computer program stored thereon, and when the computer program is executed by a processor, the method provided by the above embodiments is implemented, for example, including: obtaining at least one vehicle image, performing local area detection on each of the vehicle images, and obtaining area information of the local area of the corresponding vehicle image; the area information includes the area position and the area part type; Based on the area information, respectively obtain area mask images and mask labels from corresponding vehicle images; Based on the mask images of each region and their mask labels, the target visual encoder is trained to obtain a vehicle local feature encoder; wherein the target visual encoder is obtained by adding a multi-layer perceptron to the pre-trained visual encoder; the pre-trained visual encoder is used for local area detection of the vehicle.
[0100] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0102] Finally, it should be noted that the above implementation modes are only used to illustrate the present application, rather than to limit the present application. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that various combinations, modifications or equivalent substitutions of the technical solutions of the present application do not depart from the spirit and scope of the technical solutions of the present application.
Claims
1. A vehicle local feature encoder training method, characterized in that: include: Acquire at least one vehicle image, perform local area detection on each of the vehicle images, and obtain area information of the local area of the corresponding vehicle image; The regional information includes the regional location and the regional part type; Based on the area information, respectively obtain area mask images and mask labels from corresponding vehicle images; Based on the mask images of each region and their mask labels, the target visual encoder is trained to obtain a vehicle local feature encoder; wherein the target visual encoder is obtained by adding a multi-layer perceptron to the pre-trained visual encoder; the pre-trained visual encoder is used for local area detection of the vehicle.
2. The vehicle local feature encoder training method according to claim 1, characterized in that: The training of the target visual encoder based on each of the region mask images and the mask labels thereof includes: Performing data screening on each of the regional mask images to obtain at least one first regional mask image; Performing data enhancement on each of the first region mask images to obtain a second region mask image; Based on each of the second region mask images and their mask labels, a target visual encoder is trained.
3. The vehicle local feature encoder training method according to claim 2, characterized in that: The step of performing data screening on each of the region mask images to obtain at least one first region mask image includes: Eliminating the regional mask images whose resolution is less than a preset resolution threshold from among the regional mask images, to obtain at least one third regional mask image; Determining negative samples and anchor point samples in each of the third region mask images; Determine the feature similarity between each negative sample and the anchor sample; The negative samples whose feature similarity is greater than a preset similarity threshold are adjusted, and the remaining third region mask image is used as the first region mask image.
4. The vehicle local feature encoder training method according to claim 2, characterized in that: The training of the target visual encoder based on each of the second region mask images and the mask labels thereof includes: Resizing and formatting each of the second region mask images respectively to obtain a target region mask image; Based on the target area mask images and their mask labels, a low-rank adaptive technique is applied to train the target visual encoder.
5. The vehicle local feature encoder training method according to claim 4, characterized in that: The loss function of the target visual encoder is a triplet loss function based on cosine similarity.
6. The vehicle local feature encoder training method according to claim 1, characterized in that: The acquiring of the regional mask image and the mask label from the corresponding vehicle image based on the regional information respectively includes: For each vehicle image, the vehicle image and its region information are input into the image segmentation model to obtain the region mask image and mask label output by the image segmentation model.
7. A vehicle local feature encoder training device, characterized in that: include: A detection module, used to obtain at least one vehicle image, perform local area detection on each of the vehicle images, and obtain area information of the local area of the corresponding vehicle image; the area information includes the area position and the area part type; An acquisition module, used for acquiring a regional mask image and a mask label from the corresponding vehicle image based on each of the regional information; A training module is used to train a target visual encoder based on each of the area mask images and their mask labels to obtain a vehicle local feature encoder; wherein the target visual encoder is obtained by adding a multi-layer perceptron to a pre-trained visual encoder; the pre-trained visual encoder is used for local area detection of the vehicle.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the vehicle local feature encoder training method as described in any one of claims 1-6 is implemented.
9. A storage medium, the storage medium being a non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that: When the computer program is executed by a processor, the vehicle local feature encoder training method as described in any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the vehicle local feature encoder training method according to any one of claims 1 to 6 is implemented.