Visual recognition method and device based on local region adaptive training enhancement

Abstract

This application relates to the field of visual recognition technology, and provides a visual recognition method and apparatus based on local domain adaptive training enhancement. The method involves: annotating the pose of acquired local domain reference images; extracting the first global features of each reference image using a pre-trained visual model; determining local domain training samples based on the first global features and pose of each reference image; training the pre-trained model using the training samples to obtain a triplet loss function; fine-tuning the pre-trained model based on the training samples to determine the mean squared error loss function; weighted summing of the triplet loss function and the mean squared error loss function to obtain the total loss function; training the pre-trained model using the loss function; and then using the trained visual recognition model to recognize target images containing local domain features. This method achieves low-cost construction of a training dataset, solving the problem of the large workload of reconstructing a local domain dataset. Simultaneously, the trained model exhibits good discriminative ability for local features, improving recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of visual recognition technology, and in particular to a visual recognition method and apparatus based on local adaptive training enhancement. Background Technology

[0002] Utilizing large-scale street view visual recognition and localization technology to obtain accurate location information has become an important application scenario in the field of visual localization. Traditional image descriptor-based visual localization methods use manual descriptors, which are highly practical when computing resources are limited, but their performance is limited when dealing with complex backgrounds, lighting conditions, and significant seasonal variations in perspective. With the development of deep learning technology, network feature descriptors extracted based on Convolutional Neural Networks (CNNs) and Transformer techniques have shown significant effects in visual image matching. Furthermore, their end-to-end design facilitates deployment on various types of payload devices, truly applying large-scale street view visual localization technology to real-world scenarios.

[0003] Street view image retrieval has great application value in the field of visual positioning. Global matching of street view images mainly depends on the performance of the global image feature extractor. A global image feature extractor is obtained by training a large number of samples under different lighting seasons over a long period of time. Global matching of images is then performed based on the similarity of global features.

[0004] In large-scale street scene visual matching and localization frameworks, global image feature matching plays a decisive role in the entire localization process. Existing global image feature extractors typically consist of an end-to-end deep neural network. However, in different application scenarios, the discriminative power of large-scale training datasets for some features may decrease, posing a series of challenges to street scene image recognition and retrieval.

[0005] When the region to be identified originates from different countries and regions, there may be feature differences in different dimensions within this region; these features are called local features. Pre-trained feature extraction models are not sufficiently discriminative of these local features, which causes the performance of global image feature extractors to degrade in this region.

[0006] To address the issue of reduced discriminative power of local features, it is necessary to reconstruct a local training dataset and mix it into the previous training dataset for retraining or fine-tuning. However, constructing a long-term sample dataset is an extremely costly task.

[0007] To address the issue of reduced discriminative power in local domains, the feature extraction network model needs to be retrained. The core challenge of this incremental learning is how to avoid catastrophic forgetting when learning new features, as this can cause the model to shift and lead to a significant drop in performance.

[0008] Meanwhile, most current domain adaptation problems focus on semantic segmentation and image classification, with very little research on image retrieval. The few studies on image retrieval that exist involve adjustments made through multiple loops using Simultaneous Localization and Mapping (SLAM), and there is almost no discussion on visual location recognition retrieval tasks.

[0009] Therefore, how to provide an efficient visual recognition model for images containing specific local domains is a technical problem that needs to be solved. Summary of the Invention

[0010] In view of this, embodiments of this application provide a visual recognition method and apparatus based on local adaptive training enhancement to solve the problem in the prior art that it is difficult to fine-tune the visual recognition model in real time and efficiently, so that it can accurately achieve visual recognition in local domain images containing local features.

[0011] A first aspect of this application provides a visual recognition method based on local domain adaptive training enhancement, comprising:

[0012] Acquire the local reference image and perform pose annotation on the local reference image;

[0013] A pre-trained visual recognition model is used to extract global features from each image in the local benchmark image to obtain the first global features of each benchmark image.

[0014] Local training samples are constructed based on the first global features and poses of each benchmark image. A pre-trained visual recognition model is trained based on the local training samples to obtain the triplet loss function.

[0015] The pre-trained visual recognition model is fine-tuned based on local training samples, and the fine-tuned pre-trained visual recognition model is used to extract global features from each image in the local benchmark image to obtain the second global features of each benchmark image.

[0016] The mean squared error loss function is determined based on the similarity between the first global feature and the second global feature.

[0017] The total loss function is obtained by weighted summation of the triplet loss function and the mean squared error loss function, and a pre-trained visual recognition model is trained based on the total loss function to obtain the trained visual recognition model.

[0018] Acquire the target image to be identified, and use the trained visual recognition model to identify the target image; the target image includes local features.

[0019] A second aspect of this application provides a visual recognition device based on local domain adaptive training enhancement, comprising:

[0020] The acquisition module is configured to acquire a local reference image and perform pose annotation on the local reference image;

[0021] The feature extraction module is configured to use a pre-trained visual recognition model to perform global feature extraction on each image in the local domain benchmark image to obtain the first global features of each benchmark image.

[0022] The training module is configured to construct local training samples based on the first global features and poses of each benchmark image, train a pre-trained visual recognition model based on the local training samples, and obtain a triplet loss function.

[0023] The feature extraction module is also configured to fine-tune the pre-trained visual recognition model based on local training samples, and use the fine-tuned pre-trained visual recognition model to perform global feature extraction on each image in the local benchmark image to obtain the second global features of each benchmark image.

[0024] The training module is also configured to determine the mean squared error loss function based on the similarity between the first global feature and the second global feature;

[0025] The training module is also configured to obtain a total loss function by weighted summation of the triplet loss function and the mean squared error loss function, and to train a pre-trained visual recognition model based on the total loss function to obtain a trained visual recognition model.

[0026] The recognition module is configured to acquire the target image to be recognized and recognize the target image using a trained visual recognition model; the target image includes local features.

[0027] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0028] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0029] The beneficial effects of the embodiments in this application compared with the prior art are:

[0030] This application embodiment performs pose annotation on the acquired local domain reference images, extracts the first global features of each reference image using a pre-trained visual model, determines local domain training samples based on the first global features and pose of each reference image, trains a pre-trained visual recognition model using these local domain training samples to obtain a triplet loss function, fine-tunes the pre-trained visual recognition model based on these local domain training samples to determine the mean squared error loss function, and obtains the total loss function by weighted summation of the triplet loss function and the mean squared error loss function. The loss function is then used to train the pre-trained visual recognition model, and the trained visual recognition model is used to recognize target images containing local domain features. This achieves low-cost construction of a training dataset, solves the problem of the large workload of rebuilding a local domain dataset, and improves the recognition accuracy by providing a model with good discriminative power for local features. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating a visual recognition method based on local adaptive training enhancement provided in an embodiment of this application.

[0033] Figure 2 This is a flowchart illustrating the method for constructing local training samples based on the first global features and poses of each reference image, as provided in an embodiment of this application.

[0034] Figure 3 This is a flowchart illustrating another visual recognition method based on local adaptive training enhancement provided in this application embodiment.

[0035] Figure 4 This is a schematic diagram of a visual recognition device based on local adaptive training enhancement provided in an embodiment of this application.

[0036] Figure 5 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0037] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0038] The following describes in detail, with reference to the accompanying drawings, a visual recognition method and apparatus based on local domain adaptive training enhancement according to embodiments of this application.

[0039] As mentioned above, in the large-scale street view visual matching localization framework, global image feature matching plays a decisive role in the entire localization process.

[0040] In the current large-scale street scene visual matching and localization framework, a baseline image database is first constructed offline for the region to be identified. For each baseline image, two parts are performed: global feature extraction and local feature extraction. Global feature extraction and matching significantly narrows the scope of n-point perspective (PnP) localization by image local feature matching, improving the efficiency of visual localization while maintaining accuracy. For local features, multiple significant local feature points are extracted from an image, and corresponding 3D point cloud models are built and stored as the baseline image database. In the visual recognition stage, global and local feature extraction is performed online on the captured images. First, the global features of the captured image are matched with the global features of all baseline images. Matching is performed based on the similarity between the vectors corresponding to each image, and the geographical locations of the images with high similarity are designated as the subsequent local matching and localization regions. The 3D point cloud of the selected local matching and localization region is matched with the local feature point cloud of the image to be identified for feature point matching and outlier removal, and PnP is used to calculate the accurate local 3D pose.

[0041] In other words, in the process of street view visual image matching and retrieval, the high-dimensional vector representing the global features of each image is extracted by the model to represent the global descriptor of the image, and the matching relationship between images is determined by the similarity between the high-dimensional vectors.

[0042] Therefore, global feature matching plays a decisive role in the entire localization process. If the global feature matching is correct, the location for local feature matching will be locked, and the final localization result will be concentrated in this area. Conversely, if the global feature matching is incorrect, local feature matching will fail to find the correct area, thus preventing the calculation of the correct location. This demonstrates the significant impact of the accuracy of global feature matching on the overall localization accuracy.

[0043] Existing global image feature extractors typically consist of an end-to-end deep neural network. Once trained on a large-scale training dataset, this global image feature extractor will be used for global feature extraction of images from other regions. Training this global image feature extractor requires constructing a large-scale training dataset, which must contain the following characteristics: (1) image sample pairs across viewpoints, seasons, and lighting conditions, requiring the collection of images under different conditions at the same location to construct training sample pairs; (2) a large-scale image set covering multiple cities and countries to ensure feature diversity and support the training of large deep learning models. After training with a training set containing the above characteristics, a global image feature extractor with fixed parameters is obtained, which will be used for subsequent feature extraction. The global image feature extractor trained using this method is difficult to adapt to the retrieval and recognition of images containing local domain features.

[0044] In view of this, this application provides a visual recognition method based on local adaptive training enhancement. The method involves: annotating the acquired local reference images with poses; extracting the first global features of each reference image using a pre-trained visual model; determining local training samples based on the first global features and poses of each reference image; training a pre-trained visual recognition model using these local training samples to obtain a triplet loss function; fine-tuning the pre-trained visual recognition model based on these local training samples to determine the mean squared error loss function; weighted summing of the triplet loss function and the mean squared error loss function to obtain the total loss function; training the pre-trained visual recognition model using the loss function; and then using the trained visual recognition model to recognize target images containing local features. This method achieves low-cost construction of a training dataset, solving the problem of the large workload of rebuilding a local dataset. Furthermore, the trained model exhibits good discriminative power for local features, improving recognition accuracy.

[0045] Figure 1 This is a flowchart illustrating a visual recognition method based on local adaptive training enhancement provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0046] In step S101, a local reference image is acquired, and pose annotation is performed on the local reference image.

[0047] In step S102, a pre-trained visual recognition model is used to extract global features from each image in the local reference image to obtain the first global features of each reference image.

[0048] In step S103, local training samples are constructed based on the first global features and poses of each reference image, and a pre-trained visual recognition model is trained based on the local training samples to obtain the triplet loss function.

[0049] In step S104, the pre-trained visual recognition model is fine-tuned based on the local training samples, and the fine-tuned pre-trained visual recognition model is used to extract global features from each image in the local benchmark image to obtain the second global features of each benchmark image.

[0050] In step S105, the mean squared error loss function is determined based on the similarity between the first global feature and the second global feature.

[0051] In step S106, the total loss function is obtained by weighted summation of the triplet loss function and the mean squared error loss function, and the pre-trained visual recognition model is trained based on the total loss function to obtain the trained visual recognition model.

[0052] In step S107, the target image to be identified is acquired, and the trained visual recognition model is used to identify the target image.

[0053] The target image includes local domain features.

[0054] In some embodiments of this application, the method may be executed by a server or by a terminal device with certain processing capabilities.

[0055] In some embodiments of this application, a local reference image may be obtained first, and the pose of the local reference image may be annotated.

[0056] In one example, the local reference image can be a dense image acquired within the local scene, such as a dense street view image. The local reference image can be denoted as... , This represents the number of images in the local reference image.

[0057] Pose annotation of local reference images can involve labeling each reference image in the local reference image with its coordinates and shooting orientation. Each annotated reference image can contain geographical location information and orientation.

[0058] In some embodiments of this application, a pre-trained visual recognition model can be used to extract global features from each image in the local reference image to obtain the first global features of each reference image.

[0059] The pre-trained visual recognition model can be trained using long-term data including similar scenes in the local area, meaning it has a certain performance in visual matching tasks within the local area. Therefore, the initial knowledge carried by the parameters of this pre-trained visual recognition model can be used, combined with the geographical location information and orientation of each reference image, to establish the association between the reference images, thereby constructing local training samples.

[0060] In one example, the pre-trained visual recognition model can be denoted as... The process of extracting the first global features of each benchmark image using a pre-trained visual recognition model can be represented as follows: .

[0061] In some embodiments of this application, local training samples can be constructed based on the first global features and pose of each reference image, and a pre-trained visual recognition model can be trained based on the local training samples to obtain a triplet loss function.

[0062] In some embodiments of this application, the pre-trained visual recognition model can be fine-tuned based on local training samples, and the fine-tuned pre-trained visual recognition model can be used to extract global features from each image in the local benchmark image to obtain the second global features of each benchmark image.

[0063] In some embodiments of this application, the mean squared error loss function is determined based on the similarity between the first global feature and the second global feature, and then the total loss function is obtained by weighted summation of the triplet loss function and the mean squared error loss function.

[0064] In some implementations, a pre-trained visual recognition model can be trained based on the total loss function to obtain a trained visual recognition model. Finally, the target image to be recognized is obtained, and the trained visual recognition model is used to identify target images in the image set that contain local features.

[0065] According to the technical solution provided in the embodiments of this application, pose annotation is performed on the acquired local reference images, the first global features of each reference image are extracted using a pre-trained visual model, local training samples are determined based on the first global features and pose of each reference image, the pre-trained visual recognition model is trained using the local training samples to obtain the triplet loss function, and the pre-trained visual recognition model is fine-tuned based on the local training samples to determine the mean squared error loss function. The triplet loss function and the mean squared error loss function are weighted and summed to obtain the total loss function, the pre-trained visual recognition model is trained using the loss function, and then the trained visual recognition model is used to recognize target images containing local features. This achieves the construction of a training dataset at low cost, solves the problem of large workload in rebuilding the local dataset, and at the same time, the trained model has good discriminative power for local features, improving the recognition accuracy.

[0066] Figure 2 This is a flowchart illustrating the method for constructing local training samples based on the first global features and poses of each reference image, as provided in an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0067] In step S201, the target reference image is acquired.

[0068] The target reference image is any image from the local reference image.

[0069] In step S202, N matching reference images of the target reference image are determined.

[0070] Among them, the N matching reference images are the top N reference images with the highest similarity between the first global feature and the first global feature of the target reference image, where N is a positive integer.

[0071] In step S203, the matching reference images that meet the preset pose judgment conditions among the N matching reference images are determined as positive sample images, and the remaining matching reference images are negative sample images.

[0072] In step S204, a first distance is set between the positive sample image and the target reference image, and a second distance is set between the negative sample image and the target reference image.

[0073] The first distance is less than the second distance.

[0074] In step S205, the local domain reference images are traversed to obtain local domain training samples.

[0075] In some embodiments of this application, when constructing local training samples, a target reference image can be obtained first. This target reference image is any image from the local reference images. In one example, the target reference image can be denoted as... .

[0076] N matching reference images can be determined for the target reference image. These N matching reference images are the top N reference images whose first global feature has the highest similarity to the first global feature of the target reference image.

[0077] In other words, the target reference image can be The first global feature is matched with the first global features of other images in the local reference image for similarity matching. This yields the similarity relationships between images. The cosine similarity function is used. For image The corresponding vector, For image The corresponding vector.

[0078] The top N images with the highest similarity can be identified as the target reference images. Matching reference image .

[0079] In some embodiments of this application, the matching reference images that meet the preset pose judgment conditions among the N matching reference images can be determined as positive sample images, and the remaining matching reference images are negative sample images.

[0080] The preset pose determination conditions may be that the absolute value of the difference between the geographic location information of the target reference image and the geographic location information of the matching reference image is less than or equal to a preset distance threshold; and the absolute value of the difference between the orientation of the target reference image and the orientation of the matching reference image is less than or equal to a preset orientation threshold.

[0081] In other words, the preset pose determination condition can be expressed as: ,and ,in For image Geographical location information, For image Geographical location information, The preset distance threshold; For image Orientation, For image Orientation, Preset orientation threshold; .

[0082] and The specific value can be set according to actual needs. In one example, it can be set to... It is 25 meters (m). It is 45 degrees.

[0083] In some embodiments of this application, a first distance can be set between the positive sample image and the target reference image, and a second distance can be set between the negative sample image and the target reference image; wherein the first distance is less than the second distance.

[0084] Since the positive and negative sample images mentioned above are derived entirely from the parameters of the pre-trained visual recognition model, in order to ensure the randomness of training, in some implementations, M sample images that satisfy the condition of a distance greater than the second distance can be randomly selected from the remaining images as negative samples.

[0085] In other words, the negative samples of the target reference image can also include M redundant reference images, the distance between which is greater than the second distance to the target reference image. These redundant reference images are randomly determined from local reference images other than the target reference image and the matching reference image, where M is a positive integer less than N.

[0086] The values ​​of M and N can also be determined according to actual needs. In one example, N can be set to 10 and M to 3.

[0087] In some embodiments of this application, if the target reference image does not have positive sample images and M negative sample images that meet the preset pose judgment conditions, the target reference image can be removed and marked as a training image, and will not be used as a training sample pair for training.

[0088] Traversing the local reference image yields a number of less than [number]. The local training samples.

[0089] In some embodiments of this application, the triplet loss function can be expressed using the formula... Construct; among which, The triplet loss function, The function is a function that finds the maximum value. This represents the global feature similarity distance between two images. For the target reference image, For the positive sample image of the target reference image, The negative sample image of the target reference image. It is a preset hard margin used to ensure that there is a certain similarity gap between positive and negative sample images. In one example, The value can be set to 0.1.

[0090] Through the steps described above, the pre-trained visual recognition model can learn to acquire discriminative capabilities for visual retrieval and localization, retrieving the desired image and placing seemingly similar but actually distant images at greater distances to improve differentiation. The parameters of the pre-trained visual recognition model are also obtained through this network approach.

[0091] However, in order for the network to learn parameters that contain discriminative properties of the current local domain features when only local domain reference images are used as training data, other learning mechanisms need to be introduced. Furthermore, useful discriminative information needs to be retained for the identification of regular features. Therefore, it is necessary to consider introducing other mechanisms to learn the network.

[0092] In this application embodiment, the Mean Squared Error (MSE) loss function can be selected as another mechanism, such as the one described above.

[0093] On the other hand, since pre-trained visual recognition models typically share certain commonalities in their ability to discriminate basic image features, and this discriminative knowledge of basic image features usually resides in the shallow layers of the neural network, and this feature knowledge is typically acquired through training with a large number of carefully labeled image training samples, in order to prevent this basic knowledge from being corrupted by training on a self-supervised training dataset of local benchmark images, incremental learning training of the pre-trained visual recognition model can be performed using a local fine-tuning approach.

[0094] Local fine-tuning can involve only opening the last m layers of the network for fine-tuning training to ensure that the model retains its original knowledge while learning new feature discrimination knowledge, thus preventing catastrophic forgetting. The value of m can be set according to actual needs; in one example, m can be equal to 3 to 5.

[0095] In some embodiments of this application, the MSE loss function stabilizes the network by calculating the similarity change between the global features extracted by the fine-tuned current model network and the global features extracted by the original model network, ensuring that the original knowledge is not forgotten. Because the fine-tuned current model possesses a large amount of knowledge about image feature matching, training directly using local benchmark images lacking long-term information would inevitably lead to the model's inability to generate effective knowledge extraction capabilities. Therefore, to preserve the large amount of knowledge related to image feature matching, an MSE loss function preservation mechanism is required. The loss function is set to calculate the similarity between two images, using the triplet loss function while simultaneously using the MSE loss function to preserve the network's original knowledge.

[0096] The MSE loss function can be the L2 norm of the second global feature and the first global feature. In other words, ,in, The MSE loss function is... This is the second global feature, which is the global feature extracted using the fine-tuned pre-trained visual recognition model. The first global feature is the global feature extracted by the pre-trained visual recognition model.

[0097] When training the pre-trained visual recognition model based on the total loss function, since the model is incrementally learned based on the pre-trained model, it is usually fine-tuned based on a large-scale pre-trained model. The pre-trained model retains a significant amount of knowledge about the model. Typically, we train the model using local benchmark images for one epoch, that is, we iterate through all the training sample pairs consisting of all the benchmark images. However, to ensure a smoother transition and more reasonable knowledge transfer, the entire local benchmark image can be divided into multiple components based on quantity, and the model parameters can be updated using the Exponential Moving Average (EMA) update mechanism.

[0098] In other words, in some embodiments of this application, when training the pre-trained visual recognition model based on the total loss function, the model parameters of the pre-trained visual recognition model can be updated using the EMA update mechanism.

[0099] The EMA update mechanism can be represented as follows: ,in These are the model parameters after this round of training. These are the model parameters from the previous moment, i.e., before this round of training. As the attenuation factor, ; The larger the value, the smoother the EMA updates, but the slower the EMA updates and the greater the historical impact. The smaller the value, the more sensitive it is; the faster the EMA updates, the better it follows the latest data.

[0100] After dividing the local baseline image into several components, each component is trained sequentially. After training each component, EMA momentum updates are used to ensure a smooth transition of model parameters and guarantee perfect knowledge inheritance. The obtained model parameters are then used... The remaining group training data are used sequentially, and after all groups of data have been trained and fine-tuned once, the final parameters of the network are obtained.

[0101] The trained visual recognition model can be used to extract global features from the local benchmark image, obtain its image descriptor, and store it in the local image descriptor database.

[0102] The technical solution provided in the embodiments of this application can be verified in the following way: the captured target image to be identified is input into the trained visual recognition model for feature extraction, the extracted features are compared with each image descriptor in the entire local image descriptor database for cosine similarity calculation, and then the reference image matching the target image is determined based on the similarity ranking result.

[0103] Figure 3This is a flowchart illustrating another visual recognition method based on local adaptive training enhancement provided in an embodiment of this application. Figure 3 As shown, a local reference image can be acquired first and its pose labeled. Then, a pre-trained model is used to extract global features from the reference image to obtain its image descriptor. Reference image components can be matched against each other based on the image descriptor, and positive and negative sample images can be selected using geographic coordinates based on the matching results.

[0104] If the selected positive and negative sample images meet the criteria for forming a training sample pair, then this sample can be added to the training dataset; otherwise, this baseline image sample is discarded, and the next baseline image sample is selected. Alternatively, redundant negative samples can be added to the training samples, and then all samples are checked again to see if they meet the sample construction criteria. If they do, the sample pair is added to the local training sample set; otherwise, this baseline image sample is discarded, and the next baseline image sample is selected.

[0105] After completing the construction of the training samples, the training sample pairs can be grouped. Each group of training samples is trained using the triplet loss function and the MSE loss function, and the network parameters are updated using EMA during training. The training operation is performed iteratively until all training sample groups have been traversed.

[0106] Finally, the trained model is used to extract features from the similarity data, resulting in an updated image descriptor for the baseline image. This image descriptor can then be saved to a database. When a target image is input into the trained model, its global features can be extracted to obtain an image descriptor. This image descriptor can then be used to retrieve a matching baseline image from the database.

[0107] The technical solution provided in this application proposes an efficient and reliable local adaptive training-enhanced local image matching method, which improves the performance indicators of visual recognition and localization in proprietary regions with local features and enhances the user experience.

[0108] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0109] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0110] Figure 4 This is a schematic diagram of a visual recognition device based on local adaptive training enhancement provided in an embodiment of this application. Figure 4 As shown, the device includes:

[0111] The acquisition module 401 is configured to acquire a local reference image and perform pose annotation on the local reference image.

[0112] The feature extraction module 402 is configured to use a pre-trained visual recognition model to perform global feature extraction on each image in the local reference image to obtain the first global feature of each reference image.

[0113] Training module 403 is configured to construct local training samples based on the first global features and poses of each benchmark image, train a pre-trained visual recognition model based on the local training samples, and obtain a triplet loss function.

[0114] The feature extraction module 402 is also configured to fine-tune the pre-trained visual recognition model based on local training samples, and use the fine-tuned pre-trained visual recognition model to perform global feature extraction on each image in the local benchmark image to obtain the second global features of each benchmark image.

[0115] The training module 403 is also configured to determine the mean squared error loss function based on the similarity between the first global feature and the second global feature.

[0116] The training module 403 is also configured to obtain a total loss function by weighted summation of the triplet loss function and the mean squared error loss function, and to train a pre-trained visual recognition model based on the total loss function to obtain a trained visual recognition model.

[0117] The recognition module 404 is configured to acquire the target image to be recognized and recognize the target image using a trained visual recognition model; the target image includes local features.

[0118] According to the technical solution provided in the embodiments of this application, pose annotation is performed on the acquired local reference images, the first global features of each reference image are extracted using a pre-trained visual model, local training samples are determined based on the first global features and pose of each reference image, the pre-trained visual recognition model is trained using the local training samples to obtain the triplet loss function, and the pre-trained visual recognition model is fine-tuned based on the local training samples to determine the mean squared error loss function. The triplet loss function and the mean squared error loss function are weighted and summed to obtain the total loss function, the pre-trained visual recognition model is trained using the loss function, and then the trained visual recognition model is used to recognize target images containing local features. This achieves the construction of a training dataset at low cost, solves the problem of large workload in rebuilding the local dataset, and at the same time, the trained model has good discriminative power for local features, improving the recognition accuracy.

[0119] In some implementations, constructing local training samples based on the first global features and poses of each reference image includes: acquiring a target reference image; the target reference image is any image in the local reference images; determining N matching reference images of the target reference image; wherein, the N matching reference images are the top N reference images with the highest similarity between the first global features and the first global features of the target reference image, and N is a positive integer; determining that the matching reference images among the N matching reference images that meet the preset pose judgment conditions are positive sample images, and the remaining matching reference images are negative sample images; setting a first distance between the positive sample images and the target reference image, and setting a second distance between the negative sample images and the target reference image; wherein, the first distance is less than the second distance; traversing the local reference images to obtain local training samples.

[0120] In some implementations, the preset pose determination conditions are: the absolute value of the difference between the geographic location information of the target reference image and the geographic location information of the matching reference image is less than or equal to a preset distance threshold; and the absolute value of the difference between the orientation of the target reference image and the orientation of the matching reference image is less than or equal to a preset orientation threshold.

[0121] In some implementations, the negative samples of the target reference image also include M redundant reference images; the distance between the redundant reference images and the target reference image is greater than a second distance; the redundant reference images are randomly determined from local reference images other than the target reference image and the matching reference image, where M is a positive integer less than N.

[0122] In some implementations, the triplet loss function is constructed as follows: using the formula Determine the triplet loss function; where, The triplet loss function, The function is a function that finds the maximum value. This represents the global feature similarity distance between two images. For the target reference image, For the positive sample image of the target reference image, The negative sample image of the target reference image. It is a preset hard interval.

[0123] In some implementations, the mean squared error loss function is constructed as follows: the L2 norm of the second global feature and the first global feature is determined as the mean squared error loss function.

[0124] In some implementations, when training a pre-trained visual recognition model based on the total loss function, an exponential moving average update mechanism is used to update the model parameters of the pre-trained visual recognition model.

[0125] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0126] Figure 5 This is a schematic diagram of the electronic device provided in an embodiment of this application. For example... Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the steps in the various method embodiments described above. Alternatively, when the processor 501 executes the computer program 503, it implements the functions of each module / unit in the various device embodiments described above.

[0127] Electronic device 5 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 5 may include, but is not limited to, processor 501 and memory 502. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 5 and does not constitute a limitation on electronic device 5. It may include more or fewer components than shown, or different components.

[0128] The processor 501 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0129] The memory 502 can be an internal storage unit of the electronic device 5, such as a hard disk or RAM of the electronic device 5. The memory 502 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 5. The memory 502 can also include both internal and external storage units of the electronic device 5. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0130] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0131] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0132] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A visual recognition method based on local domain self-adaptive training enhancement, characterized in that, include: Acquire a local reference image and perform pose annotation on the local reference image; A pre-trained visual recognition model is used to extract global features from each image in the local domain reference image to obtain the first global features of each reference image. Local training samples are constructed based on the first global features and poses of each reference image. The pre-trained visual recognition model is trained based on the local training samples to obtain the triplet loss function. Based on the local training samples, the pre-trained visual recognition model is fine-tuned, and the fine-tuned pre-trained visual recognition model is used to extract global features from each image in the local benchmark image to obtain the second global features of each benchmark image. The mean squared error loss function is determined based on the similarity between the first global feature and the second global feature; The total loss function is obtained by weighted summation of the triplet loss function and the mean squared error loss function, and the pre-trained visual recognition model is trained based on the total loss function to obtain the trained visual recognition model. Acquire the target image to be identified, and use the trained visual recognition model to identify the target image; The target image includes local domain features; Among them, local training samples are constructed based on the first global features and pose of each benchmark image, including: Obtain the target reference image; the target reference image is any image from the local reference images; N matching reference images are determined for the target reference image; wherein, the N matching reference images are the top N reference images whose first global feature has the highest similarity to the first global feature of the target reference image, and N is a positive integer; Among the N matching reference images, the matching reference images that meet the preset pose judgment conditions are determined as positive sample images, and the remaining matching reference images are determined as negative sample images. A first distance is set between the positive sample image and the target reference image, and a second distance is set between the negative sample image and the target reference image; wherein the first distance is less than the second distance; The local domain reference images are traversed to obtain the local domain training samples; The preset pose determination condition is: The absolute value of the difference between the geographic location information of the target reference image and the geographic location information of the matching reference image is less than or equal to a preset distance threshold; and, The absolute value of the difference between the orientation of the target reference image and the orientation of the matching reference image is less than or equal to a preset orientation threshold.

2. The method of claim 1, wherein, The negative samples of the target reference image also include M redundant reference images; The distance between the redundant reference image and the target reference image is greater than the second distance; The redundant reference image is randomly determined from local domain reference images other than the target reference image and the matching reference image, where M is a positive integer less than N.

3. The method of claim 1, wherein, The triplet loss function is constructed as follows: Using the formula determining the triplet loss function; wherein, is the triplet loss function, is a max function, represents a global feature similarity distance of two images, is the target reference image, is a positive sample image of the target reference image, is a negative sample image of the target reference image, is a preset hard margin.

4. The method of claim 1, wherein, The mean squared error loss function is constructed as follows: The L2 norm of the second global feature and the first global feature is determined as the mean squared error loss function.

5. The method of claim 1, wherein, When training the pre-trained visual recognition model based on the total loss function, the method further includes: The model parameters of the pre-trained visual recognition model are updated using an exponential moving average update mechanism.

6. A visual recognition device based on local domain self-adaptive training enhancement, characterized in that, include: The acquisition module is configured to acquire a local domain reference image and perform pose annotation on the local domain reference image; The feature extraction module is configured to use a pre-trained visual recognition model to perform global feature extraction on each image in the local domain reference image to obtain the first global feature of each reference image; The training module is configured to construct local training samples based on the first global features and poses of each reference image, train the pre-trained visual recognition model based on the local training samples, and obtain the triplet loss function. The feature extraction module is further configured to fine-tune the pre-trained visual recognition model based on the local training samples, and use the fine-tuned pre-trained visual recognition model to perform global feature extraction on each image in the local reference image to obtain the second global features of each reference image. The training module is also configured to determine the mean squared error loss function based on the similarity between the first global feature and the second global feature; The training module is further configured to perform a weighted summation of the triplet loss function and the mean squared error loss function to obtain a total loss function, and to train the pre-trained visual recognition model based on the total loss function to obtain the trained visual recognition model. The recognition module is configured to acquire a target image to be recognized and to recognize the target image using the trained visual recognition model. The target image includes local domain features; Among them, local training samples are constructed based on the first global features and pose of each benchmark image, including: Obtain the target reference image; the target reference image is any image from the local reference images; N matching reference images are determined for the target reference image; wherein, the N matching reference images are the top N reference images whose first global feature has the highest similarity to the first global feature of the target reference image, and N is a positive integer; Among the N matching reference images, the matching reference images that meet the preset pose judgment conditions are determined as positive sample images, and the remaining matching reference images are determined as negative sample images. A first distance is set between the positive sample image and the target reference image, and a second distance is set between the negative sample image and the target reference image; wherein the first distance is less than the second distance; The local domain reference images are traversed to obtain the local domain training samples; The preset pose determination condition is: The absolute value of the difference between the geographic location information of the target reference image and the geographic location information of the matching reference image is less than or equal to a preset distance threshold; and, The absolute value of the difference between the orientation of the target reference image and the orientation of the matching reference image is less than or equal to a preset orientation threshold.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images