Multimodal large model visual positioning method and device based on automated data screening

Abstract

This invention provides a multimodal large-scale model visual localization method and apparatus based on automated data filtering. The method includes: performing a first-stage fine-tuning of the multimodal large-scale model using an original image-text pair dataset; repeatedly inputting the original image and visual localization question text from the same original image-text pair dataset into the initially-tuned multimodal large-scale model a predetermined number of times to obtain multiple candidate bounding boxes; calculating the cross-modal semantic similarity and model generation confidence of each candidate bounding box and weighted fusion to obtain a comprehensive quality score; filtering the optimal result based on the comprehensive quality score, comparing its intersection-union ratio (IU) with the original labeled bounding boxes with a predetermined threshold, and constructing a preferred or negative sample dataset based on the comparison result; sequentially performing a second and third-stage fine-tuning of the initially-tuned multimodal large-scale model using the two datasets; inputting the image to be localized and the visual localization question text into the fine-tuned multimodal large-scale model to obtain the final visual localization result. The automated filtering of this invention is efficient and accurate.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal large model visual localization method and apparatus based on automated data screening. Background Technology

[0002] Multimodal large models require accurate spatial perception capabilities in visual localization tasks. Existing technologies pre-train and fine-tune multimodal large models by constructing image-text pairs containing spatial location information (such as normalized bounding box coordinates), enabling the model to learn the positional relationships of targets in images and output target coordinates, effectively addressing the problem of missing spatial information in the model's learning. However, in practical implementation, the fine-tuned multimodal large model often generates multiple candidate localization results for the same image-text pair input. Existing solutions manually select the best correct result and use it to construct new training data for further fine-tuning. This manual selection method has significant drawbacks: low efficiency, high labor costs, and inconsistent results due to subjective judgments from different annotators. Furthermore, existing solutions lack objective quantitative evaluation of candidate result quality, failing to consider both the semantic matching degree between the result and the problem, and the confidence level of model generation. Additionally, the fixed parameters of the multimodal large model prevent it from forming an adaptive feedback loop with the data selection quality, hindering continuous optimization of the model's spatial localization capabilities.

[0003] Therefore, the existing technology lacks a multimodal large model visual localization method and device based on automated data screening to achieve automated evaluation of candidate result quality and collaborative optimization of model data. Summary of the Invention

[0004] Therefore, it is necessary to provide a multimodal large model visual positioning method and device based on automated data screening to address the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a multimodal large model visual localization method based on automated data filtering, comprising:

[0006] Obtain the original image-text pair dataset, which includes multiple original image-text pair data. Each original image-text pair data includes the original image, the visual localization question text, and the original labeled bounding box.

[0007] The first stage of fine-tuning of the multimodal large model was performed using the original image and text dataset to obtain the initially tuned multimodal large model;

[0008] By repeatedly inputting the original image and visual localization problem text from the same original image-text pair into the pre-adjusted multimodal large model a preset number of times, multiple candidate bounding boxes for the same original image-text pair are obtained.

[0009] The cross-modal semantic similarity and model generation confidence of each candidate bounding box are calculated separately, and the cross-modal semantic similarity and model generation confidence of each candidate bounding box are weighted and fused to obtain the comprehensive quality score of each candidate bounding box.

[0010] The optimal result is selected based on the comprehensive quality score of each candidate bounding box, and the intersection-union ratio of the optimal result and the original labeled bounding boxes in the original image-text pair data is compared with a preset threshold. Based on the comparison results, the preferred dataset or the negative sample dataset is constructed respectively.

[0011] The optimized dataset and the negative sample dataset are used sequentially to perform the second and third stage fine-tuning on the initially tuned multimodal large model, resulting in the fine-tuned multimodal large model.

[0012] The image to be localized and the corresponding visual localization question text are input into the fine-tuned multimodal large model, and the predicted bounding box coordinates are output as the final visual localization result.

[0013] Optionally, cross-modal semantic similarity is calculated for each candidate bounding box, including:

[0014] For each candidate bounding box generated by the multimodal large model after initial adjustment, the image region is cropped from the corresponding original image according to its coordinates and scaled to a preset size;

[0015] The scaled image region is input into the image encoder of the multimodal pre-trained model to obtain the normalized image feature vector;

[0016] The visual localization problem text is input into the text encoder of the multimodal pre-trained model to obtain the normalized text feature vector;

[0017] Calculate the cosine similarity between the image feature vector and the text feature vector, and use it as the cross-modal semantic similarity of the candidate bounding box.

[0018] Optionally, the model generation confidence score for each candidate bounding box is calculated, including:

[0019] For each candidate bounding box generated by the multimodal large model after initial adjustment, extract the log probability of each token from the sequence that generated the candidate bounding box;

[0020] Calculate the arithmetic mean of the log probabilities of all tokens to obtain the corresponding average log probability;

[0021] The model generation confidence score is obtained by exponentially calculating the average log probability.

[0022] Optionally, the cross-modal semantic similarity of each candidate bounding box is weighted and fused with the model generation confidence to obtain a comprehensive quality score for each candidate bounding box, including:

[0023] According to the formula The cross-modal semantic similarity of each candidate bounding box is weighted and fused with the model generation confidence to obtain a comprehensive quality score for each candidate bounding box; where, The overall quality score for each candidate bounding box. For cross-modal semantic similarity, Generate confidence scores for the model, where α is a preset weight coefficient.

[0024] Optionally, the method further includes:

[0025] After processing a preset number of original image-text pairs, the average quality score of the comprehensive quality score of the best results included in the preferred dataset is calculated for that preset number of original image-text pairs.

[0026] The temperature parameters of the multimodal large model are dynamically updated based on the average mass fraction.

[0027] Optionally, the temperature parameter is updated using the following formula: ,in, For the updated temperature parameters, The current temperature parameter, The preset adjustment coefficient, The average quality score.

[0028] Optionally, the optimal result is selected based on the comprehensive quality score of each candidate bounding box, and the intersection-union ratio (IUU) of this optimal result and the original labeled bounding boxes in the original image-text pair data is compared with a preset threshold. Based on the comparison results, a preferred dataset or a negative sample dataset is constructed, including:

[0029] For multiple candidate bounding boxes of the same original image-text pair, the candidate bounding box with the highest comprehensive quality score is selected as the optimal result.

[0030] Calculate the intersection-union ratio (IoU) of this optimal result with the original labeled bounding boxes in the original image-text pair data;

[0031] If the intersection-union ratio is greater than or equal to the preset threshold, the original labeled bounding boxes in the original image-text pair data are replaced with the optimal result, and the replaced original image-text pair data is included in the preferred dataset.

[0032] If the intersection-union ratio is less than a preset threshold, negative samples are constructed and included in the negative sample dataset based on the optimal result and the original images and visual positioning problem text in the original image-text pair data.

[0033] Optionally, the initial temperature parameter of the multimodal large model is 1.0.

[0034] Optionally, when performing the first-stage fine-tuning, second-stage fine-tuning, and third-stage fine-tuning on the multimodal large model, the fine-tuning method adopts the full-parameter fine-tuning method or the LoRA fine-tuning method, and the loss function adopts the cross-entropy loss function.

[0035] Secondly, the present invention provides a multimodal large model visual positioning device based on automated data filtering, comprising: an acquisition module, a quality assessment module, a filtering and construction module, and a multimodal large model; wherein,

[0036] The acquisition module is used to acquire the original image-text pair dataset, which includes multiple original image-text pair data. Each original image-text pair data includes the original image, the visual localization question text, and the original labeled bounding box.

[0037] The quality assessment module is connected to the acquisition module and the multimodal large model respectively. It is used to calculate the cross-modal semantic similarity and model generation confidence of each candidate bounding box, and to perform weighted fusion of the cross-modal semantic similarity and model generation confidence of each candidate bounding box to obtain the comprehensive quality score of each candidate bounding box.

[0038] The filtering module, connected to the quality assessment module, is used to filter out the best result based on the comprehensive quality score of each candidate bounding box. It then compares the intersection-union ratio of the best result with the original labeled bounding boxes in the original image-text pair data with a preset threshold, and constructs either the preferred dataset or the negative sample dataset based on the comparison results.

[0039] The multimodal large model, connected to the acquisition module and the filtering and construction module respectively, is used to perform the first stage of fine-tuning on the multimodal large model using the original image-text pair dataset to obtain the initially tuned multimodal large model; the original image and visual localization question text from the same original image-text pair dataset are repeatedly input into the initially tuned multimodal large model a preset number of times to obtain multiple candidate bounding boxes for the same original image-text pair dataset; the optimized dataset and the negative sample dataset are used to perform the corresponding second stage and third stage fine-tuning on the initially tuned multimodal large model to obtain the fine-tuned multimodal large model; and the image to be localized and the corresponding visual localization question text are input into the fine-tuned multimodal large model, and the predicted bounding box coordinates are output as the final visual localization result.

[0040] This invention provides a multimodal large-scale model visual localization method and apparatus based on automated data screening. It achieves automated candidate bounding box screening through a dual-dimensional quantitative evaluation of cross-modal semantic similarity and model generation confidence, completely replacing manual annotation, significantly improving efficiency and reducing costs. Based on this, a second-stage fine-tuning is performed using a preferred dataset to improve the localization accuracy of the multimodal large-scale model. Simultaneously, a third-stage fine-tuning is performed using a negative sample dataset to enhance the model's rejection capability, enabling it to output a no-target label when unable to locate, thus avoiding erroneous output. Furthermore, a dynamic temperature parameter adjustment mechanism is introduced, adaptively adjusting the generation parameters based on the batch average quality score, forming a closed-loop optimization between the multimodal large-scale model and the data. Attached Figure Description

[0041] Figure 1a This is a flowchart illustrating a multimodal large model visual localization method based on automated data filtering provided in an embodiment of the present invention.

[0042] Figure 1b This is another flowchart illustrating the multimodal large model visual localization method based on automated data filtering provided in this embodiment of the invention.

[0043] Figure 2 This is a schematic diagram of the circuit module structure of the multimodal large model visual positioning device based on automated data filtering provided in an embodiment of the present invention;

[0044] Figure 3 This is an internal structural diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] like Figure 1a As shown, this invention provides a multimodal large model visual localization method based on automated data filtering, including:

[0047] Step S10: Obtain the original image-text pair dataset, which includes multiple original image-text pair data, each of which includes the original image, visual localization question text, and original labeled bounding box;

[0048] Optionally, the original image-text pair dataset uses the Shikra dataset. Each original image-text pair in the Shikra dataset includes: the original image (or the storage path of the original image), a visual localization question text for the target in the original image, and the original labeled bounding box corresponding to the target. The original labeled bounding box is represented using normalized bounding box coordinates, such as [x1, y1, x2, y2], with each coordinate value ranging from [0, 1].

[0049] Step S11: Use the original image and text dataset to perform the first stage of fine-tuning on the multimodal large model to obtain the initially tuned multimodal large model;

[0050] In this invention, the multimodal large model can be any existing model such as Qwen-VL, CogVLM, or GPT-4o. Those skilled in the art can choose according to actual needs, and no limitation is made here. The initial temperature parameter of the multimodal large model can be set to 1.0. When the temperature parameter is 1.0, the probability distribution output by the multimodal large model is relatively smooth, and the sampling has high randomness, which is beneficial for generating diverse candidate bounding boxes. It should be noted that those skilled in the art can also set other temperature parameters (such as 0.8, 1.2, etc.) according to actual needs, and this invention does not limit this.

[0051] In this invention, during the first stage of fine-tuning of the multimodal large model, the fine-tuning method can be the existing full-parameter fine-tuning method or the LoRA fine-tuning method, and the loss function can be the existing cross-entropy loss function.

[0052] Step S12: Repeatedly input the original image and visual localization problem text from the same original image-text pair data into the pre-adjusted multimodal large model a preset number of times to obtain multiple candidate bounding boxes for the same original image-text pair data.

[0053] In this invention, the preset number of times can be flexibly selected by those skilled in the art according to actual needs, and is not limited here. For example, the value of the preset number of times can be in the range of 3 to 20 times, and preferably 5 times.

[0054] For example, suppose there is a pair of original image and text data, where the original image is a photo containing a red car and a blue car; the visual location question text is "Where is the red car?"; and the original labeled bounding box is [0.20, 0.30, 0.50, 0.60]. The original image and visual localization problem text from the same original image-text pair data were repeatedly input into the pre-tuned multimodal large model 5 times (each time the temperature parameter was 1.0), resulting in 5 candidate bounding boxes: the candidate bounding box output for the first time was [0.21, 0.31, 0.51, 0.59]; the candidate bounding box output for the second time was [0.19, 0.32, 0.50, 0.61]; the candidate bounding box output for the third time was [0.22, 0.28, 0.52, 0.60]; the candidate bounding box output for the fourth time was [0.18, 0.33, 0.49, 0.62]; and the candidate bounding box output for the fifth time was [0.23, 0.30, 0.53, 0.58].

[0055] It should be noted that, in this invention, before inputting the original image and visual localization problem text into the multimodal large model, those skilled in the art can perform necessary preprocessing on the input data according to the standard interface requirements of the selected specific model. For example, for the original image, decoding, scaling, pixel value normalization, and conversion to the tensor format required by the model are usually required; for the visual localization problem text, word segmentation, conversion to token identifier sequences, and then mapping to vector sequences through an embedding layer are usually required. The above preprocessing is common knowledge in the art, and the specific preprocessing parameters of different models can be obtained from publicly available model documentation, and will not be elaborated here. In addition, other models in this invention can be deduced similarly, and will not be elaborated here.

[0056] Step S13: Calculate the cross-modal semantic similarity and model generation confidence of each candidate bounding box, and perform weighted fusion of the cross-modal semantic similarity and model generation confidence of each candidate bounding box to obtain the comprehensive quality score of each candidate bounding box;

[0057] In an optional embodiment of the present invention, step S13, calculating the cross-modal semantic similarity of each candidate bounding box, includes:

[0058] Step S131: For each candidate bounding box generated by the multimodal large model after initial adjustment, crop the image region from the corresponding original image according to its coordinates and scale it to the preset size;

[0059] In step S131, scaling the cropped image region to a preset size is to meet the fixed input image size requirement of subsequent multimodal pre-trained models (such as the CLIP model, full name: Contrastive Language-Image Pre-training). These multimodal pre-trained models typically use convolutional neural networks or visual Transformers as image encoders, and their input size is predefined (e.g., 224×224 pixels). If the cropped region is not uniformly scaled, the image region sizes corresponding to different candidate bounding boxes will vary, making it impossible to batch input features into the image encoder for feature extraction, and also impossible to obtain image feature vectors with consistent dimensions, thus making it impossible to calculate cosine similarity. Those skilled in the art can determine the preset size based on the standard interface of the selected multimodal pre-trained model; for example, it can be set to 224×224 pixels for the CLIP model. This invention does not limit this.

[0060] Step S132: Input the scaled image region into the image encoder of the multimodal pre-trained model to obtain the normalized image feature vector;

[0061] In this invention, the multimodal pre-trained model can adopt the existing CLIP model.

[0062] Step S133: Input the visual localization problem text into the text encoder of the multimodal pre-trained model to obtain the normalized text feature vector;

[0063] Step S134: Calculate the cosine similarity between the image feature vector and the text feature vector, and use it as the cross-modal semantic similarity of the candidate bounding box.

[0064] Continuing with the previous example, the original image is a photo containing a red car and a blue car, and the visual localization question text is "Where is the red car?". The candidate bounding boxes output in the first iteration are [0.21, 0.31, 0.51, 0.59] (normalized candidate bounding box coordinates).

[0065] First, based on the candidate bounding box coordinates [0.21, 0.31, 0.51, 0.59] from the first output, the corresponding image region (the local area where the red car is located) is cropped from the original image (a photo containing a red car and a blue car) and scaled to a preset size (e.g., 224×224 pixels).

[0066] Secondly, the scaled image region is input into the image encoder of a multimodal pre-trained model (such as the CLIP model) to obtain a normalized image feature vector, such as... .

[0067] Then, the visual localization question text "Where is the red car?" is input into the text encoder of the same multimodal pre-trained model to obtain a normalized text feature vector, such as... .

[0068] Finally, the cosine similarity between the image feature vector and the text feature vector is calculated as the cross-modal semantic similarity of the candidate bounding box. ,in, The cross-modal semantic similarity of the candidate bounding box. This is the normalized image feature vector of the candidate bounding box. This is the normalized text feature vector of the candidate bounding box. It is the L2 norm (i.e., the Euclidean norm). Assume... The closer this value is to 1, the higher the semantic matching degree between the cropped image region and the visual positioning problem text.

[0069] Based on the above process, the cross-modal semantic similarity of other candidate bounding boxes (such as the other 4 candidate bounding boxes) can be calculated, which will not be elaborated here.

[0070] In an optional embodiment of the present invention, the calculation of the model generation confidence score for each candidate bounding box in step S13 includes:

[0071] Step S135: For each candidate bounding box generated by the multimodal large model after initial adjustment, extract the log probability of each token from the sequence that generated the candidate bounding box;

[0072] Step S136: Calculate the arithmetic mean of the log probabilities of all tokens to obtain the corresponding average log probability;

[0073] Step S137: Take the exponential operation on the average log probability to obtain the model generation confidence score.

[0074] Continuing with the previous example, the first output candidate bounding box is [0.21, 0.31, 0.51, 0.59]. This candidate bounding box is generated by the multimodal large model. When generating, the multimodal large model will output a sequence, such as "0.21, 0.31, 0.51, 0.59". Assume that this sequence contains 4 coordinate tokens (each number is a token).

[0075] First, extract the log probability of each token from the sequence that generated the candidate bounding box. Assume the log probabilities of the multimodal large model at each coordinate when generating the candidate bounding box are as follows: log probability of token 0.21 = -0.12; log probability of token 0.31 = -0.08; log probability of token 0.51 = -0.15; log probability of token 0.59 = -0.10.

[0076] Secondly, calculate the arithmetic mean of the log probabilities of all tokens to obtain the corresponding average log probability, i.e. ,in, This represents the average logarithmic probability.

[0077] Finally, the average log probability is exponentially calculated to obtain the model generation confidence score: .

[0078] Based on the above process, the model generation confidence of other candidate bounding boxes (such as the other 4 candidate bounding boxes) can be calculated, which will not be elaborated here.

[0079] In an optional embodiment of the present invention, step S13, which involves weighted fusion of the cross-modal semantic similarity and model generation confidence of each candidate bounding box to obtain a comprehensive quality score for each candidate bounding box, includes:

[0080] Step S138: According to the formula The cross-modal semantic similarity of each candidate bounding box is weighted and fused with the model generation confidence to obtain a comprehensive quality score for each candidate bounding box; where, The overall quality score for each candidate bounding box. The cross-modal semantic similarity of the candidate bounding box. Generate a confidence score for the model of the candidate bounding box, where α is a preset weight coefficient.

[0081] In this invention, the specific value of the preset weight coefficient α can be flexibly adjusted according to actual needs, and this invention does not limit it. Preferably, α is set to 0.6. This preferred value focuses more on cross-modal semantic similarity while also taking into account the model generation confidence, so as to ensure the stability of the results.

[0082] Step S14: Select the best result based on the comprehensive quality score of each candidate bounding box, and compare the intersection-union ratio of the best result and the original labeled bounding boxes in the original image-text pair data with a preset threshold. Construct the preferred dataset or the negative sample dataset according to the comparison results.

[0083] In one optional embodiment of the present invention, such as Figure 1b As shown, step S14 specifically includes:

[0084] Step S141: For multiple candidate bounding boxes of the same original image-text pair, select the candidate bounding box with the highest comprehensive quality score as the optimal result;

[0085] Step S142: Calculate the intersection-union ratio (IUU) of the optimal result with the original labeled bounding boxes in the original image-text pair data;

[0086] For example, suppose the optimal result is [0.20, 0.30, 0.50, 0.60] (normalized coordinates), where the coordinates of the top left corner are (0.20, 0.30) and the coordinates of the bottom right corner are (0.50, 0.60); the original labeled bounding box (true label) is [0.25, 0.35, 0.55, 0.65], where the coordinates of the top left corner are (0.25, 0.35) and the coordinates of the bottom right corner are (0.55, 0.65).

[0087] First, calculate the coordinates of the intersection rectangle. Specifically, the x-coordinate of the top-left corner of the intersection rectangle is max(0.20, 0.25) = 0.25, the y-coordinate of the top-left corner is max(0.30, 0.35) = 0.35, the x-coordinate of the bottom-right corner is min(0.50, 0.55) = 0.50, and the y-coordinate of the bottom-right corner is min(0.60, 0.65) = 0.60.

[0088] Next, calculate the area of ​​the intersection rectangle. Specifically, the width of the intersection rectangle = 0.50 - 0.25 = 0.25, the height of the intersection rectangle = 0.60 - 0.35 = 0.25, and the area of ​​the intersection rectangle = 0.25 × 0.25 = 0.0625.

[0089] Next, calculate the area of ​​the optimal result and the area of ​​the original labeled bounding box. Specifically, the area of ​​the optimal result = (0.50-0.20)×(0.60-0.30)=0.30×0.30=0.09, and the area of ​​the original labeled bounding box = (0.55-0.25)×(0.65-0.35)=0.30×0.30=0.09.

[0090] Then, calculate the area of ​​the union of the optimal result and the original labeled bounding box. Specifically, the area of ​​the union of the optimal result and the original labeled bounding box = the sum of the areas of the two boxes - the intersection area = 0.09 + 0.09 - 0.0625 = 0.1175.

[0091] Finally, the intersection-union ratio (IUR) of the optimal result with the original labeled bounding boxes in the original image-text pair data is calculated. Specifically, the IUR of the optimal result with the original labeled bounding boxes in the original image-text pair data = area of ​​the intersection rectangle / area of ​​the union of the optimal result and the original labeled bounding boxes = 0.0625 / 0.1175 ≈ 0.5319.

[0092] The above is merely an optional implementation of the present invention for calculating the intersection-union ratio of the optimal result with the original labeled bounding boxes in the original image-text pair data. Those skilled in the art can also choose other existing methods to calculate according to actual needs, which are not limited here.

[0093] Step S143: If the intersection-union ratio is greater than or equal to the preset threshold, replace the original labeled bounding boxes in the original image-text pair data with the optimal result, and include the replaced original image-text pair data into the preferred dataset;

[0094] Step S144: If the intersection-union ratio is less than a preset threshold, based on the optimal result and the original image and visual positioning problem text in the original image-text pair data, construct negative samples and include them in the negative sample dataset.

[0095] In this invention, a preset threshold is used to determine whether the selected optimal result should be included in the preferred dataset or the negative sample dataset. When the intersection-union ratio (IoU) of the optimal result and the original labeled bounding boxes in the original image-text pair data is greater than or equal to the preset threshold, the optimal result replaces the original labeled bounding boxes in the original image-text pair data, and the replaced original image-text pair data is included in the preferred dataset. When the IoU is less than the preset threshold, the optimal result, along with its corresponding original image and visual localization problem text, constitutes a negative sample and is included in the negative sample dataset. Those skilled in the art can flexibly set the preset threshold according to actual needs, and this invention does not impose specific limitations on it. Preferably, the preset threshold is set to 0.1. This preset threshold can effectively eliminate invalid results that deviate significantly from the true labels (i.e., the original labeled bounding boxes) (IoU represents the intersection-union ratio, and IoU < 0.1 usually indicates that the two bounding boxes have almost no overlap), while retaining high-quality results with high localization accuracy. If the preset threshold is set too high (e.g., 0.5), a large amount of valid data may be misjudged as negative samples; if it is set too low (e.g., 0.01), it is difficult to filter out the optimal results with poor quality. Therefore, setting the preset threshold to 0.1 is an optimal value that balances strict screening with data utilization.

[0096] Step S15: Use the preferred dataset and the negative sample dataset to perform the corresponding second-stage fine-tuning and third-stage fine-tuning on the initially tuned multimodal large model in sequence to obtain the fine-tuned multimodal large model;

[0097] In this invention, when performing the second and third stage fine-tuning of the multimodal large model, the fine-tuning method can be the existing full-parameter fine-tuning method or the LoRA fine-tuning method, and the loss function can be the existing cross-entropy loss function.

[0098] For example, samples from the preferred dataset are used for the second stage of fine-tuning. Each sample includes: the original image, the visual localization question text, and the corresponding optimal result (which is included in the preferred dataset because IoU ≥ 0.1). During fine-tuning, the original image and the visual localization question text are fed as input into the initially tuned multimodal large model for fine-tuning, enabling the multimodal large model to learn to generate high-quality localization results that are close to the optimal bounding box.

[0099] Before the third stage of fine-tuning, a special tag needs to be added to the output vocabulary of the multimodal large model.<no_box> This tag corresponds to a newly added token ID, used to indicate "target does not exist" or "cannot be accurately located". Samples from the negative sample dataset are used for the third stage of fine-tuning. Each sample includes: the original image, the visual localization question text, and the corresponding optimal result (the optimal result is included in the negative sample dataset because IoU < 0.1). During fine-tuning, the original image and the visual localization question text are again fed as input into the multimodal large model after the second stage of fine-tuning, but the expected output at this time is a rejection tag.<no_box> Similarly, using either full-parameter fine-tuning or LoRA fine-tuning methods, and employing the cross-entropy loss function, the predicted output is calculated as...<no_box> The differences between them enable the multimodal large model to learn to avoid outputting inaccurate bounding boxes when it is difficult to locate.

[0100] After the above two stages of fine-tuning, the fine-tuned multimodal large model is obtained. This fine-tuned multimodal large model can output accurate bounding box coordinates for locatable targets and output rejection markers for unlocatable targets, significantly improving the robustness and reliability of visual localization.

[0101] Step S16: Input the image to be localized and the corresponding visual localization question text into the fine-tuned multimodal large model, and output the predicted bounding box coordinates as the final visual localization result.

[0102] In an optional embodiment of the present invention, the method further includes:

[0103] Step S17: After processing a preset number of original image-text pairs, calculate the average comprehensive quality score of the best results included in the preferred dataset from the preset number of original image-text pairs to obtain the average quality score.

[0104] The preset quantity can be flexibly set by those skilled in the art according to actual needs, and is not limited here. For example, the preset quantity is 1000 items.

[0105] Step S18: Dynamically update the temperature parameters of the multimodal large model based on the average mass fraction.

[0106] The temperature parameter is updated using the following formula: ,in, For the updated temperature parameters, The current temperature parameter, The preset adjustment coefficient, The average quality score.

[0107] The preset adjustment coefficient can be flexibly selected by those skilled in the art according to actual needs, and is not limited here. Preferably, the preset adjustment coefficient is 0.2.

[0108] Assume a preset quantity of 1000 original image-text pairs. After processing steps S12 to S14, 800 of these 1000 original image-text pairs pass the IoU validation (i.e., IoU ≥ 0.1) and are included in the preferred dataset. For each of these 800 original image-text pairs, there is an optimal overall quality score. Calculate these 800 The arithmetic mean of the values ​​yields the average mass fraction. Assuming .

[0109] Set the current temperature parameter Preset adjustment coefficient According to the temperature parameter update formula: .

[0110] Updated temperature parameters The updated temperature parameter will be used for candidate bounding box generation in the next batch (data pairs 1001 to 2000). Because the average quality score of this batch is high (0.75), the temperature was reduced from 1.0 to 0.85, resulting in more focused and accurate results for subsequent multimodal large models.

[0111] If no original image-text pairs are included in the preferred dataset in a certain batch (i.e.) If the calculation fails, maintain the current temperature parameter, or set... Restore the temperature parameter to Those skilled in the art can handle this flexibly according to the actual situation.

[0112] It should be noted that in the dynamic temperature parameter adjustment mechanism of this invention, after each batch is processed, the temperature parameter is updated based on the temperature parameter used in the current batch (rather than a fixed initial temperature parameter) and the average mass fraction of the current batch, and is recursively calculated. That is, the updated temperature parameter will be used as the current temperature parameter for the next batch, thereby achieving continuous adaptive adjustment of the temperature parameter.

[0113] It should be noted that steps S131 to S138, S17 and S18 are only for ease of description and are not shown in the figure.

[0114] This invention provides a multimodal large-scale model visual localization method based on automated data filtering. It achieves automated candidate bounding box filtering through a dual-dimensional quantitative evaluation of cross-modal semantic similarity and model generation confidence, completely replacing manual annotation and significantly improving efficiency while reducing costs. Building upon this, a second-stage fine-tuning is performed using a preferred dataset to improve the localization accuracy of the multimodal large-scale model. Simultaneously, a third-stage fine-tuning is performed using a negative sample dataset to enhance the model's rejection capability, enabling it to output a no-target label when unable to locate, thus avoiding erroneous outputs. Furthermore, a dynamic temperature parameter adjustment mechanism is introduced, adaptively adjusting the generation parameters based on the batch average quality score, forming a closed-loop optimization between the multimodal large-scale model and the data.

[0115] Based on the same inventive concept, embodiments of the present invention also provide a multimodal large model visual localization device based on automated data filtering for implementing the aforementioned multimodal large model visual localization method based on automated data filtering. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal large model visual localization device based on automated data filtering provided below can be found in the limitations of the multimodal large model visual localization method based on automated data filtering described above, and will not be repeated here.

[0116] like Figure 2 As shown, this invention provides a multimodal large-scale model visual positioning device based on automated data filtering, characterized by comprising: an acquisition module 21, a quality assessment module 22, a filtering and construction module 23, and a multimodal large-scale model 24; wherein,

[0117] The acquisition module 21 is used to acquire the original image-text pair dataset, which includes multiple original image-text pair data, and each original image-text pair data includes the original image, visual positioning question text and original labeled bounding box.

[0118] The quality assessment module 22 is connected to the acquisition module 21 and the multimodal large model 24 respectively. It is used to calculate the cross-modal semantic similarity and model generation confidence of each candidate bounding box respectively, and to perform weighted fusion of the cross-modal semantic similarity and model generation confidence of each candidate bounding box to obtain the comprehensive quality score of each candidate bounding box.

[0119] The screening and construction module 23, connected to the quality assessment module 22, is used to select the best result based on the comprehensive quality score of each candidate bounding box, and compare the intersection-union ratio of the best result and the original labeled bounding boxes in the original image-text pair data with a preset threshold. Based on the comparison results, the preferred dataset or the negative sample dataset is constructed respectively.

[0120] The multimodal large model 24, connected to the acquisition module 21 and the filtering and construction module 23, is used to perform a first-stage fine-tuning of the multimodal large model using the original image-text pair dataset to obtain the initially tuned multimodal large model; repeatedly inputting the original image and visual localization question text from the same original image-text pair dataset into the initially tuned multimodal large model a preset number of times to obtain multiple candidate bounding boxes for the same original image-text pair dataset; sequentially using the preferred dataset and the negative sample dataset to perform corresponding second-stage and third-stage fine-tuning of the initially tuned multimodal large model to obtain the fine-tuned multimodal large model; and inputting the image to be localized and the corresponding visual localization question text into the fine-tuned multimodal large model to output the predicted bounding box coordinates as the final visual localization result.

[0121] Optionally, cross-modal semantic similarity is calculated for each candidate bounding box, including:

[0122] For each candidate bounding box generated by the multimodal large model after initial adjustment, the image region is cropped from the corresponding original image according to its coordinates and scaled to a preset size;

[0123] The scaled image region is input into the image encoder of the multimodal pre-trained model to obtain the normalized image feature vector;

[0124] The visual localization problem text is input into the text encoder of the multimodal pre-trained model to obtain the normalized text feature vector;

[0125] Calculate the cosine similarity between the image feature vector and the text feature vector, and use it as the cross-modal semantic similarity of the candidate bounding box.

[0126] Optionally, the model generation confidence score for each candidate bounding box is calculated, including:

[0127] For each candidate bounding box generated by the multimodal large model after initial adjustment, extract the log probability of each token from the sequence that generated the candidate bounding box;

[0128] Calculate the arithmetic mean of the log probabilities of all tokens to obtain the corresponding average log probability;

[0129] The model generation confidence score is obtained by exponentially calculating the average log probability.

[0130] Optionally, the cross-modal semantic similarity of each candidate bounding box is weighted and fused with the model generation confidence to obtain a comprehensive quality score for each candidate bounding box, including:

[0131] According to the formula The cross-modal semantic similarity of each candidate bounding box is weighted and fused with the model generation confidence to obtain a comprehensive quality score for each candidate bounding box; where, The overall quality score for each candidate bounding box. For cross-modal semantic similarity, Generate confidence scores for the model, where α is a preset weight coefficient.

[0132] In an optional embodiment of the present invention, the multimodal large model visual positioning device for automated data filtering of the present invention further includes: a temperature update module (not shown in the figure), connected to the multimodal large model 24, used to calculate the average comprehensive quality score of the best results included in the preferred dataset in the preset number of original image-text pairs after processing each preset number of original image-text pairs, and obtain the average quality score; and dynamically update the temperature parameters of the multimodal large model according to the average quality score.

[0133] The temperature parameter can be updated using the following formula: ,in, For the updated temperature parameters, The current temperature parameter, The preset adjustment coefficient, The average quality score.

[0134] Optionally, the filtering module 23 is specifically used for: selecting the candidate bounding box with the highest comprehensive quality score as the optimal result from multiple candidate bounding boxes of the same original image-text pair data; calculating the intersection-union ratio (IU) of the optimal result with the original labeled bounding boxes in the original image-text pair data; if the IU is greater than or equal to a preset threshold, replacing the original labeled bounding boxes in the original image-text pair data with the optimal result, and including the replaced original image-text pair data in the preferred dataset; if the IU is less than the preset threshold, constructing negative samples based on the optimal result and the original image and visual localization problem text in the corresponding original image-text pair data, and including them in the negative sample dataset.

[0135] Optionally, the initial temperature parameter of the multimodal large model is 1.0.

[0136] Optionally, when performing the first-stage fine-tuning, second-stage fine-tuning, and third-stage fine-tuning on the multimodal large model, the fine-tuning method adopts the full-parameter fine-tuning method or the LoRA fine-tuning method, and the loss function adopts the cross-entropy loss function.

[0137] The multimodal large-scale model visual localization device provided by this invention achieves automated candidate bounding box selection through a two-dimensional quantitative evaluation of cross-modal semantic similarity and model generation confidence, completely replacing manual annotation, significantly improving efficiency and reducing costs. Based on this, a second-stage fine-tuning is performed using a preferred dataset to improve the localization accuracy of the multimodal large-scale model, while a third-stage fine-tuning is performed using a negative sample dataset to enhance the model's rejection capability, enabling it to output a no-target label when unable to locate, thus avoiding erroneous output. Furthermore, a dynamic temperature parameter adjustment mechanism is introduced, adaptively adjusting the generation parameters based on the batch average quality score, forming a closed-loop optimization between the multimodal large-scale model and the data.

[0138] It should be noted that "multiple" in this invention includes two or more.

[0139] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0140] Each module in the devices of this invention can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0141] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data required for or generated by the aforementioned multimodal large-scale model visual localization method based on automated data filtering. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal large-scale model visual localization method based on automated data filtering.

[0142] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a multimodal large-model visual positioning method based on automated data filtering. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0143] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0144] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0145] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0146] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0147] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties.

[0148] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided by this invention may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided by this invention may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0149] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0150] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A multimodal large-scale model visual localization method based on automated data filtering, characterized in that, include: Obtain the original image-text pair dataset, which includes multiple original image-text pair data. Each original image-text pair data includes the original image, the visual localization question text, and the original labeled bounding box. The first stage of fine-tuning of the multimodal large model was performed using the original image and text dataset to obtain the initially tuned multimodal large model; By repeatedly inputting the original image and visual localization problem text from the same original image-text pair into the pre-adjusted multimodal large model a preset number of times, multiple candidate bounding boxes for the same original image-text pair are obtained. The cross-modal semantic similarity and model generation confidence of each candidate bounding box are calculated separately, and the cross-modal semantic similarity and model generation confidence of each candidate bounding box are weighted and fused to obtain the comprehensive quality score of each candidate bounding box. The optimal result is selected based on the comprehensive quality score of each candidate bounding box, and the intersection-union ratio of the optimal result and the original labeled bounding boxes in the original image-text pair data is compared with a preset threshold. Based on the comparison results, the preferred dataset or the negative sample dataset is constructed respectively. The optimized dataset and the negative sample dataset are used sequentially to perform the second and third stage fine-tuning on the initially tuned multimodal large model, resulting in the fine-tuned multimodal large model. The image to be localized and the corresponding visual localization question text are input into the fine-tuned multimodal large model, and the predicted bounding box coordinates are output as the final visual localization result.

2. The method according to claim 1, characterized in that, Calculate the cross-modal semantic similarity for each candidate bounding box, including: For each candidate bounding box generated by the multimodal large model after initial adjustment, the image region is cropped from the corresponding original image according to its coordinates and scaled to a preset size; The scaled image region is input into the image encoder of the multimodal pre-trained model to obtain the normalized image feature vector; The visual localization problem text is input into the text encoder of the multimodal pre-trained model to obtain the normalized text feature vector; Calculate the cosine similarity between the image feature vector and the text feature vector, and use it as the cross-modal semantic similarity of the candidate bounding box.

3. The method according to claim 1, characterized in that, Calculate the model generation confidence score for each candidate bounding box, including: For each candidate bounding box generated by the multimodal large model after initial adjustment, extract the log probability of each token from the sequence that generated the candidate bounding box; Calculate the arithmetic mean of the log probabilities of all tokens to obtain the corresponding average log probability; The model generation confidence score is obtained by exponentially calculating the average log probability.

4. The method according to claim 1, characterized in that, The weighted fusion of cross-modal semantic similarity and model generation confidence for each candidate bounding box yields a comprehensive quality score for each candidate bounding box, including: According to the formula The cross-modal semantic similarity of each candidate bounding box is weighted and fused with the model generation confidence to obtain a comprehensive quality score for each candidate bounding box; where, The overall quality score for each candidate bounding box. For cross-modal semantic similarity, Generate confidence scores for the model, where α is a preset weight coefficient.

5. The method according to claim 1, characterized in that, The method further includes: After processing a preset number of original image-text pairs, the average quality score of the comprehensive quality score of the best results included in the preferred dataset is calculated for that preset number of original image-text pairs. The temperature parameters of the multimodal large model are dynamically updated based on the average mass fraction.

6. The method according to claim 5, characterized in that, The temperature parameter is updated using the following formula: ,in, For the updated temperature parameters, The current temperature parameter, The preset adjustment coefficient, The average quality score.

7. The method according to claim 1, characterized in that, The optimal result is selected based on the comprehensive quality score of each candidate bounding box, and the intersection-union ratio (IUU) of this optimal result and the original labeled bounding boxes in the original image-text pair data is compared with a preset threshold. Based on the comparison results, a preferred dataset or a negative sample dataset is constructed, including: For multiple candidate bounding boxes of the same original image-text pair, the candidate bounding box with the highest comprehensive quality score is selected as the optimal result. Calculate the intersection-union ratio (IoU) of this optimal result with the original labeled bounding boxes in the original image-text pair data; If the intersection-union ratio is greater than or equal to the preset threshold, the original labeled bounding boxes in the original image-text pair data are replaced with the optimal result, and the replaced original image-text pair data is included in the preferred dataset. If the intersection-union ratio is less than a preset threshold, negative samples are constructed and included in the negative sample dataset based on the optimal result and the original images and visual positioning problem text in the original image-text pair data.

8. The method according to claim 1, characterized in that, The initial temperature parameter of the multimodal large model is 1.

0.

9. The method according to claim 1, characterized in that, When performing the first-stage, second-stage, and third-stage fine-tuning on the multimodal large model, the fine-tuning method adopted is either the full-parameter fine-tuning method or the LoRA fine-tuning method, and the loss function adopted is the cross-entropy loss function.

10. A multimodal large-scale visual positioning device based on automated data filtering, characterized in that, include: The module includes an acquisition module, a quality assessment module, a screening and construction module, and a multimodal large model; among them, The acquisition module is used to acquire the original image-text pair dataset, which includes multiple original image-text pair data. Each original image-text pair data includes the original image, the visual localization question text, and the original labeled bounding box. The quality assessment module is connected to the acquisition module and the multimodal large model respectively. It is used to calculate the cross-modal semantic similarity and model generation confidence of each candidate bounding box, and to perform weighted fusion of the cross-modal semantic similarity and model generation confidence of each candidate bounding box to obtain the comprehensive quality score of each candidate bounding box. The filtering module, connected to the quality assessment module, is used to filter out the best result based on the comprehensive quality score of each candidate bounding box. It then compares the intersection-union ratio of the best result with the original labeled bounding boxes in the original image-text pair data with a preset threshold, and constructs either the preferred dataset or the negative sample dataset based on the comparison results. The multimodal large model, connected to the acquisition module and the filtering and construction module respectively, is used to perform the first stage of fine-tuning on the multimodal large model using the original image-text pair dataset to obtain the initially tuned multimodal large model; the original image and visual localization question text from the same original image-text pair dataset are repeatedly input into the initially tuned multimodal large model a preset number of times to obtain multiple candidate bounding boxes for the same original image-text pair dataset; the optimized dataset and the negative sample dataset are used to perform the corresponding second stage and third stage fine-tuning on the initially tuned multimodal large model to obtain the fine-tuned multimodal large model; and the image to be localized and the corresponding visual localization question text are input into the fine-tuned multimodal large model, and the predicted bounding box coordinates are output as the final visual localization result.

Images