An image and video retrieval method based on domain fine-tuning of a large language model

Abstract

The application provides an image and video retrieval method based on field fine-tuning of a large language model, which comprises the following steps: fine-tuning a pre-training model to obtain a fine-tuning pre-training model, which is used for intent classification and keyword extraction; screening an optimal prompt template through dynamic iteration by combining a Monte Carlo tree search with a hidden Markov model (HMM) model; filtering noise from a keyword list, and predicting the category label of the filtered keyword through a conditional random field model to obtain a keyword enhanced set; combining the keyword enhanced set with a user intent to generate a query condition, and obtaining a candidate resource set; obtaining a retrieved image or video according to the similarity between a user query text and the candidate resource set, and combining the optimal prompt template to obtain a resource path with the highest matching score of the user query and the candidate resource, which can effectively solve the recognition deviation of a general model when processing proper nouns and professional terms, improve the retrieval accuracy and response speed, and improve the accuracy and professional adaptability of retrieval.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence retrieval technology, and in particular to an image and video retrieval method based on a domain-fine-tuned large language model. Background Technology

[0002] With the rapid development of digital media technology, massive amounts of video and image data are widely produced and disseminated in fields such as social media, e-commerce, education, and entertainment. When users search for this content, they often need to sift through a huge library of materials to find resources that highly match their needs.

[0003] A video retrieval method based on deep learning and hash coding, disclosed in patent CN107229757B, utilizes a deep network to extract feature vectors from each frame of a video sample to form a feature matrix. The extracted video feature matrix is ​​then used as a whole for modeling to obtain a high-dimensional real-valued representation. A target loss function is constructed to constrain the ranking relationships between images and videos, videos and videos, and images and images. A hash function is implemented through a nonlinear mapping operation in the network backend, mapping the feature representations of images and videos to a common Hamming space to obtain binary hash codes. The Hamming distance is calculated between the binary hash codes of all video data in the video database for similarity matching, and the results are sorted from smallest to largest distance to obtain the query results.

[0004] Traditional retrieval methods mostly rely on simple keyword tags or metadata indexes, which are difficult to deeply understand the complex and diverse search intentions of users. Especially when faced with professional terminology, domain-specific vocabulary, and complex semantic scenarios, they cannot accurately match user needs with massive amounts of multimedia content, resulting in low search accuracy and frequent false positives and false negatives. At the same time, traditional retrieval methods only focus on a single modality of video or image, making it difficult to function in scenarios that require processing two modalities simultaneously. Furthermore, existing retrieval methods are slow when searching massive amounts of data, with long system response delays, which reduces the reliability of the retrieval system. Summary of the Invention

[0005] In view of this, the present invention proposes an image and video retrieval method based on a domain-fine-tuned large language model, which can effectively solve the recognition bias that may occur when the general model processes proper nouns and professional terms, and improve the retrieval accuracy and response speed, thereby enhancing the accuracy and professional adaptability of the retrieval.

[0006] The technical solution of this invention is implemented as follows: Firstly, this invention provides an image and video retrieval method based on a domain-fine-tuned large language model, comprising the following steps:

[0007] S1. Obtain the user query text, fine-tune the pre-trained model using low-rank decomposition and regularization constraints to obtain the fine-tuned pre-trained model, and perform intent classification and keyword extraction on the user query text to obtain the user intent label and keyword list.

[0008] S2, construct the prompt word sequence and initialize the prompt template. Using Monte Carlo tree search combined with a Hidden Markov Model (HMM) and mutual information as the optimization objective, dynamically iterate to generate and select the optimal prompt template.

[0009] S3. Construct a set of noisy words, filter the keyword list based on similarity calculation, and predict the category label of the filtered keywords by combining a conditional random field model with univariate and binary potential functions to obtain the keyword enhancement set.

[0010] S4: Combine the keywords in the keyword enhancement set according to Boolean logic and combine them with the user intent to generate query conditions. Use a distributed inverted index to execute the query conditions in parallel to obtain a candidate resource set.

[0011] S5: Based on the user query text and the candidate resource set, calculate the similarity between the user query text and each candidate resource, and combine the optimal prompt template to generate attention weight calculation, obtain the matching score between the user query and each candidate resource, obtain the resource path with the highest matching score, and obtain the retrieved image or video.

[0012] Based on the above technical solutions, preferably, the step S1 of fine-tuning the pre-trained model using low-rank decomposition and regularization constraints to obtain the fine-tuned pre-trained model includes the following sub-steps:

[0013] S11: Extract multidimensional parameter vectors from the attention subspace of the pre-trained model and calculate cosine similarity pairwise to generate a symmetric similarity matrix.

[0014] S12, normalize the symmetric similarity matrix to obtain the normalized symmetric similarity matrix;

[0015] S13: Preset the range of candidate cluster numbers, perform eigenvalue decomposition on the normalized symmetric similarity matrix, extract the feature vectors of the top candidate clusters, perform k-means clustering, calculate the average distance between each data point and other data points in the same cluster, obtain the silhouette coefficient of each candidate cluster number, determine the cluster number corresponding to the largest silhouette coefficient value as the optimal cluster number, and generate the final cluster grouping based on the optimal cluster number, and output parameter grouping labels;

[0016] S14. Construct an inter-group mask matrix based on the parameter grouping labels, and combine the spectral norm constraint to constrain the low rank of intra-group parameters and the sparsity constraint to sparsify the inter-group connections to construct a regularized loss function.

[0017] S15. Construct a projection mask matrix based on the parameter grouping labels and the symmetric similarity matrix, perform SVD decomposition on the attention weight matrix, extract singular values ​​and corresponding vectors, and calculate the projection update matrix.

[0018] S16. Construct the objective function based on the regularization loss function and the task loss function of intent classification, and use the projection update matrix to train and update the low-rank matrix to obtain the fine-tuned pre-trained model.

[0019] Based on the above technical solutions, preferably, step S2, which involves constructing the prompt word sequence and initializing the prompt template, uses Monte Carlo tree search combined with a Hidden Markov Model (HMM) to dynamically iterate and filter the optimal prompt template with mutual information as the optimization objective. This includes the following sub-steps:

[0020] S21, construct a prompt word sequence using preset prompt words and define the target output probability distribution;

[0021] S22, initialize the Monte Carlo search tree, load the initial hint template set, download the HMM parameter triplet, the HMM parameter triplet includes the state transition matrix, the observation emission matrix and the initial state distribution, and attach the initial hint template as a child node of the root node;

[0022] S23, calculate the UCT index of the current node, move down along the path of the maximum UCT index, select the node that has not been expanded or whose access count is less than the access threshold, and obtain the prompt template corresponding to the node.

[0023] S24, based on the obtained current prompt template, extract the hidden state sequence from the initial state distribution, generate candidate word modification operations based on the observation emission matrix, generate a new prompt template for each modification operation, create child nodes for each new prompt template, and initialize the access count and reward;

[0024] S25, perform HMM decoding on each created child node to generate an output sequence, and calculate the mutual information as the simulation reward value for each created child node;

[0025] S26, the simulation report back-updates the UCT index of all nodes on the path, repeats the iteration until the predetermined number of iterations is reached, and selects the prompt template with the highest average mutual information from the direct child nodes of the root node as the optimal prompt template for output.

[0026] Based on the above technical solutions, preferably, step S3 involves constructing a noisy word set, filtering the keyword list based on similarity calculation, and predicting the category labels of the filtered keywords using a conditional random field model combined with univariate and binary potential functions to obtain a keyword enhancement set. This includes the following sub-steps:

[0027] S31, Perform vector processing on each word in the keyword list to generate a corresponding semantic vector;

[0028] S32, predefine the noise word set and filtering threshold, calculate the maximum similarity between the semantic vector corresponding to each word in the keyword list and all noise words in the noise word set, and filter according to the filtering threshold to obtain the preliminary keyword set;

[0029] S33, define corresponding video or image tags for each keyword in the initial keyword set, and construct univariate potential functions and binary potential functions;

[0030] S34. Construct a conditional random field (CRF) probability model based on the univariate potential function and the binary potential function. Input the initial keyword set into the CRF probability model, output the probability of predicting the category label corresponding to each keyword, and retain keywords with a probability greater than the probability threshold and remove keywords with a binary potential less than 0 to obtain the keyword enhancement set.

[0031] Based on the above technical solutions, preferably, step S4, which involves combining keywords from the keyword enhancement set according to Boolean logic and combining them with user intent to generate query conditions, and then using a distributed inverted index to execute the query conditions in parallel to obtain a candidate resource set, includes: obtaining user intent tags and keyword enhancement sets; combining keywords from the keyword enhancement set according to Boolean logic and concatenating them with user intent tags to generate query conditions; executing the query conditions in parallel using a distributed inverted index, processing the query simultaneously in multiple resource ID storage areas in the database, merging the return structures of each area, and removing duplicate IDs to obtain a resource ID set; and obtaining the storage path of the corresponding resource based on the resource ID set to obtain a candidate resource set.

[0032] Based on the above technical solutions, preferably, step S5 involves calculating the similarity between the user query text and each candidate resource based on the user query text and the candidate resource set, and combining this with the optimal prompt template to generate attention weight calculations, obtaining the matching score between the user query and each candidate resource, obtaining the resource path with the highest matching score, and obtaining the retrieved image or video. This includes the following steps:

[0033] S51, Input the user query text and the resource data in the candidate resource set into the fine-tuning pre-trained model respectively to obtain the query vector and the candidate resource vector;

[0034] S52, calculate the similarity between the query vector and each candidate resource vector to obtain the semantic similarity score;

[0035] S53, input the optimal prompt template into the fine-tuned pre-trained model to obtain the attention weight vector;

[0036] S54: Based on the semantic similarity score and the attention weight vector, the matching score between the user query and each candidate resource is calculated, the resource path with the highest matching score is obtained, and the retrieved image or video is obtained.

[0037] Based on the above technical solution, preferably, step S5 further includes calculating the matching confidence score for each search result based on the matching score, and setting a preset trigger threshold. If the matching confidence score is less than the preset trigger threshold, a hybrid search is triggered to obtain the matching score between the candidate resource set and the keyword. The resource type is filtered according to the user intent tag, and a comprehensive score is calculated based on the resource creation time decay coefficient. The resource path with the highest score is obtained by sorting according to the comprehensive score, and the retrieved image or video is obtained.

[0038] Secondly, the present invention also provides an image and video retrieval system based on a domain-fine-tuned large language model, implemented using an image and video retrieval method based on a domain-fine-tuned large language model. The system includes:

[0039] The parameter tuning module is used to obtain user query text, fine-tune the pre-trained model using low-rank decomposition and regularization constraints to obtain the fine-tuned pre-trained model, and perform intent classification and keyword extraction on user query text to obtain user intent labels and keyword lists.

[0040] The prompt template generation module is used to construct prompt word sequences and initialize prompt templates. It dynamically generates and selects the optimal prompt template by combining Monte Carlo tree search with a Hidden Markov Model (HMM) with mutual information as the optimization objective.

[0041] The filtering module is used to construct a set of noisy words, filter the keyword list based on similarity calculation, and predict the category label of the filtered keywords by combining univariate and binary potential functions through a conditional random field model, thus obtaining the keyword enhancement set.

[0042] The filtering module is used to combine keywords in the keyword enhancement set according to Boolean logic and combine them with user intent to generate query conditions. The query conditions are executed in parallel using a distributed inverted index to obtain a candidate resource set.

[0043] The matching module is used to calculate the similarity between the user query text and each candidate resource based on the user query text and the candidate resource set. It also combines the optimal prompt template to generate attention weight calculation, obtain the matching score between the user query and each candidate resource, obtain the resource path with the highest matching score, and obtain the retrieved image or video.

[0044] Thirdly, the present invention also provides a terminal device, the terminal device comprising: a memory, a processor, and an image and video retrieval method program based on a domain-fine-tuned large language model stored in the memory and executable on the processor, the image and video retrieval method program based on a domain-fine-tuned large language model being configured to implement the steps of an image and video retrieval method based on a domain-fine-tuned large language model.

[0045] Fourthly, the present invention also provides a storage medium, which is a computer storage medium, storing an image and video retrieval method program based on a domain-fine-tuned large language model. When the image and video retrieval method program based on a domain-fine-tuned large language model is executed by a processor, it implements the steps of the image and video retrieval method based on a domain-fine-tuned large language model.

[0046] The image and video retrieval method based on a domain-fine-tuned large language model of the present invention has the following advantages over existing technologies:

[0047] (1) By using low-rank decomposition and regularization to fine-tune the pre-trained model, user intent is accurately identified and keywords are extracted. The optimal prompt template is generated dynamically by combining Monte Carlo tree search and HMM. Noise words are filtered and keyword categories are predicted by using conditional random fields. Then, through parallel querying of distributed inverted index and matching based on similarity and attention weight, the identification bias that may occur when general models deal with proper nouns and professional terms is effectively solved, thereby improving the accuracy and professional adaptability of retrieval.

[0048] (2) By combining Monte Carlo tree search with Hidden Markov Model and using mutual information as the evaluation index, the optimal prompt template can be searched efficiently, improving the performance of related tasks, while having low resource consumption and improving efficiency.

[0049] (3) By constructing a probability model based on semantic analysis and conditional random field (CRF), it can effectively filter noise words and accurately predict the category labels of keywords, generating a high-quality keyword enhancement set;

[0050] (4) By introducing the optimal prompt template in the matching process to generate the attention weight vector and weighting the matching score, the key terms gain a higher influence in the similarity calculation, further distinguishing fine-grained semantic differences and improving the accuracy of matching. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart of the image and video retrieval method based on a domain-fine-tuned large language model according to the present invention;

[0053] Figure 2 This is a schematic diagram showing the performance comparison of the image and video retrieval method based on a domain-fine-tuned large language model according to the present invention. Detailed Implementation

[0054] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0055] like Figure 1-2 As shown, in a first aspect, the present invention provides an image and video retrieval method based on a domain-fine-tuned large language model, comprising the following steps:

[0056] S1. Obtain the user query text, fine-tune the pre-trained model using low-rank decomposition and regularization constraints to obtain the fine-tuned pre-trained model, and perform intent classification and keyword extraction on the user query text to obtain the user intent label and keyword list.

[0057] Step S1 in this embodiment includes the following sub-steps:

[0058] S11, extract the multidimensional parameter vectors of the attention subspace from the pre-trained model, and calculate the cosine similarity pairwise to generate a symmetric similarity matrix, the expression of which is:

[0059]

[0060] In the formula, A i，j Let A be a symmetric similarity matrix, and A ∈ R. d×d sim(i,j) is the cosine similarity value between the i-th d-dimensional parameter vector and the j-th d-dimensional parameter vector. If the similarity is greater than 0.5, it is retained; otherwise, it is 0.

[0061] S12, normalize the symmetric similarity matrix to obtain the normalized symmetric similarity matrix, expressed as:

[0062]

[0063] In the formula, L norm Let A be the normalized Laplace matrix, D be the diagonal matrix, and A be the symmetric similarity matrix.

[0064] S13, preset the range of candidate cluster numbers, perform eigenvalue decomposition on the normalized symmetric similarity matrix, extract the feature vectors of the top candidate clusters, perform k-means clustering, calculate the average distance between each data point and other data points in the same cluster, obtain the silhouette coefficient of each candidate cluster number, determine the cluster number corresponding to the largest silhouette coefficient value as the optimal cluster number, and generate the final cluster group based on the optimal cluster number, output the parameter group label G;

[0065]

[0066] In the formula, k is the number of candidate clusters, ranging from [5, 15], k* is the optimal number of clusters, s(i) is the silhouette coefficient of sample i, s(i) = (b(i) - a(i)) / (max(a(i), b(i))), a(i) is the average distance between sample i and other samples in the same cluster, b(i) is the average distance between sample i and the nearest sample in a different cluster, and n k The total number of samples when the number of clusters is k;

[0067] S14. Construct an inter-group mask matrix based on the parameter grouping labels, and combine the spectral norm constraint to constrain the low rank of intra-group parameters and the sparsity constraint to sparsify the inter-group connections to construct a regularized loss function.

[0068] Construct an inter-group mask matrix O based on the parameter grouping label G. If the two obtained parameters belong to the same group, the value is 0; if the two obtained parameters belong to different groups, the value is 1.

[0069] The regularization loss function is expressed as follows:

[0070]

[0071] In the formula, L reg For regularization loss, A and B are both low-rank matrices, with the rank r of A and B set to 8, the scaling factor a set to 32, the dropout rate set to 0.1, and the number of training epochs set to 3. Here, rank r is a core parameter of LoRA, defining the middle dimension of the low-rank decomposition matrix added to the original weight matrix. The scaling factor a balances the update amount in each training epoch, and dropout prevents overfitting. O is the between-group mask matrix, ||·|| F Let λ be the spectral norm, ||·||1 be the sparsity constraint, and λ1 and λ2 be the regularization coefficients.

[0072] S15, construct a projection mask matrix based on the parameter grouping labels and the symmetric similarity matrix, perform SVD decomposition on the attention weight matrix, extract singular values ​​and their corresponding vectors, and calculate the projection update matrix, expressed as:

[0073]

[0074] In the formula, ΔW is the projection update matrix, σ i For the first r significant singular values, u i Let v be a left singular vector. i Let M be the right singular vector, and M be the projection mask matrix;

[0075] S16. Construct the objective function based on the regularization loss function and the task loss function of intent classification, and use the projection update matrix to train and update the low-rank matrix to obtain the fine-tuned pre-trained model.

[0076]

[0077] In the formula, θ * Here, θ represents the optimal model parameters, and D represents the trainable parameters. finetune For the training dataset, (x, y) is a tuple where x is the user query for this text and y is the intent label; f(x, θ) is the prediction function for fine-tuning the model.

[0078] The projection update matrix is ​​injected into the attention layer to update the parameters of the low-rank matrices A and B, as expressed by:

[0079]

[0080] In the formula, W new For the updated attention layer weight matrix, W pre These are the original attention layer weights of the pre-trained model.

[0081] It should be noted that in this embodiment, the GLM-9B model, which has been fine-tuned by LoRA technology, is used for intent judgment and keyword extraction. The LLM's ability to understand user intent is used to classify user needs into one or more categories such as videos or images, summarize clear keywords, and specify that they be output in JSON format.

[0082] By adding a regularization term to the loss function, the parameters of the low-rank matrix are ensured to remain sparse or have a low norm, thus avoiding excessive interference with the pre-trained weights. Since the attention layer and MLP layer are most sensitive to task-adaptive adjustments, and the matrices of the attention layer and MLP layer are both high-dimensional dense matrices, low-rank decomposition can significantly reduce the number of trainable parameters while maintaining performance. Furthermore, by injecting low-rank matrices only into the attention layer and MLP layer, the main parameters of the GLM-9B model are frozen, and only a small number of weight modules are optimized for the intent judgment and keyword extraction tasks, thereby reducing the consumption of computing power and hardware resources.

[0083] After fine-tuning, during inference, the local pre-trained language model is loaded, and the model is initialized by combining the pre-configured LoRA local file and weights. This ensures that the model can efficiently recognize the user's input intent and extract keywords. The output generated by this process is represented in JSON format, which clearly contains the user intent and the list of extracted keywords, providing accurate data support for subsequent processing.

[0084] S2: Construct the prompt word sequence and initialize the prompt template. Using Monte Carlo tree search combined with a Hidden Markov Model (HMM) and mutual information as the optimization objective, dynamically iterate to generate and select the optimal prompt template.

[0085] Step S2 includes the following sub-steps:

[0086] S21, construct a prompt word sequence using preset prompt words, and define the target output probability distribution;

[0087] S22, initialize the Monte Carlo search tree, load the initial hint template set, download the HMM parameter triplet, the HMM parameter triplet includes the state transition matrix, the observation emission matrix and the initial state distribution, and attach the initial hint template as a child node of the root node;

[0088] S23, calculate the UCT index of the current node, move down along the path of the maximum UCT index, select the node that has not been expanded or whose access count is less than the access threshold, and obtain the prompt template corresponding to the node.

[0089] S24, based on the obtained current prompt template, extract the hidden state sequence from the initial state distribution, generate candidate word modification operations based on the observation emission matrix, generate a new prompt template for each modification operation, create child nodes for each new prompt template, and initialize the access count and reward;

[0090] S25, perform HMM decoding on each created child node to generate an output sequence, and calculate the mutual information as the simulation reward value for each created child node, expressed as:

[0091]

[0092] In the formula, Y is the simulated return value. (m) Let p(·) be the output sequence generated by HMM decoding under prompt template P in the m-th simulation experiment, where p(·) is the generation probability, M is the number of simulation experiments, and P' is the new prompt template.

[0093] S26, the simulation report back-updates the UCT index of all nodes on the path, repeats the iteration until the predetermined number of iterations is reached, and selects the prompt template with the highest average mutual information from the direct child nodes of the root node as the optimal prompt template for output.

[0094] It should be noted that the UCT metric in Monte Carlo Tree Search can strike a balance between exploring new prompt templates and utilizing known effective templates. The UCT metric comprehensively considers the number of node visits and the reward value, making full use of already discovered high-reward templates, thus searching for the optimal prompt template more efficiently. Furthermore, by utilizing the hidden state transition mechanism and observation emission matrix of HMM, diverse candidate word modification operations can be generated, thereby creating new prompt templates. It can capture the latent structure and patterns in language, generating prompts that are more in line with semantic and grammatical rules, improving the quality and effectiveness of prompts. Using mutual information as the simulation reward value can measure the degree of correlation between the prompt template and the target output. The higher the mutual information, the more accurately the prompt template can guide the model to generate outputs that conform to the target distribution, thus facilitating the selection of the optimal prompt template and improving the performance of tasks such as intent recognition and keyword extraction. At the same time, by performing small-scale variant operations and searches based on existing prompt templates, it avoids generating a large number of prompt templates from scratch, reducing the consumption of computational resources and time. And by performing detailed HMM decoding and mutual information calculation only on a portion of the nodes, efficiency is further improved.

[0095] In this embodiment, by combining MCTS and HMM and using mutual information as an evaluation metric, the optimal prompt template can be searched efficiently, improving the performance of related tasks while having low resource consumption.

[0096] S3. Construct a noisy word set, filter the keyword list based on similarity calculation, and predict the category label of the filtered keywords by combining univariate and binary potential functions through a conditional random field model, thus obtaining the keyword enhancement set.

[0097] Step S3 includes the following sub-steps:

[0098] S31, Perform vector processing on each word in the keyword list to generate a corresponding semantic vector;

[0099] S32, predefine the noise word set and filtering threshold, calculate the maximum similarity between the semantic vector corresponding to each word in the keyword list and all noise words in the noise word set, and filter according to the filtering threshold to obtain the preliminary keyword set, expressed as:

[0100]

[0101] In the formula, C is the initial keyword set, and w i Let be the i-th word in the keyword list, c be the noise word in the noise word set, and τ1 be the filtering threshold, τ1 = 0.7, used to determine whether a keyword is a noise word;

[0102] S33, define corresponding video or image tags for each keyword in the initial keyword set, and construct univariate potential functions and binary potential functions;

[0103] The expression for the univariate potential function is:

[0104]

[0105] In the formula, ψ(y i ,w i Let w be a univariate potential function, representing the keyword w at position i. i With tag y i The degree of correlation between them, b yi For the tag y i Related bias, u yi For the tag y i Relevant weights;

[0106] The expression for the binary potential function is:

[0107]

[0108] In the formula, Let y be a binary potential function, expressed as the label y at adjacent positions i and j. i and y j The dependency relationship between them, e(y) i ) and e(y j ) are labels y i and y j The corresponding embedding vector;

[0109] S34. Construct a Conditional Random Field (CRF) probabilistic model based on the univariate and binary potential functions. Input the initial keyword set into the CRF probabilistic model, output the probability of predicting the category label corresponding to each keyword, and retain keywords with probabilities greater than the probability threshold and remove those with binary potentials less than 0 to obtain the keyword enhancement set. The expression of the CRF probabilistic model is as follows:

[0110]

[0111] In the formula, P(Y|X) is the conditional random field (CRF) probability model, representing the probability of the tag sequence Y given the keyword sequence X, and Z is the normalization factor.

[0112] It should be noted that by filtering out noisy words from the keyword list and utilizing contrastive learning-based semantic embedding and spectral clustering methods, the semantic features of words can be captured more precisely, improving the accuracy of noise word detection and thus enhancing the quality of subsequent keyword classification. Furthermore, employing a conditional random field model combined with univariate and binary potential functions fully considers the dependencies between labels. The univariate potential function reflects the degree of association between a single keyword and a specific label, while the binary potential function captures the constraint relationships between adjacent keyword labels, making the predicted labels more coherent and accurate, thus improving the accuracy of keyword category prediction. This achieves joint optimization of noise word elimination and category alignment. Simultaneously, by setting probability thresholds and removing keywords with a binary potential less than 0, the quality of the keyword augmentation set can be guaranteed. Only high-probability keywords and reasonable label relationships are retained, reducing the impact of uncertainty and erroneous labels and improving the reliability of the keyword augmentation set.

[0113] In this embodiment, a semantic analysis and Conditional Random Field (CRF) probability model are used to effectively filter out noisy words and accurately predict the category labels of keywords, generating a high-quality keyword enhancement set.

[0114] S4 combines the keywords in the keyword enhancement set according to Boolean logic and combines them with the user intent to generate query conditions. The query conditions are then executed in parallel using a distributed inverted index to obtain a candidate resource set.

[0115] Step S4 includes the following sub-steps:

[0116] S41, obtain the user intent tag and keyword enhancement set, combine the keywords in the keyword enhancement set according to Boolean logic and concatenate them with the user intent tag to generate query conditions;

[0117] S42, based on the query conditions, the distributed inverted index is used to execute the query conditions in parallel, and the query is processed simultaneously in multiple resource ID storage areas in the database. The return structures of each area are merged and duplicate IDs are removed to obtain a set of resource IDs.

[0118] S43, obtain the storage path of the corresponding resource based on the resource ID set, and obtain the candidate resource set.

[0119] It should be noted that using a distributed inverted index to execute query conditions in parallel can make full use of the computing resources of the distributed system and process queries simultaneously in multiple resource ID storage areas of the database; this greatly shortens the query time and improves query efficiency, and is especially suitable for fast retrieval of large-scale datasets.

[0120] S5: Based on the user query text and the candidate resource set, calculate the similarity between the user query text and each candidate resource, and combine the optimal prompt template to generate attention weight calculation, obtain the matching score between the user query and each candidate resource, obtain the resource path with the highest matching score, and obtain the retrieved image or video.

[0121] Step S5 includes the following steps:

[0122] S51, Input the user query text and the resource data in the candidate resource set into the fine-tuning pre-trained model respectively to obtain the query vector and the candidate resource vector;

[0123] S52, calculate the similarity between the query vector and each candidate resource vector to obtain the semantic similarity score, expressed as:

[0124]

[0125] In the formula, sim(q,r) is the semantic similarity score, q is the query vector, and r is the candidate resource vector;

[0126] S53, input the optimal prompt template into the fine-tuned pre-trained model to obtain the attention weight vector;

[0127] S54: Based on the semantic similarity score and the attention weight vector, the matching score between the user query and each candidate resource is calculated, the resource path with the highest matching score is obtained, and the retrieved image or video is obtained.

[0128] It should be noted that by fine-tuning the pre-trained model to map user query text and candidate resources into vectors and calculating semantic similarity scores, the similarity between user queries and candidate resources can be accurately measured at the semantic level, improving the accuracy of retrieval. Introducing the optimal prompt template to generate attention weight vectors and calculating matching scores by weighting them makes key terms have a higher influence in similarity calculation, further distinguishing fine-grained semantic differences and improving the accuracy of matching.

[0129] Step S5 also includes calculating the matching confidence score for each search result based on the matching score, and setting a preset trigger threshold. If the matching confidence score is less than the preset trigger threshold, a hybrid search is triggered to obtain the matching score between the candidate resource set and the keyword. The resource type is filtered according to the user intent tag, and a comprehensive score is calculated based on the resource creation time decay coefficient. The resource path with the highest score is obtained by sorting according to the comprehensive score, and the retrieved image or video is obtained.

[0130] The expression for calculating the matching confidence score is:

[0131] S=σ(α·TF-IDF max +β·max{sim(q,r)})

[0132] In the formula, S is the matching confidence score, σ is the sigmoid function, α and β are empirically tuned weights, and TF-IDF is the finite element density function. max is the highest keyword matching score in the candidate resource set, and max{sim(q,r)} is the highest semantic similarity score in the candidate resource set.

[0133] When S < 0.7, the retrieval quality is deemed insufficient, triggering a hybrid retrieval mode. The TF-IDF matching score between the candidate resource set and the keywords is calculated, and the overall score is obtained by comprehensively calculating the resource creation time decay coefficient. The expression is:

[0134] Z1=W1*TF-IDF+W2*1 / (t n -t o )

[0135] In the formula, Z1 is the overall score, W1 and W2 are weighting coefficients, TF-IDF is the matching score between the candidate resource set and the keyword, and t n t represents the current time. o Creation time.

[0136] It should be noted that when the matching confidence score is lower than the preset trigger threshold, the hybrid retrieval mechanism is triggered. By combining semantic matching score, user intent tag filtering, and resource creation time decay coefficient to calculate a comprehensive score, a relatively suitable resource can be found by comprehensively considering multiple factors when the semantic matching effect is poor, thus enhancing the robustness of the retrieval.

[0137] Secondly, the present invention also provides an image and video retrieval system based on a domain-fine-tuned large language model, implemented using an image and video retrieval method based on a domain-fine-tuned large language model. The system includes:

[0138] The parameter tuning module is used to obtain user query text, fine-tune the pre-trained model using low-rank decomposition and regularization constraints to obtain the fine-tuned pre-trained model, and perform intent classification and keyword extraction on user query text to obtain user intent labels and keyword lists.

[0139] The prompt template generation module is used to construct prompt word sequences and initialize prompt templates. It dynamically generates and selects the optimal prompt template by combining Monte Carlo tree search with a Hidden Markov Model (HMM) with mutual information as the optimization objective.

[0140] The filtering module is used to construct a set of noisy words, filter the keyword list based on similarity calculation, and predict the category label of the filtered keywords by combining univariate and binary potential functions through a conditional random field model, thus obtaining the keyword enhancement set.

[0141] The filtering module is used to combine keywords in the keyword enhancement set according to Boolean logic and combine them with user intent to generate query conditions. The query conditions are executed in parallel using a distributed inverted index to obtain a candidate resource set.

[0142] The matching module is used to calculate the similarity between the user query text and each candidate resource based on the user query text and the candidate resource set. It also combines the optimal prompt template to generate attention weight calculation, obtain the matching score between the user query and each candidate resource, obtain the resource path with the highest matching score, and obtain the retrieved image or video.

[0143] It should be noted that this system is the same as the image and video retrieval method based on the domain fine-tuning large language model described above. All implementation methods in the above method embodiments are applicable to the embodiments of this system and can achieve the same technical effect.

[0144] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0145] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0146] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0147] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0149] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0150] Furthermore, it should be noted that in the system and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of the present invention. Moreover, the steps performing the above series of processes can naturally be executed in the order described, but are not necessarily required to be executed in chronological order; some steps can be executed in parallel or independently of each other. Those skilled in the art will understand that all or any step or component of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software, or a combination thereof. This is something that those skilled in the art can achieve by using their basic programming skills after reading the description of the present invention.

[0151] Therefore, the object of the present invention can also be achieved by running a program or a set of programs on any computing system. The computing system can be a known general-purpose system. Therefore, the object of the present invention can also be achieved simply by providing a program product containing program code implementing the method or apparatus. That is, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any known storage medium or any storage medium developed in the future. It should also be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent to the present invention. Furthermore, the steps performing the above series of processes can naturally be performed in the order described, but are not necessarily required to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image and video retrieval method based on a domain-fine-tuned large language model, characterized in that, Includes the following steps: S1. Obtain the user query text, fine-tune the pre-trained model using low-rank decomposition and regularization constraints to obtain the fine-tuned pre-trained model, and perform intent classification and keyword extraction on the user query text to obtain the user intent label and keyword list. S2, construct the prompt word sequence and initialize the prompt template. Using Monte Carlo tree search combined with a Hidden Markov Model (HMM) and mutual information as the optimization objective, dynamically iterate to generate and select the optimal prompt template. Step S2 includes the following sub-steps: S21, construct a prompt word sequence using preset prompt words, and define the target output probability distribution; S22, initialize the Monte Carlo search tree, load the initial hint template set, download the HMM parameter triplet, the HMM parameter triplet includes the state transition matrix, the observation emission matrix and the initial state distribution, and attach the initial hint template as a child node of the root node; S23, calculate the UCT index of the current node, move down along the path of the maximum UCT index, select the node that has not been expanded or whose access count is less than the access threshold, and obtain the prompt template corresponding to the node. S24, based on the obtained current prompt template, extract the hidden state sequence from the initial state distribution, generate candidate word modification operations based on the observation emission matrix, generate a new prompt template for each modification operation, create child nodes for each new prompt template, and initialize the access count and reward; S25, perform HMM decoding on each created child node to generate an output sequence, and calculate the mutual information as the simulation reward value for each created child node; S26, update the UCT index of all nodes on the simulation return backtracking path, repeat the iteration until the predetermined number of iterations is reached, and select the prompt template with the highest average mutual information from the direct child nodes of the root node as the optimal prompt template for output. S3. Construct a noisy word set, filter the keyword list based on similarity calculation, and predict the category label of the filtered keywords by combining a conditional random field model with univariate and binary potential functions to obtain the keyword enhancement set. S4: Combine the keywords in the keyword enhancement set according to Boolean logic and combine them with the user intent to generate query conditions. Use a distributed inverted index to execute the query conditions in parallel to obtain a candidate resource set. S5: Based on the user query text and the candidate resource set, calculate the similarity between the user query text and each candidate resource, and combine the optimal prompt template to generate attention weight calculation, obtain the matching score between the user query and each candidate resource, obtain the resource path with the highest matching score, and obtain the retrieved image or video.

2. The image and video retrieval method based on a domain-fine-tuned large language model as described in claim 1, characterized in that, Step S1, which involves fine-tuning the pre-trained model using low-rank decomposition and regularization constraints to obtain a fine-tuned pre-trained model, includes the following sub-steps: S11: Extract multi-dimensional parameter vectors from the attention subspace of the pre-trained model and calculate cosine similarity pairwise to generate a symmetric similarity matrix. S12, normalize the symmetric similarity matrix to obtain the normalized symmetric similarity matrix; S13: Preset the range of candidate cluster numbers, perform eigenvalue decomposition on the normalized symmetric similarity matrix, extract the feature vectors of the top candidate clusters, perform k-means clustering, calculate the average distance between each data point and other data points in the same cluster, obtain the silhouette coefficient of each candidate cluster number, determine the cluster number corresponding to the largest silhouette coefficient value as the optimal cluster number, and generate the final cluster grouping based on the optimal cluster number, and output parameter grouping labels; S14. Construct an inter-group mask matrix based on the parameter grouping labels, and combine the spectral norm constraint to constrain the low rank of intra-group parameters and the sparsity constraint to sparsify the inter-group connections to construct a regularized loss function. S15. Construct a projection mask matrix based on the parameter grouping labels and the symmetric similarity matrix, perform SVD decomposition on the attention weight matrix, extract singular values ​​and corresponding vectors, and calculate the projection update matrix. S16. Construct the objective function based on the regularization loss function and the task loss function of intent classification, and use the projection update matrix to train and update the low-rank matrix to obtain the fine-tuned pre-trained model.

3. The image and video retrieval method based on a domain-fine-tuned large language model as described in claim 2, characterized in that, Step S3, which involves constructing a noisy word set, filtering the keyword list based on similarity calculations, and predicting the category labels of the filtered keywords using a conditional random field model combined with univariate and binary potential functions to obtain the keyword enhancement set, includes the following sub-steps: S31, Perform vector processing on each word in the keyword list to generate a corresponding semantic vector; S32, predefine the noise word set and filtering threshold, calculate the maximum similarity between the semantic vector corresponding to each word in the keyword list and all noise words in the noise word set, and filter according to the filtering threshold to obtain the preliminary keyword set; S33, define corresponding video or image tags for each keyword in the initial keyword set, and construct univariate potential functions and binary potential functions; S34. Construct a conditional random field (CRF) probability model based on the univariate potential function and the binary potential function. Input the initial keyword set into the CRF probability model, output the probability of predicting the category label corresponding to each keyword, and retain keywords with a probability greater than the probability threshold and remove keywords with a binary potential less than 0 to obtain the keyword enhancement set.

4. The image and video retrieval method based on a domain-fine-tuned large language model as described in claim 3, characterized in that, Step S4 involves combining keywords from the keyword enhancement set using Boolean logic and combining them with user intent to generate query conditions. A distributed inverted index is then used to execute the query conditions in parallel to obtain a candidate resource set. This process includes: obtaining user intent tags and the keyword enhancement set; combining keywords from the keyword enhancement set using Boolean logic and concatenating them with the user intent tags to generate query conditions; executing the query conditions in parallel using a distributed inverted index, processing the query simultaneously in multiple resource ID storage areas in the database, merging the returned structures from each area, and removing duplicate IDs to obtain a resource ID set; and obtaining the storage path of the corresponding resource based on the resource ID set to obtain the candidate resource set.

5. The image and video retrieval method based on a domain-fine-tuned large language model as described in claim 4, characterized in that: Step S5, which involves calculating the similarity between the user query text and each candidate resource based on the user query text and the candidate resource set, and combining this with the optimal prompt template to generate attention weights, calculates the matching score between the user query and each candidate resource, obtains the resource path with the highest matching score, and finally retrieves the image or video, includes the following steps: S51, Input the user query text and the resource data in the candidate resource set into the fine-tuning pre-trained model respectively to obtain the query vector and the candidate resource vector; S52, calculate the similarity between the query vector and each candidate resource vector to obtain the semantic similarity score; S53, input the optimal prompt template into the fine-tuned pre-trained model to obtain the attention weight vector; S54: Based on the semantic similarity score and the attention weight vector, the matching score between the user query and each candidate resource is calculated, the resource path with the highest matching score is obtained, and the retrieved image or video is obtained.

6. The image and video retrieval method based on a domain-fine-tuned large language model as described in claim 5, characterized in that: Step S5 also includes calculating the matching confidence score for each search result based on the matching score, and setting a preset trigger threshold. If the matching confidence score is less than the preset trigger threshold, a hybrid search is triggered to obtain the matching score between the candidate resource set and the keyword. The resource type is filtered according to the user intent tag, and a comprehensive score is calculated based on the resource creation time decay coefficient. The resource path with the highest score is obtained by sorting according to the comprehensive score, and the retrieved image or video is obtained.

7. An image and video retrieval system based on a domain-fine-tuned large language model, implemented using the image and video retrieval method based on a domain-fine-tuned large language model as described in any one of claims 1-6, characterized in that, The system includes: The parameter tuning module is used to obtain user query text, fine-tune the pre-trained model using low-rank decomposition and regularization constraints to obtain the fine-tuned pre-trained model, and perform intent classification and keyword extraction on user query text to obtain user intent labels and keyword lists. The prompt template generation module is used to construct prompt word sequences and initialize prompt templates. It dynamically generates and selects the optimal prompt template by combining Monte Carlo tree search with a Hidden Markov Model (HMM) with mutual information as the optimization objective. The filtering module is used to construct a set of noisy words, filter the keyword list based on similarity calculation, and predict the category label of the filtered keywords by combining univariate and binary potential functions through a conditional random field model, thus obtaining the keyword enhancement set. The filtering module is used to combine keywords in the keyword enhancement set according to Boolean logic and combine them with user intent to generate query conditions. The query conditions are executed in parallel using a distributed inverted index to obtain a candidate resource set. The matching module is used to calculate the similarity between the user query text and each candidate resource based on the user query text and the candidate resource set. It also combines the optimal prompt template to generate attention weight calculation, obtain the matching score between the user query and each candidate resource, obtain the resource path with the highest matching score, and obtain the retrieved image or video.

8. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and an image and video retrieval method program based on a domain-fine-tuned large language model stored in the memory and executable on the processor. The image and video retrieval method program based on a domain-fine-tuned large language model is configured to implement the steps of the image and video retrieval method based on a domain-fine-tuned large language model as claimed in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer storage medium, and the computer storage medium stores a program for an image and video retrieval method based on a domain-fine-tuned large language model. When the program for an image and video retrieval method based on a domain-fine-tuned large language model is executed by a processor, it implements the steps of the image and video retrieval method based on a domain-fine-tuned large language model as described in any one of claims 1 to 6.

Images