Image video retrieval method based on domain fine-tuning large language model

Abstract

The invention provides an image video retrieval method based on a domain fine-tuning large language model, which comprises the following steps: performing fine-tuning on a pre-training model to obtain a fine-tuning pre-training model for intention classification and keyword extraction; performing dynamic iteration screening on an optimal prompt template through Monte Carlo tree search in combination with a hidden Markov model (HMM); performing noise filtering on the keyword list, and predicting category labels of the filtered keywords through a conditional random field model to obtain a keyword enhancement set; combining with the user intention to generate a query condition, and obtaining a candidate resource set; and according to the similarity between the user query text and the candidate resource set, and in combination with the optimal prompt template, obtaining the resource path with the highest matching score between the user query and the candidate resource, and obtaining the retrieved image or video, so that the identification deviation possibly occurring when a general model processes proper nouns and terminologies can be effectively solved, and the user experience is improved. And the retrieval accuracy and response speed are improved, so that the retrieval accuracy and professional adaptability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence retrieval, and particularly relates to an image and video retrieval method based on domain fine-tuning of a large language model. BACKGROUND

[0002] With the rapid development of digital media technology, massive video and image data are widely produced and spread in the fields of social media, e-commerce, education and entertainment, etc. When users search for these contents, they often need to filter out resources highly matched with their needs from a large number of material libraries.

[0003] A video retrieval method based on deep learning and hash coding disclosed in patent CN107229757B uses a deep network to extract feature vectors of each frame image in a video sample to form a feature matrix, models the extracted video feature matrix as a whole to obtain a high-dimensional real value representation, constructs a target loss function, and constrains the ordering relationship among images and videos, videos and videos, and images and images. A hash function is realized through a nonlinear mapping operation of a network backend to map the feature representations of images and videos to a common Hamming space to obtain binary hash codes. The Hamming distance between the binary hash codes of the images and videos and all video data in a video database is calculated for similarity matching, and the query results are sorted in ascending order of distance.

[0004] Traditional retrieval modes mostly rely on simple keyword tags or metadata indexes, and it is difficult to deeply understand the complex and diverse retrieval intentions of users. Especially in the face of professional terms, domain characteristic words and complex semantic scenes, it is difficult to accurately match user needs and massive multimedia contents, resulting in low retrieval accuracy and frequent misdiagnosis and missed diagnosis. At the same time, traditional retrieval methods only focus on a single mode of video or image, and it is difficult to play a role in scenarios that need to process both modes. Moreover, the existing retrieval mode is slow in retrieval speed and long in system response delay when retrieving massive data, which reduces the reliability of the retrieval system. SUMMARY

[0005] Therefore, the present application provides an image and video retrieval method based on domain fine-tuning of a large language model, which can effectively solve the recognition deviation that may occur when a general model processes proper nouns and professional terms, and improve the retrieval accuracy and response speed, thereby improving the accuracy and professional adaptability of retrieval.

[0006] The technical scheme of the present application is implemented as follows: In a first aspect, the present application provides an image and video retrieval method based on domain fine-tuning of a large language model, comprising the following steps:

[0007] S1, obtain a user query text, fine-tune a pre-trained model by low-rank decomposition and regularization constraint to obtain a fine-tuned pre-trained model, and classify the user query text and extract keywords to obtain a user intent label and a keyword list;

[0008] S2, construct a prompt word sequence and initialize a prompt template, dynamically generate and screen an optimal prompt template by Monte Carlo tree search combined with a hidden Markov model (HMM) and taking mutual information as an optimization target;

[0009] S3, construct a noise word set, filter the keyword list according to similarity calculation, and predict the class label of the filtered keyword by a conditional random field model combined with a unary potential function and a binary potential function to obtain a keyword enhanced set;

[0010] S4, combine the keywords in the keyword enhanced set according to Boolean logic and combine them with the user intent to generate a query condition, perform the query condition in parallel by using a distributed inverted index to obtain a candidate resource set;

[0011] S5, calculate the similarity between the user query text and each candidate resource according to the user query text and the candidate resource set, and generate attention weight calculation combined with the optimal prompt template to obtain the matching score between the user query and each candidate resource, obtain the resource path with the highest matching score, and obtain the retrieved image or video.

[0012] On the basis of the above technical scheme, preferably, in step S1, the fine-tuned pre-trained model is obtained by fine-tuning the pre-trained model by low-rank decomposition and regularization constraint, comprising the following sub-steps:

[0013] S11, extract a multi-dimensional parameter vector of an attention subspace from the pre-trained model, and calculate the cosine similarity of each two to generate a symmetric similarity matrix;

[0014] S12, normalize the symmetric similarity matrix to obtain a normalized symmetric similarity matrix;

[0015] S13, preset the value range of the candidate cluster number, perform eigenvalue decomposition on the normalized symmetric similarity matrix, extract the first candidate cluster number of feature vectors for k-means clustering, calculate the average distance between each data point and other data points in the same cluster, obtain the silhouette coefficient of each candidate cluster number, determine the cluster number value corresponding to the maximum silhouette coefficient value as the best cluster number, and generate a final clustering group according to the optimal cluster number, and output a parameter grouping label;

[0016] S14, construct an inter-group mask matrix according to the parameter grouping label, and combine the spectral norm constraint of intra-group parameters with the sparsity constraint of inter-group connection to construct a regularization loss function;

[0017] S15, constructing a projection mask matrix according to the parameter grouping label and the symmetric similarity matrix, performing SVD decomposition on the attention weight matrix, extracting singular values and corresponding vectors, and calculating a projection update matrix;

[0018] S16, constructing an objective function according to a regularization loss function and a task loss function of intent classification, and training and updating the low-rank matrix by using the projection update matrix to obtain a fine-tuned pre-training model.

[0019] On the basis of the above technical scheme, preferably, in step S2, the prompt word sequence is constructed and the prompt template is initialized, a Monte Carlo tree search is combined with a hidden Markov (HMM) model, mutual information is taken as an optimization objective, and the optimal prompt template is dynamically iteratively generated and screened, including the following sub-steps:

[0020] S21, constructing a prompt word sequence by using a preset prompt word, and defining a target output probability distribution;

[0021] S22, initializing a Monte Carlo search tree, loading an initial prompt template set, downloading an HMM parameter triple, the HMM parameter triple including a state transition matrix, an observation emission matrix and an initial state distribution, and mounting the initial prompt template as a child node of a root node;

[0022] S23, calculating a UCT index of a current node, moving down along a maximum UCT index path, selecting a node that has not been expanded or has a visit count less than a visit threshold, and obtaining a prompt template corresponding to the node;

[0023] S24, extracting a hidden state sequence from the initial state distribution according to the obtained prompt template, generating a candidate word modification operation based on the observation emission matrix, generating a new prompt template for each modification operation, creating a child node for each new prompt template, initializing a visit count and a reward;

[0024] S25, generating an output sequence by performing HMM decoding on each created child node, and calculating mutual information as a simulation reward value of each created child node;

[0025] S26, updating the UCT index of all nodes on the path by backtracking the simulation reward, repeating iteration until a predetermined iteration number is reached, and selecting a prompt template with the highest average mutual information from direct child nodes of the root node as an optimal prompt template output.

[0026] On the basis of the above technical scheme, preferably, in step S3, the noise word set is constructed, the keyword list is filtered according to similarity calculation, and the class label of the filtered keyword is predicted by using a conditional random field model combined with a unary potential function and a binary potential function to obtain a keyword enhancement set, including the following sub-steps:

[0027] S31, vector processing is performed on each word in the keyword list to generate a corresponding semantic vector;

[0028] S32, a set of noise words and a filtering threshold are predefined, the maximum similarity of the semantic vector corresponding to each word in the keyword list to all noise words in the set of noise words is calculated, and filtering is performed according to the filtering threshold to obtain a preliminary keyword set;

[0029] S33, a corresponding video or picture label is defined for each keyword in the preliminary keyword set, and a unary potential function and a binary potential function are constructed;

[0030] S34, a conditional random field (CRF) probability model is constructed according to the unary potential function and the binary potential function, the preliminary keyword set is input into the conditional random field (CRF) probability model, the probability of predicting the category label corresponding to each keyword is output, and the keyword corresponding to the binary potential less than 0 is retained and removed, to obtain a keyword enhanced set.

[0031] On the basis of the above technical solutions, preferably, in step S4, the keywords in the keyword enhanced set are combined according to Boolean logic and combined with the user intent to generate a query condition, distributed inverted indexes are used to execute the query condition in parallel to obtain a candidate resource set, including: obtaining a user intent label and a keyword enhanced set, combining the keywords in the keyword enhanced set according to Boolean logic and juxtaposing the user intent label to generate a query condition; according to the query condition, distributed inverted indexes are used to execute the query condition in parallel, the query is processed in multiple resource ID storage areas in the database at the same time, the returned structures of each area are combined, and duplicate IDs are removed to obtain a resource ID set; according to the resource ID set, the storage path of the corresponding resource is obtained, and the candidate resource set is obtained.

[0032] On the basis of the above technical solutions, preferably, in step S5, according to the user query text and the candidate resource set, the similarity between the user query text and each candidate resource is calculated, and the attention weight calculation is generated by combining the optimal prompt template to obtain the matching score between the user query and each candidate resource, the resource path with the highest matching score is obtained, and the retrieved image or video is obtained, including the following steps:

[0033] S51, the user query text and the resource data in the candidate resource set are input into the fine-tuned pre-training model respectively to obtain a query vector and a candidate resource vector;

[0034] S52, the similarity between the query vector and each candidate resource vector is calculated to obtain a semantic similarity score;

[0035] S53, the optimal prompt template is input into the fine-tuned pre-training model to obtain an attention weight vector;

[0036] S54, obtaining a matching score between the user query and each candidate resource according to the semantic similarity score and the attention weight vector weighted calculation, obtaining the resource path with the highest matching score, and obtaining the retrieved image or video.

[0037] On the basis of the above technical scheme, preferably, step S5 further includes calculating a matching confidence according to the matching score for each retrieval result, and presetting a trigger threshold value, if the matching confidence is less than the preset trigger threshold value, triggering a hybrid retrieval, obtaining a matching score of the candidate resource set and the keyword, filtering the resource type according to the user intention label, and comprehensively calculating a comprehensive score according to a resource creation time decay coefficient, and obtaining the resource path with the highest score according to the comprehensive score sorting, and obtaining the retrieved image or video.

[0038] In a second aspect, the present application also provides an image and video retrieval system based on domain fine-tuning of a large language model, which is implemented by using an image and video retrieval method based on domain fine-tuning of a large language model, and the system comprises:

[0039] A parameter adjustment module is configured to obtain a user query text, fine-tune a pre-training model by using low-rank decomposition and regularization constraint to obtain a fine-tuned pre-training model, and perform intention classification and keyword extraction on the user query text to obtain a user intention label and a keyword list.

[0040] A prompt template generation module is configured to construct a prompt word sequence and initialize a prompt template, dynamically and iteratively generate and screen an optimal prompt template by using Monte Carlo tree search combined with a hidden Markov model (HMM) and taking mutual information as an optimization objective.

[0041] A filtering module is configured to construct a noise word set, filter the keyword list according to similarity calculation, and predict the class label of the filtered keyword by using a conditional random field model combined with a unary potential function and a binary potential function, to obtain a keyword enhanced set.

[0042] A screening module is configured to combine the keywords in the keyword enhanced set according to Boolean logic and combine them with the user intention to generate a query condition, perform the query condition by using distributed inverted index in parallel to obtain a candidate resource set.

[0043] A matching module is configured to calculate the similarity between the user query text and each candidate resource according to the user query text and the candidate resource set, and generate an attention weight calculation by combining the optimal prompt template to obtain a matching score between the user query and each candidate resource, obtain the resource path with the highest matching score, and obtain the retrieved image or video.

[0044] In a third aspect, the present application also provides a terminal device, which comprises a memory, a processor, and a program of the image and video retrieval method based on the large language model fine-tuned in a domain stored in the memory and executable on the processor, and the program of the image and video retrieval method based on the large language model fine-tuned in a domain is configured to implement the steps of the image and video retrieval method based on the large language model fine-tuned in a domain.

[0045] In a fourth aspect, the present application also provides a storage medium, which is a computer storage medium, and the program of the image and video retrieval method based on the large language model fine-tuned in a domain is stored on the computer storage medium, and the program of the image and video retrieval method based on the large language model fine-tuned in a domain is executed by the processor to implement the steps of the image and video retrieval method based on the large language model fine-tuned in a domain.

[0046] The image and video retrieval method based on the large language model fine-tuned in a domain of the present application has the following beneficial effects compared with the prior art:

[0047] (1) By accurately identifying user intent and extracting keywords through low-rank decomposition and regularization fine-tuning of the pre-trained model, combining Monte Carlo tree search and HMM to dynamically generate the optimal prompt template, using conditional random fields to filter noise words and predict keyword categories, and then performing parallel query through distributed inverted index and matching based on similarity and attention weight, the recognition deviation that may occur when a general model processes proper nouns and professional terms is effectively solved, thereby improving the accuracy and professional adaptability of the retrieval;

[0048] (2) By combining Monte Carlo tree search with hidden Markov model and using mutual information as an evaluation index, the optimal prompt template can be efficiently searched, the performance of related tasks is improved, and the efficiency is improved with lower resource consumption;

[0049] (3) By semantic analysis and conditional random field (CRF) probability model construction, noise words can be effectively filtered and the category labels of keywords can be accurately predicted to generate a high-quality keyword enhanced set;

[0050] (4) By introducing the optimal prompt template to generate an attention weight vector in the matching process and weighting the matching score, the key terms can obtain higher influence in the similarity calculation, further distinguishing fine-grained semantic differences, and the accuracy of the matching is improved. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0052] Fig. 1 Flow chart of the image video retrieval method based on the field fine-tuning large language model of the present application;

[0053] Fig. 2 Performance comparison diagram of the image video retrieval method based on the field fine-tuning large language model of the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0055] As shown in Figs. 1-2 the first aspect, the present application provides an image video retrieval method based on field fine-tuning large language model, comprising the following steps:

[0056] S1, obtaining a user query text, fine-tuning a pre-training model by low-rank decomposition and regularization constraint to obtain a fine-tuned pre-training model, and classifying the user query text and extracting keywords to obtain user intent label and keyword list.

[0057] The step S1 in the present embodiment comprises the following sub-steps:

[0058] S11, extracting a multi-dimensional parameter vector of an attention subspace from the pre-training model, and calculating the cosine similarity between each two to generate a symmetric similarity matrix, the expression is:

[0059]

[0060] In the formula, A i，j is a symmetric similarity matrix, and A∈R d×d , sim(i, j) is the cosine similarity value between the ith d-dimensional parameter vector and the jth d-dimensional parameter vector, and if the similarity is greater than 0.5, it is reserved, otherwise it is 0;

[0061] S12, normalizing the symmetric similarity matrix to obtain a normalized symmetric similarity matrix, the expression is:

[0062]

[0063] In the formula, L norm is a normalized Laplace matrix, D is a diagonal matrix, and A is a symmetric similarity matrix;

[0064] S13, preset the value range of the candidate cluster number, perform eigenvalue decomposition on the normalized symmetric similarity matrix, extract the first candidate cluster number of feature vectors for k-means clustering, calculate the average distance of each data point and other data points in the same cluster, obtain the silhouette coefficient of each candidate cluster number, and determine the cluster number value corresponding to the maximum silhouette coefficient value as the optimal cluster number; and generate a final clustering group according to the optimal cluster number, and output a parameter grouping label G;

[0065]

[0066] In the formula, k is the candidate clustering cluster number, the value range is [5, 15], k* is the optimal cluster number, s(i) is the silhouette coefficient of sample i, s(i) = (b(i)-a(i)) / (max(a(i), b(i))), a(i) is the average distance of sample i and other samples in the same cluster, b(i) is the average distance of sample i and the nearest sample in the different cluster, n k The total number of samples when the cluster number is k;

[0067] S14, constructing an inter-group mask matrix according to the parameter grouping label, and combining the spectral norm to constrain the low-rank of the intra-group parameters and the sparsity of the inter-group connection to construct a regularization loss function;

[0068] According to the parameter grouping label G, an inter-group mask matrix O is constructed, if the two parameters obtained belong to the same group, it is 0, and if the two parameters obtained belong to different groups, it is 1;

[0069] The regularization loss function expression is:

[0070]

[0071] In the formula, L reg is the regularization loss, A and B are both low-rank matrices, the rank r of A and B is set to 8, the scaling factor a is set to 32, the dropout rate is set to 0.1, and the training round number is set to 3, wherein the rank r is the core parameter of LoRA, which defines the middle dimension of the low-rank decomposition matrix added to the original weight matrix, the scaling factor a balances the update amount of each training round, and the dropout prevents overfitting, O is the inter-group mask matrix, ||·||F F is the spectral norm, ||·||1 is the sparse constraint, and λ1 and λ2 are regularization coefficients.

[0072] S15, constructing a projection mask matrix according to the parameter grouping label and the symmetric similarity matrix, performing SVD decomposition on the attention weight matrix, and extracting singular values and corresponding vectors to calculate a projection update matrix, the expression is:

[0073]

[0074] wherein, AW is a projection update matrix, σ i are the first r significant singular values, u i are left singular vectors, v i are right singular vectors, and M is a projection mask matrix.

[0075] S16, constructing an objective function according to a regularization loss function and a task loss function of intent classification, and training and updating the low-rank matrix by using the projection update matrix to obtain a fine-tuned pre-trained model;

[0076]

[0077] wherein, θ * is an optimal model parameter, θ is a trainable parameter, D finetune is a training data set, (x, y) is a tuple, x is a user query text, y is an intent label; f(x, θ) is a fine-tuned model prediction function;

[0078] Injecting the projection update matrix into the attention layer, updating the low-rank matrix A, B parameters, the expression is:

[0079]

[0080] wherein, W new is an updated attention layer weight matrix, W pre is an original attention layer weight of the pre-trained model.

[0081] It should be noted that in the present embodiment, the GLM-9B model fine-tuned by the LoRA technology is used for intent judgment and keyword extraction, and the understanding ability of the LLM for user intent is used to classify user demand into one or other of a video or a picture, summarize clear keywords, and specify output in json format.

[0082] By adding a regularization term in the loss function, the parameters of the low-rank matrix are kept sparse or low-norm to avoid excessive interference with the pre-trained weights. Since the attention layer and the MLP layer are most sensitive to task adaptability adjustment, the matrices of the attention layer and the MLP layer are high-dimensional dense matrices. Low-rank decomposition can significantly reduce the trainable parameter amount while maintaining performance, and then by injecting low-rank matrix only in the attention layer and the MLP layer, the main parameters of the GLM-9B model are frozen, and only a small number of weight modules are optimized for intent judgment and keyword extraction tasks, thereby reducing the computing power and hardware resource consumption.

[0083] After fine-tuning, the local pre-trained language model is loaded, and the pre-configured LoRA local file and weight are combined to complete model initialization, ensuring that the model can efficiently identify the user's input intent and extract keywords. The output generated by this process is in JSON format, which explicitly includes the user's intent and the list of extracted keywords, providing accurate data support for subsequent processing.

[0084] S2, constructing a prompt word sequence and initializing a prompt template, dynamically generating and screening the optimal prompt template by Monte Carlo tree search combined with a hidden Markov model (HMM) with mutual information as the optimization objective.

[0085] Step S2 includes the following sub-steps:

[0086] S21, constructing a prompt word sequence with preset prompt words, and defining a target output probability distribution;

[0087] S22, initializing a Monte Carlo search tree, loading an initial prompt template set, downloading HMM parameter triplets, which include a state transition matrix, an observation emission matrix, and an initial state distribution, and mounting the initial prompt template as a child node of the root node;

[0088] S23, calculating the UCT index of the current node, descending along the path with the maximum UCT index, selecting nodes that have not been expanded or have a visit count less than a visit threshold, and obtaining the prompt template corresponding to the node;

[0089] S24, according to the obtained prompt template, extracting a hidden state sequence from the initial state distribution, generating candidate word modification operations based on the observation emission matrix, generating a new prompt template for each modification operation, creating a child node for each new prompt template, initializing the visit count and reward;

[0090] S25, performing HMM decoding on each created child node to generate an output sequence, and calculating mutual information as the simulation reward value of each created child node, the expression being:

[0091]

[0092] In the formula, is the simulation reward value, Y (m) is the output sequence generated by HMM decoding under the prompt template P in the mth simulation experiment, p(·) is the generation probability, M is the number of simulation experiments, and P' is the new prompt template;

[0093] S26, updating the UCT index of all nodes on the simulation reward backtracking path, repeating the iteration until a predetermined number of iterations is reached, and selecting the prompt template with the highest average mutual information from the direct child nodes of the root node as the optimal prompt template output.

[0094] It should be noted that the UCT index in the Monte Carlo tree search can find a balance between exploring new prompt templates and utilizing known effective templates. The UCT index comprehensively considers the access frequency and return value of the node, can make full use of the high return templates found, thereby more efficiently searching for the optimal prompt template. In addition, the hidden state transition mechanism and observation emission matrix of the HMM can generate diversified candidate word modification operations, thereby creating new prompt templates. The potential structure and pattern in the language can be captured, and the prompt that conforms to the semantic and grammatical rules can be generated, thereby improving the quality and effectiveness of the prompt. The mutual information is used as the simulation return value, which can measure the degree of association between the prompt template and the target output. The higher the mutual information, the more accurately the prompt template can guide the model to generate the output conforming to the target distribution, thereby facilitating the selection of the optimal prompt template and improving the performance of the intent recognition and keyword extraction tasks. At the same time, by performing small-scale variant operations and searching on the basis of the existing prompt templates, the generation of a large number of prompt templates from scratch is avoided, and the consumption of computing resources and time is reduced. In addition, only part of the nodes are subjected to detailed HMM decoding and mutual information calculation, thereby further improving the efficiency.

[0095] In this embodiment, by combining MCTS and HMM and using mutual information as an evaluation index, the optimal prompt template can be efficiently searched, and the performance of related tasks can be improved while the resource consumption is low.

[0096] S3, a noise word set is constructed, the keyword list is filtered according to the similarity calculation, and the class label of the filtered keyword is predicted by combining a unary potential function and a binary potential function through a conditional random field model, thereby obtaining a keyword enhancement set.

[0097] Step S3 includes the following sub-steps:

[0098] S31, each word in the keyword list is subjected to vector processing, thereby generating a corresponding semantic vector;

[0099] S32, a noise word set and a filtering threshold are predefined, the maximum similarity between the semantic vector corresponding to each word in the keyword list and all noise words in the noise word set is calculated, and the filtering is performed according to the filtering threshold, thereby obtaining a preliminary keyword set, and the expression is as follows:

[0100]

[0101] In the formula, C is the preliminary keyword set, w i is the i th word in the keyword list, c is a noise word in the noise word set, τ1 is the filtering threshold, τ1 = 0.7, and is used to determine whether the keyword is a noise word;

[0102] S33, defining a corresponding video or picture label for each keyword in the preliminary keyword set, and constructing a unary potential function and a binary potential function;

[0103] The unary potential function expression is:

[0104]

[0105] In the formula, ψ(y i ,w i ) is a unary potential function, representing the degree of association between the keyword w i and the label y i at position i, b yi is a bias related to the label y i , and u yi is a weight related to the label y i ;

[0106] The binary potential function expression is:

[0107]

[0108] In the formula, is a binary potential function, representing the dependence between the labels y i and y j at adjacent positions i and j, e(y i ) and e(y j ) are embedding vectors corresponding to the labels y i and y j , respectively;

[0109] S34, constructing a conditional random field (CRF) probability model according to the unary potential function and the binary potential function, inputting the preliminary keyword set into the conditional random field (CRF) probability model, outputting the probability of predicting the corresponding category label of each keyword, and retaining the keywords whose probabilities are greater than a probability threshold and whose binary potentials are less than 0, to obtain a keyword enhanced set, and the conditional random field (CRF) probability model expression is:

[0110]

[0111] In the formula, P(Y|X) is a conditional random field (CRF) probability model, representing the probability of the label sequence Y given the keyword sequence X, and Z is a normalization factor.

[0112] It should be noted that by filtering the noise words in the keyword list, and using the semantic embedding based on contrast learning and spectral clustering method, the semantic features of the words can be captured more finely, the accuracy of noise word detection can be improved, and the quality of subsequent keyword classification can be improved; and the conditional random field model is adopted and combined with the unary potential function and the binary potential function, the dependency between the labels can be fully considered, the unary potential function reflects the association degree of a single keyword with a specific label, the binary potential function captures the constraint relationship between adjacent keyword labels, so that the predicted label is more coherent and accurate, and the accuracy of keyword category prediction is improved; the joint optimization of noise word elimination and category alignment is realized, and at the same time, by setting the probability threshold and eliminating the keywords with binary potential less than 0, the quality of the keyword enhanced set can be guaranteed; only high-probability keywords and reasonable label relationships are retained, the influence of uncertainty and incorrect labels is reduced, and the reliability of the keyword enhanced set is improved.

[0113] In the embodiment, the semantic analysis and the conditional random field (CRF) probability model are constructed, which can effectively filter noise words and accurately predict the category labels of keywords, and generate a high-quality keyword enhanced set.

[0114] S4, combine the keywords in the keyword enhanced set according to the Boolean logic and combine them with the user intent to generate a query condition, use distributed inverted index to execute the query condition in parallel, and obtain a candidate resource set.

[0115] Step S4 includes the following sub-steps:

[0116] S41, obtain the user intent label and the keyword enhanced set, combine the keywords in the keyword enhanced set according to the Boolean logic and parallel splice them with the user intent label to generate a query condition;

[0117] S42, according to the query condition, use distributed inverted index to execute the query condition in parallel, process the query in multiple resource ID storage areas in the database at the same time, combine the returned structures of each area, and eliminate the duplicate IDs to obtain a resource ID set;

[0118] S43, according to the resource ID set, obtain the storage path of the corresponding resource to obtain a candidate resource set.

[0119] It should be noted that using distributed inverted index to execute the query condition in parallel can make full use of the computing resources of the distributed system, and at the same time process the query in multiple resource ID storage areas in the database; the query time is greatly shortened, the query efficiency is improved, and it is especially suitable for fast retrieval of large-scale data sets.

[0120] S5, according to the user query text and the candidate resource set, calculating the similarity of the user query text and each candidate resource, and combining the optimal prompt template to generate attention weight calculation, obtaining the matching score between the user query and each candidate resource, obtaining the resource path with the highest matching score, and obtaining the retrieved image or video.

[0121] Step S5 includes the following steps:

[0122] S51, inputting the user query text and the resource data in the candidate resource set into the fine-tuned pre-training model respectively to obtain a query vector and a candidate resource vector;

[0123] S52, calculating the similarity between the query vector and each candidate resource vector to obtain a semantic similarity score, and the expression is:

[0124]

[0125] In the formula, sim(q, r) is the semantic similarity score, q is the query vector, and r is the candidate resource vector;

[0126] S53, inputting the optimal prompt template into the fine-tuned pre-training model to obtain an attention weight vector;

[0127] S54, according to the semantic similarity score and the attention weight vector, obtaining the matching score between the user query and each candidate resource, obtaining the resource path with the highest matching score, and obtaining the retrieved image or video.

[0128] It should be noted that the user query text and the candidate resource are mapped into vectors by the fine-tuned pre-training model, the semantic similarity score is calculated, the similarity between the user query and the candidate resource can be accurately measured from the semantic level, the accuracy of the retrieval is improved, the attention weight vector is generated by introducing the optimal prompt template, and the matching score is calculated by weighting, so that the key terms have higher influence in the similarity calculation, further distinguishing the fine-grained semantic difference, and the matching accuracy is improved.

[0129] In step S5, for each retrieval result, a matching confidence is calculated according to the matching score, a trigger threshold is preset, if the matching confidence is less than the preset trigger threshold, a hybrid retrieval is triggered, the matching score of the candidate resource set and the key word is obtained, the resource type is filtered according to the user intention label, and the comprehensive score is obtained by comprehensive calculation according to the resource creation time decay coefficient, the resource path with the highest score is obtained according to the comprehensive score sorting, and the retrieved image or video is obtained.

[0130] The matching confidence calculation expression is:

[0131] S = σ (α·TF-IDF max +β·max{sim(q,r)})

[0132] In the formula, S is a matching confidence, sigma is a sigmoid function, alpha and beta are experience tuning weights, TF-IDF max is a highest keyword matching score in the candidate resource set, and max{sim(q,r)} is a highest semantic similarity score in the candidate resource set.

[0133] When S<0.7, it is determined that the search quality is insufficient, the mixed search mode is triggered, a matching score TF-IDF of the candidate resource set and the keyword is calculated, a comprehensive score is calculated according to a resource creation time decay coefficient, and the expression is as follows:

[0134] Z1=W1*TF-IDF+W2*1 / (t n -t o )

[0135] In the formula, Z1 is a comprehensive score, W1 and W2 are weight coefficients, TF-IDF is a matching score of the candidate resource set and the keyword, t n is a current time, and t o is a creation time.

[0136] It should be noted that when the matching confidence is less than a preset triggering threshold, the mixed search mechanism is triggered. The comprehensive score is calculated by combining a semantic matching score, a user intention label filtering, and a resource creation time decay coefficient. When the semantic matching effect is poor, a relatively suitable resource can be found by comprehensive consideration of various factors, and the robustness of the search is enhanced.

[0137] In a second aspect, the present application also provides an image and video search system based on a domain fine-tuned large language model, which is implemented by using an image and video search method based on a domain fine-tuned large language model. The system comprises:

[0138] A parameter adjustment module is configured to obtain a user query text, fine-tune a pre-training model by using low-rank decomposition and regularization constraints to obtain a fine-tuned pre-training model, and perform intention classification and keyword extraction on the user query text to obtain a user intention label and a keyword list.

[0139] A prompt template generation module is configured to construct a prompt word sequence and initialize a prompt template, dynamically and iteratively generate and screen an optimal prompt template by using Monte Carlo tree search in combination with a hidden Markov model (HMM) and taking mutual information as an optimization objective.

[0140] A filtering module is configured to construct a noise word set, filter the keyword list according to similarity calculation, and predict the class label of the filtered keyword by using a conditional random field model in combination with a unary potential function and a binary potential function to obtain a keyword enhanced set.

[0141] The screening module is configured to combine keywords in the keyword enhancement set according to Boolean logic, combine the keywords with the user intent, generate a query condition, perform the query condition in parallel using distributed inverted indexes, and obtain a candidate resource set;

[0142] The matching module is configured to calculate the similarity between the user query text and each candidate resource according to the user query text and the candidate resource set, generate an attention weight calculation based on the optimal prompt template, obtain a matching score between the user query and each candidate resource, obtain a resource path with the highest matching score, and obtain the retrieved image or video.

[0143] It should be noted that the system corresponds to the image and video retrieval method based on the field fine-tuning large language model, and all implementation manners in the above method embodiments are applicable to the embodiments of the system and can achieve the same technical effects.

[0144] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system and the module described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0146] In the embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented by other ways. For example, the device embodiments described above are only schematic, and the division of the units is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0147] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.

[0148] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit.

[0149] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various program code storage media.

[0150] In addition, it should be noted that in the system and method of the present application, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalent solutions of the present application. And the steps of executing the above series of processes can naturally be executed in time sequence according to the order of description, but it is not necessary to be executed in time sequence. Some steps can be executed in parallel or independently of each other. It can be understood by those skilled in the art that all or any steps or components of the method and device of the present application can be realized in hardware, firmware, software or their combination in any computing device (including processor, storage medium, etc.) or network of computing devices, which can be realized by those skilled in the art with their basic programming skills after reading the description of the present application.

[0151] Therefore, the purpose of the present application can also be realized by running a program or a group of programs on any computing system. The computing system can be a known general system. Therefore, the purpose of the present application can also be realized only by providing a program product containing program code for realizing the method or device. That is, such a program product also constitutes the present application, and the storage medium storing such a program product also constitutes the present application. Obviously, the storage medium can be any known storage medium or any storage medium developed in the future. It should be noted that in the device and method of the present application, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalent solutions of the present application. And the steps of executing the above series of processes can naturally be executed in time sequence according to the order of description, but it is not necessary to be executed in time sequence. Some steps can be executed in parallel or independently of each other.

[0152] The above merely provides the preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image video retrieval method based on domain fine-tuning a large language model, characterized in that, The method comprises the following steps: S1, obtaining a user query text, fine-tuning a pre-trained model by low-rank decomposition and regularization constraint to obtain a fine-tuned pre-trained model, and classifying the user query text and extracting keywords to obtain a user intent label and a keyword list; S2, constructing a prompt word sequence and initializing a prompt template, dynamically and iteratively generating and screening an optimal prompt template by Monte Carlo tree search combined with a hidden Markov model (HMM) and taking mutual information as an optimization target; S3, constructing a noise word set, filtering the keyword list according to similarity calculation, and predicting the category label of the filtered keyword by a conditional random field model combined with a unary potential function and a binary potential function to obtain a keyword enhanced set; S4, combining the keywords in the keyword enhanced set according to Boolean logic and combining with the user intent to generate a query condition, performing the query condition in parallel by using a distributed inverted index to obtain a candidate resource set; S5, calculating the similarity between the user query text and each candidate resource according to the user query text and the candidate resource set, and generating attention weight calculation combined with the optimal prompt template to obtain the matching score between the user query and each candidate resource, obtaining the resource path with the highest matching score, and obtaining the retrieved image or video.

2. The image video retrieval method based on domain fine-tuning large language model of claim 1, wherein, The step S1 comprises the following sub-steps: S11, extracting a multi-dimensional parameter vector of an attention subspace from the pre-trained model, and calculating a cosine similarity to generate a symmetric similarity matrix; S12, normalizing the symmetric similarity matrix to obtain a normalized symmetric similarity matrix; S13, presetting a value range of a candidate cluster number, performing eigenvalue decomposition on the normalized symmetric similarity matrix, extracting a feature vector of the candidate cluster number, performing k-means clustering, calculating the average distance between each data point and other data points in the same cluster, obtaining the silhouette coefficient of each candidate cluster number, determining the cluster number value corresponding to the maximum silhouette coefficient value as the best cluster number, and generating a final clustering group according to the optimal cluster number and outputting a parameter grouping label; S14, constructing an inter-group mask matrix according to the parameter grouping label, and combining a spectral norm constraint to constrain the low-rank of intra-group parameters and the sparsity of inter-group connections to construct a regularization loss function; S15, constructing a projection mask matrix according to the parameter grouping label and the symmetric similarity matrix, performing SVD decomposition on the attention weight matrix, extracting singular values and corresponding vectors, and calculating a projection update matrix; S16, constructing an objective function according to the regularization loss function and the task loss function of intent classification, and training and updating the low-rank matrix by using the projection update matrix to obtain the fine-tuned pre-trained model. 3.The domain fine-tuning based large language model image video retrieval method of claim 2, wherein, The step S2 comprises the following sub-steps: S21, presetting a prompt word to construct a prompt word sequence, and defining a target output probability distribution; S22, initializing a Monte Carlo search tree, loading an initial prompt template set, downloading HMM parameter triplets including a state transition matrix, an observation emission matrix and an initial state distribution, and mounting the initial prompt template as a child node of a root node; S23, calculating a UCT index of a current node, descending along a path with a maximum UCT index, selecting a node that is not expanded or has a visit count less than a visit threshold, and obtaining a prompt template corresponding to the node; S24, extracting a hidden state sequence from the initial state distribution according to the obtained prompt template, generating candidate word modification operations based on the observation emission matrix, generating a new prompt template for each modification operation, creating a child node for each new prompt template, and initializing a visit count and a return; S25, performing HMM decoding on each created child node to generate an output sequence, and calculating mutual information as a simulation return value of each created child node; S26, updating the UCT index of all nodes on the path according to the simulation return, repeating iteration until a predetermined iteration number is reached, and selecting a prompt template with the highest average mutual information from the direct child nodes of the root node as the optimal prompt template output. 4.The image and video retrieval method based on domain fine-tuning large language model according to claim 3, wherein, In step S3, the noise word set is constructed, the keyword list is filtered according to the similarity calculation, and the class label of the filtered keyword is predicted through the conditional random field model combined with the unary potential function and the binary potential function to obtain the keyword enhancement set, including the following sub-steps: S31, performing vector processing on each word in the keyword list to generate a corresponding semantic vector; S32, predefining a noise word set and a filtering threshold, calculating the maximum similarity of the semantic vector corresponding to each word in the keyword list with all noise words in the noise word set, and filtering according to the filtering threshold to obtain a preliminary keyword set; S33, defining a corresponding video or picture label for each keyword in the preliminary keyword set, and constructing a unary potential function and a binary potential function; S34, constructing a conditional random field CRF probability model according to the unary potential function and the binary potential function, inputting the preliminary keyword set into the conditional random field CRF probability model, outputting the probability of predicting the corresponding class label of each keyword, and retaining the keywords with a probability greater than a probability threshold and a binary potential less than 0, to obtain a keyword enhancement set. 5.The domain fine-tuning based large language model image video retrieval method of claim 4, wherein, In step S4, the keywords in the keyword enhancement set are combined according to Boolean logic and combined with user intent to generate a query condition, and distributed inverted indexes are used to execute the query condition in parallel to obtain a candidate resource set, including: obtaining a user intent label and a keyword enhancement set, combining the keywords in the keyword enhancement set according to Boolean logic and juxtaposing with the user intent label to generate a query condition; according to the query condition, using distributed inverted indexes to execute the query condition in parallel, simultaneously processing the query in multiple resource ID storage areas in the database, merging the returned structures of each area, and removing duplicate IDs to obtain a resource ID set; according to the resource ID set, obtaining the storage path of the corresponding resource to obtain a candidate resource set.

6. The image and video retrieval method based on domain fine-tuning large language model according to claim 5, wherein: In step S5, the similarity between the user query text and each candidate resource is calculated according to the user query text and the candidate resource set, and the attention weight calculation is generated in combination with the optimal prompt template to obtain the matching score between the user query and each candidate resource, the resource path with the highest matching score is obtained, and the retrieved image or video is obtained, including the following steps: S51, input the user query text and the resource data in the candidate resource set into the fine-tuned pre-trained model respectively to obtain the query vector and the candidate resource vector; S52, calculate the similarity between the query vector and each candidate resource vector to obtain the semantic similarity score; S53, input the optimal prompt template into the fine-tuned pre-trained model to obtain the attention weight vector; S54, according to the semantic similarity score and the attention weight vector, the matching score between the user query and each candidate resource is obtained, the resource path with the highest matching score is obtained, and the retrieved image or video is obtained.

7. The image and video retrieval method based on domain fine-tuning large language model according to claim 6, wherein: In step S5, for each retrieval result, the matching confidence is calculated according to the matching score, and a preset trigger threshold is preset. If the matching confidence is less than the preset trigger threshold, the hybrid retrieval is triggered, the matching score of the candidate resource set and the keyword is obtained, the resource type is filtered according to the user intent label, and the comprehensive score is obtained by comprehensive calculation according to the resource creation time decay coefficient. According to the comprehensive score, the resource path with the highest score is obtained, and the retrieved image or video is obtained.

8. An image and video retrieval system based on fine-tuning a large language model for a domain, which is implemented by using the image and video retrieval method based on fine-tuning a large language model for a domain according to any one of claims 1-7. The system comprises: A parameter adjustment module is configured to obtain a user query text, fine-tune a pre-trained model by low-rank decomposition and regularization constraint to obtain a fine-tuned pre-trained model, and classify and extract keywords of the user query text to obtain a user intent label and a keyword list; A prompt template generation module is configured to construct a prompt word sequence and initialize a prompt template, dynamically and iteratively generate and screen the optimal prompt template by Monte Carlo tree search combined with a hidden Markov model (HMM) and taking mutual information as an optimization objective; A filtering module is configured to construct a noise word set, filter the keyword list according to the similarity calculation, and predict the class label of the filtered keyword by a conditional random field model combined with a unary potential function and a binary potential function to obtain a keyword enhanced set; A screening module is configured to combine the keywords in the keyword enhanced set according to Boolean logic and combine with the user intent to generate a query condition, perform the query condition in parallel by using a distributed inverted index to obtain a candidate resource set; A matching module is configured to calculate the similarity between the user query text and each candidate resource according to the user query text and the candidate resource set, and generate the attention weight calculation in combination with the optimal prompt template to obtain the matching score between the user query and each candidate resource, obtain the resource path with the highest matching score, and obtain the retrieved image or video.

9. A terminal device, comprising: The terminal device comprises a memory, a processor, and an image and video retrieval method based on a domain fine-tuned large language model stored on the memory and executable on the processor, and the image and video retrieval method based on the domain fine-tuned large language model is configured to implement the steps of the image and video retrieval method based on the domain fine-tuned large language model in any one of claims 1 to 7.

10. A storage medium, characterized by The storage medium is a computer storage medium, and the computer storage medium stores a large language model-based image and video retrieval method program based on domain fine-tuning. When the large language model-based image and video retrieval method program based on domain fine-tuning is executed by the processor, the steps of the large language model-based image and video retrieval method based on domain fine-tuning according to any one of claims 1 to 7 are implemented.

Images