Intelligent data retrieval optimization method based on large model and deep learning

Abstract

The invention provides an intelligent data retrieval optimization method based on a large model and deep learning. The method comprises the following steps: performing data cleaning, labeling and index construction on original data; performing semantic analysis on user query by using a large-scale pre-training language model to generate deep semantic representation of the query; performing semantic matching in a distributed database according to deep semantic representation queried by a user, and retrieving related high-dimensional feature vectors; classifying and sorting the matched high-dimensional feature vectors through a deep learning network; dynamically adjusting parameters of the pre-training language model and the deep learning network in combination with online learning and a real-time feedback mechanism; and a cache strategy and an index optimization technology are adopted, and a retrieval result most relevant to user query is output. According to the method, by combining an advanced language model and a deep learning technology, efficient and accurate intelligent data retrieval is realized, and the retrieval experience of a user is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent data retrieval technology, and in particular to an optimization method for intelligent data retrieval based on large models and deep learning. Background Technology

[0002] In the context of rapid development in information technology, the volume of data is exploding. How to efficiently and accurately retrieve the information users need from massive amounts of data has become a core challenge in the field of information retrieval. Traditional information retrieval methods mainly rely on keyword matching and Boolean logic-based retrieval models. While these methods achieved good results in the early stages, their limitations have gradually become apparent with the increase in data volume and complexity. First, traditional retrieval methods have significant shortcomings in semantic understanding. They mainly rely on literal keyword matching, ignoring the semantic relationships between words, thus greatly reducing the accuracy of search results.

[0003] Secondly, traditional retrieval methods perform poorly when handling complex queries, especially in their limited ability to understand and process natural language queries. When users submit queries containing multiple conditions or requiring reasoning, traditional methods often fail to provide satisfactory results. Furthermore, with the diversification of data types and the increasing dimensionality of data features, traditional methods are prone to the curse of dimensionality when processing high-dimensional data, leading to low retrieval efficiency and difficulty in meeting real-time response requirements.

[0004] Furthermore, traditional methods are typically based on static models, which, once established, are rarely dynamically adjusted. However, user needs and data characteristics are constantly changing, making it difficult for static models to adapt, leading to a gradual decline in retrieval performance. In recent years, the development of deep learning and large-scale pre-trained language models has brought new opportunities to information retrieval. Deep learning, through its multi-layered neural network structure, can automatically extract complex features from massive amounts of data and performs exceptionally well in various tasks. In particular, large-scale pre-trained language models, through pre-training on massive amounts of text data, have acquired rich semantic information and contextual relationships, significantly improving the ability to understand and process natural language. Summary of the Invention

[0005] One objective of this invention is to propose a data intelligent retrieval optimization method based on large models and deep learning. This invention combines advanced language models and deep learning techniques to significantly improve semantic understanding capabilities; it employs caching strategies and index optimization techniques to output the most relevant retrieval results to the user's query. This achieves efficient and accurate data intelligent retrieval, significantly enhancing the user's search experience.

[0006] A data intelligent retrieval optimization method based on large models and deep learning according to an embodiment of the present invention includes the following steps:

[0007] S1. Perform data preprocessing on the raw data, including data cleaning, data labeling, and index construction;

[0008] S2. Use large-scale pre-trained language models to perform semantic analysis on user queries and generate deep semantic representations of the queries;

[0009] S3. Use a deep learning network to extract features from the preprocessed data to obtain a high-dimensional feature vector of the data;

[0010] S4. Construct a distributed retrieval architecture and store feature vectors in a distributed database;

[0011] S5. Based on the deep semantic representation of the user query, perform semantic matching in the distributed database to retrieve relevant high-dimensional feature vectors;

[0012] S6. Classify and sort the matched high-dimensional feature vectors using a deep learning network to filter out relevant data;

[0013] S7. Combine online learning and real-time feedback mechanisms to dynamically adjust the parameters of large-scale pre-trained language models and deep learning networks;

[0014] S8. Employs caching strategies and index optimization techniques for data retrieval, outputting the most relevant search results to the user's query.

[0015] Optionally, S1 specifically includes:

[0016] S11. Data cleaning: Remove noisy data and outliers from the original data, and process numerical data using standardization and normalization methods.

[0017]

[0018] Where X represents the original data, μ represents the mean of the data, and σ represents the standard deviation of the data; missing data is imputed using interpolation or missing value prediction:

[0019]

[0020] Where, x i For missing data points, x i-1 and x i+1 For adjacent data points;

[0021] S12. Data labeling: Label the data using a predefined set of labels;

[0022] S13. Index Construction: A data index is constructed using inverted indexing techniques. Data features are vectorized to generate a feature vector matrix V, where each feature vector v... iRepresents the feature set of data item i.

[0023] Optionally, S2 specifically includes:

[0024] S21. Perform word segmentation on the user query to generate a word sequence Q = {q1, q2, ..., q} n}, where q i The i-th word in the query;

[0025] S22. Encode the word sequence Q using a large-scale pre-trained language model to generate a context embedding vector E = {e1, e2, ..., e} for each word. n}, where e i For the word q i Context embedding vector;

[0026] S23. Aggregate the context embedding vector E through a multi-layer attention mechanism to generate the global semantic representation vector S of the query:

[0027]

[0028] Among them, A i Here, Q represents the attention weights, K represents the query matrix, K represents the key matrix, V represents the value matrix, and d represents the attention weights. k Let be the dimension of the key matrix;

[0029] S24. Input the global semantic representation vector S into the deep neural network for further processing to generate the final deep semantic representation of the query S′. The specific processing includes forward propagation and backward propagation steps.

[0030] S25. Output the deep semantic representation vector S′ of the query, which will be used in the subsequent retrieval and matching process.

[0031] Optionally, S3 specifically includes:

[0032] S31. Use inverted indexing technology to construct a data index. The inverted index structure includes document ID and keyword mapping.

[0033] S32. Perform feature vectorization on the preprocessed data to generate a feature vector matrix V, where each feature vector v... i Represents the feature set of data item i;

[0034] S33. The feature vector is hashed using a locality-sensitive hashing (LSH) method:

[0035]

[0036] Where a and b are randomly selected constants, w is the segment width, and v is the feature vector;

[0037] S34. Use an optimized index structure, such as a B-tree or a Trie tree, to store and manage the hashed feature vectors;

[0038] S35. Dynamically update the constructed index, combining the addition of new data with the deletion of old data, to maintain the index's up-to-dateness and effectiveness;

[0039] S36. During retrieval, matching is performed by calculating cosine similarity using the inverted index structure and the hashed feature vectors:

[0040]

[0041] Among them, v i and v j Let v be two eigenvectors, where · represents the dot product of the vectors and ||v|| represents the magnitude of the vectors.

[0042] S37. Sort the search results according to the similarity score and output the data most relevant to the user's query.

[0043] Optionally, S4 specifically includes:

[0044] S41. Construct an efficient distributed retrieval architecture, wherein the distributed architecture includes multiple nodes, each node stores a portion of the data feature vectors and is capable of performing retrieval operations independently;

[0045] S42. Distribute the feature vectors to each node according to a certain sharding strategy, which is based on a hash function:

[0046]

[0047] Among them, v i p is the i-th component of the eigenvector. i The corresponding weights are N, where N is the number of nodes.

[0048] S43. Within each node, the feature vector is stored and managed using a tree-based indexing method, such as KD-tree or R-tree.

[0049] S44. Perform consistency processing on data between distributed nodes, and use a consistent hashing algorithm to ensure the uniform distribution of data and load balancing among nodes;

[0050] S45. When performing a search, the user query is routed to the corresponding set of nodes, and each node in the set independently computes its local search results;

[0051] S46. Merge and deduplicate the local search results of each node, and then sort the merged results using a global ranking algorithm. The ranking criteria include the similarity of feature vectors and node response time.

[0052]

[0053] Where, sim(v) i ,q) represents the eigenvector v i Similarity to query q, t i Let be the response time of node i, and α and β be weighting coefficients;

[0054] S47. Output the final sorted search results.

[0055] Optionally, S5 specifically includes:

[0056] S51. Calculate the similarity between the query vector S′ and the high-dimensional feature vector V stored in the database using cosine similarity.

[0057] S52. Sort all feature vectors according to their similarity scores, and select the k feature vectors with the highest similarity {V1, V2, ..., V...} k} as the initial matching result;

[0058] S53. Refine the preliminary matching results by using a large-scale pre-trained language model to perform secondary semantic analysis on the matching results and the query, generating a similarity score:

[0059] sim2(S′,V i )=α·sim(S′,V i )+β·BERT(S′,V i );

[0060] Where α and β are weighting coefficients, BERT(S′,V i () represents the similarity score calculated by a large-scale pre-trained language model;

[0061] S54. Combining the secondary similarity scores, the matching results are finally sorted, and the high-dimensional feature vectors {V′1,V′2,…,V′} most relevant to the user query are selected. m}, where m≤k;

[0062] S55. Output the final matched high-dimensional feature vector for subsequent data classification and sorting.

[0063] Optionally, S6 specifically includes:

[0064] S61. The matched high-dimensional feature vector {V′1,V′2,…,V′} is used to... m The input is fed into a pre-trained deep neural network;

[0065] S62. Adaptive sparse coding is used in the output layer of the deep neural network to process the feature vectors in order to extract sparse features:

[0066]

[0067] Where X is the input feature vector, D is the dictionary matrix, α is the sparse representation, and λ is the regularization parameter;

[0068] S63. Input the feature vector processed by adaptive sparse coding into the attention-based transformer model for further feature extraction:

[0069]

[0070] Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k Let be the dimension of the key matrix;

[0071] S64. Input the output of the self-attention mechanism into a multi-kernel support vector machine to capture the complex nonlinear relationships between features:

[0072]

[0073] Where K(x) i ,x j ) is the kernel function, k m (x i ,x j ) is the m-th fundamental kernel function, β m The weights of the kernel functions are M, and the number of kernel functions is M.

[0074] S65. Input the output of the multi-kernel support vector machine into the adaptive boosting model for further classification and feature extraction:

[0075]

[0076] Where F(x) is the final output of the classifier, α t h represents the weight of the t-th weak classifier. t (x) represents the t-th weak classifier, and T represents the total number of weak classifiers;

[0077] S66. Input the output of the adaptive boosting model into the Gaussian process model for probability classification and uncertainty assessment:

[0078]

[0079] Where k(x,x′) is the covariance function of the inputs x and x′, σ f Let l be the signal standard deviation and l be the length scale.

[0080] S67. Based on the classification results of the Gaussian process classification model, perform a preliminary sorting of the high-dimensional feature vectors and select the top n feature vectors {V″1,V″2,…,V″} with the highest classification probabilities. n Further processing is required;

[0081] S68. Refine the sorting of the initially ranked feature vectors, and calculate the comprehensive score by combining the semantic similarity and classification probability of the feature vectors:

[0082] score(V″ i )=γ·sim(V″ i ,q)+δ·P(y=j|V″ i );

[0083] Where γ and δ are weighting coefficients, sim(V″ i ,q) is the feature vector V″ i The semantic similarity with query q, P(y=j|V″) i ) is the feature vector V″ i The probability of belonging to category j;

[0084] S68. Based on the comprehensive score, sort the feature vectors and select the high-dimensional feature vectors that are most relevant to the user's query;

[0085] S69. Output the sorted feature vectors to generate the final search results.

[0086] Optionally, S7 specifically includes:

[0087] S71. Obtain user retrieval behavior data and feedback data, use them as part of the training data, and generate a real-time feedback dataset D. feedback ;

[0088] S72. Transfer the real-time feedback dataset D feedback Compared with the original training dataset D train The datasets are merged to form an expanded training dataset D. ext ;

[0089] S73. Using the extended training dataset D ext Online fine-tuning of the loss function is performed on a large-scale pre-trained language model to adjust the model parameters θ:

[0090]

[0091] Among them, y i For actual labels, The model predicts the probability, where N is the number of training samples;

[0092] S74. Use gradient descent to dynamically update the parameters of deep learning networks:

[0093]

[0094] Where, θ t Let be the parameters for the t-th iteration, and η be the learning rate. The loss function L with respect to the parameter θ t The gradient;

[0095] S75. Utilize information such as user clicks and dwell time from real-time feedback data to adjust the model's weighting coefficients and optimize the relevance of search results:

[0096] α new =α old +β·Δfeedback;

[0097] Where, α new and α old These are the weighting coefficients before and after adjustment, respectively; β is the adjustment rate; and Δfeedback is the feedback change.

[0098] S76. Through periodic online learning and real-time feedback mechanisms, the parameters of large-scale pre-trained language models and deep learning networks are continuously updated and optimized.

[0099] Optionally, S8 specifically includes:

[0100] S81. Cache frequently queried and high-frequency access data, and store the data with the highest cache hit rate in the high-speed cache, using the Least Recently Used algorithm for cache management:

[0101] C new =C old +△t·F;

[0102] Among them, C new and C old These represent the cache state before and after adjustment, respectively, where △ is the time increment and F is the access frequency;

[0103] S82. For different data characteristics, construct optimized index structures and use a hybrid index structure based on inverted indexes and B-trees to improve index retrieval efficiency:

[0104] I = (K, P);

[0105] Where I represents the index structure, K represents the keyword, and P represents the data location;

[0106] S83. Regularly update and maintain cached data and index structures to ensure data up-to-dateness and consistency; adopt a timestamp-based update strategy:

[0107] Tnew =T old +△t;

[0108] Among them, T new and T old These are the timestamps before and after the update, respectively, with △t representing the time increment.

[0109] S84. During the retrieval process, the cache is checked first. If the cache is hit, the result is returned directly; otherwise, an index retrieval is performed.

[0110]

[0111] Where H is the cache hit rate, and C is the cache hit rate. hit To cache hit count, C total This represents the total number of queries.

[0112] S85. Utilizing parallel computing and distributed processing technologies, the index retrieval process is accelerated, enabling efficient operation in large-scale data environments:

[0113]

[0114] Where S is the acceleration ratio, T serial For serial computation time, T parallel For parallel computing time;

[0115] S86. Combining caching strategies and index optimization techniques, output the most relevant search results to the user's query.

[0116] The beneficial effects of this invention are:

[0117] This invention realizes a highly efficient and intelligent data retrieval system. This system can accurately understand the deep semantics of user queries, significantly improving retrieval efficiency and accuracy, and dynamically optimizing retrieval performance. It provides users with a fast and accurate data retrieval solution, significantly enhancing user experience and retrieval efficiency through intelligent processing and continuous performance optimization. Attached Figure Description

[0118] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0119] Figure 1 This is a flowchart of a data intelligent retrieval optimization method based on large models and deep learning proposed in this invention. Detailed Implementation

[0120] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0121] refer to Figure 1 A data intelligent retrieval optimization method based on large models and deep learning includes the following steps:

[0122] S1. Perform data preprocessing on the raw data, including data cleaning, data labeling, and index construction;

[0123] S2. Use large-scale pre-trained language models to perform semantic analysis on user queries and generate deep semantic representations of the queries;

[0124] S3. Use a deep learning network to extract features from the preprocessed data to obtain a high-dimensional feature vector of the data;

[0125] S4. Construct a distributed retrieval architecture and store feature vectors in a distributed database;

[0126] S5. Based on the deep semantic representation of the user query, perform semantic matching in the distributed database to retrieve relevant high-dimensional feature vectors;

[0127] S6. Classify and sort the matched high-dimensional feature vectors using a deep learning network to filter out relevant data;

[0128] S7. Combine online learning and real-time feedback mechanisms to dynamically adjust the parameters of large-scale pre-trained language models and deep learning networks;

[0129] S8. Employs caching strategies and index optimization techniques for data retrieval, outputting the most relevant search results to the user's query.

[0130] In this embodiment, S1 specifically includes:

[0131] S11. Data cleaning: Remove noisy data and outliers from the original data, and process numerical data using standardization and normalization methods.

[0132]

[0133] Where X represents the original data, μ represents the mean of the data, and σ represents the standard deviation of the data; missing data is imputed using interpolation or missing value prediction:

[0134]

[0135] Where, x i For missing data points, x i-1 and xi+1 For adjacent data points;

[0136] S12. Data labeling: Label the data using a predefined set of labels;

[0137] S13. Index Construction: A data index is constructed using inverted indexing techniques. Data features are vectorized to generate a feature vector matrix V, where each feature vector v... i Represents the feature set of data item i.

[0138] In this embodiment, S2 specifically includes:

[0139] S21. Perform word segmentation on the user query to generate a word sequence Q = {q1, q2, ..., q} n}, where q i For the i-th word in the query;

[0140] S22. Encode the word sequence Q using a large-scale pre-trained language model to generate a context embedding vector E = {e1, e2, ..., e} for each word. n}, where e i For the word q i Context embedding vector;

[0141] S23. Aggregate the context embedding vector E through a multi-layer attention mechanism to generate the global semantic representation vector S of the query:

[0142]

[0143] Among them, A i Here, Q represents the attention weights, K represents the query matrix, K represents the key matrix, V represents the value matrix, and d represents the attention weights. k Let be the dimension of the key matrix;

[0144] S24. Input the global semantic representation vector S into the deep neural network for further processing to generate the final deep semantic representation of the query S′. The specific processing includes forward propagation and backward propagation steps.

[0145] S25. Output the deep semantic representation vector S′ of the query, which will be used in the subsequent retrieval and matching process.

[0146] In this embodiment, S3 specifically includes:

[0147] S31. Use inverted indexing technology to construct a data index. The inverted index structure includes document ID and keyword mapping.

[0148] S32. Perform feature vectorization on the preprocessed data to generate a feature vector matrix V, where each feature vector v... i Represents the feature set of data item i;

[0149] S33. The feature vector is hashed using a locality-sensitive hashing (LSH) method:

[0150]

[0151] Where a and b are randomly selected constants, w is the segment width, and v is the feature vector;

[0152] S34. Use an optimized index structure, such as a B-tree or a Trie tree, to store and manage the hashed feature vectors;

[0153] S35. Dynamically update the constructed index, combining the addition of new data with the deletion of old data, to maintain the index's up-to-dateness and effectiveness;

[0154] S36. During retrieval, matching is performed by calculating cosine similarity using the inverted index structure and the hashed feature vectors:

[0155]

[0156] Among them, v i and v j Let v be two eigenvectors, where · represents the dot product of the vectors and ||v|| represents the magnitude of the vectors.

[0157] S37. Sort the search results according to the similarity score and output the data most relevant to the user's query.

[0158] In this embodiment, S4 specifically includes:

[0159] S41. Construct an efficient distributed retrieval architecture, wherein the distributed architecture includes multiple nodes, each node stores a portion of the data feature vectors and is capable of performing retrieval operations independently;

[0160] S42. Distribute the feature vectors to each node according to a certain sharding strategy, which is based on a hash function:

[0161]

[0162] Among them, v i p is the i-th component of the eigenvector. i The corresponding weights are N, where N is the number of nodes.

[0163] S43. Within each node, the feature vector is stored and managed using a tree-based indexing method, such as KD-tree or R-tree.

[0164] S44. Perform consistency processing on data between distributed nodes, and use a consistent hashing algorithm to ensure the uniform distribution of data and load balancing among nodes;

[0165] S45. When performing a search, the user query is routed to the corresponding set of nodes, and each node in the set independently computes its local search results;

[0166] S46. Merge and deduplicate the local search results of each node, and then sort the merged results using a global ranking algorithm. The ranking criteria include the similarity of feature vectors and node response time.

[0167]

[0168] Where, sim(v) i ,q) represents the eigenvector v i Similarity to query q, t i Let be the response time of node i, and α and β be weighting coefficients;

[0169] S47. Output the final sorted search results.

[0170] In this embodiment, S5 specifically includes:

[0171] S51. Calculate the similarity between the query vector S′ and the high-dimensional feature vector V stored in the database using cosine similarity.

[0172] S52. Sort all feature vectors according to their similarity scores, and select the k feature vectors with the highest similarity {V1, V2, ..., V...} k} as the initial matching result;

[0173] S53. Refine the preliminary matching results by using a large-scale pre-trained language model to perform secondary semantic analysis on the matching results and the query, generating a similarity score:

[0174] sim2(S′,V i )=α·sim(S′,V i )+β·BERT(S′,V i );

[0175] Where α and β are weighting coefficients, BERT(S′,V i () represents the similarity score calculated by a large-scale pre-trained language model;

[0176] S54. Combining the secondary similarity scores, the matching results are finally sorted, and the high-dimensional feature vectors {V′1,V′2,…,V′} most relevant to the user query are selected. m}, where m≤k;

[0177] S55. Output the final matched high-dimensional feature vector for subsequent data classification and sorting.

[0178] In this embodiment, S6 specifically includes:

[0179] S61. The matched high-dimensional feature vector {V′1,V′2,…,V′} is used to... m The input is fed into a pre-trained deep neural network;

[0180] S62. Adaptive sparse coding is used in the output layer of the deep neural network to process the feature vectors in order to extract sparse features:

[0181]

[0182] Where X is the input feature vector, D is the dictionary matrix, α is the sparse representation, and λ is the regularization parameter;

[0183] S63. Input the feature vector processed by adaptive sparse coding into the attention-based transformer model for further feature extraction:

[0184]

[0185] Where Q is the query matrix, K is the key matrix, V is the value matrix, and d k Let be the dimension of the key matrix;

[0186] S64. Input the output of the self-attention mechanism into a multi-kernel support vector machine to capture the complex nonlinear relationships between features:

[0187]

[0188] Where K(x) i ,x j ) is the kernel function, k m (x i ,x j ) is the m-th fundamental kernel function, β m The weights of the kernel functions are M, and the number of kernel functions is M.

[0189] S65. Input the output of the multi-kernel support vector machine into the adaptive boosting model for further classification and feature extraction:

[0190]

[0191] Where F(x) is the final output of the classifier, α t h represents the weight of the t-th weak classifier. t (x) represents the t-th weak classifier, and T represents the total number of weak classifiers;

[0192] S66. Input the output of the adaptive boosting model into the Gaussian process model for probability classification and uncertainty assessment:

[0193]

[0194] Where k(x,x′) is the covariance function of the inputs x and x′, σ f Let l be the signal standard deviation and l be the length scale.

[0195] S67. Based on the classification results of the Gaussian process classification model, perform a preliminary sorting of the high-dimensional feature vectors and select the top n feature vectors {V″1,V″2,…,V″} with the highest classification probabilities. n Further processing is required;

[0196] S68. Refine the sorting of the initially ranked feature vectors, and calculate the comprehensive score by combining the semantic similarity and classification probability of the feature vectors:

[0197] score(V″ i )=γ·sim(V″ i ,q)+δ·P(y=j|V″ i );

[0198] Where γ and δ are weighting coefficients, sim(V″ i ,q) is the feature vector V″ i The semantic similarity with query q, P(y=j|V″) i ) is the feature vector V″ i The probability of belonging to category j;

[0199] S68. Based on the comprehensive score, sort the feature vectors and select the high-dimensional feature vectors that are most relevant to the user's query;

[0200] S69. Output the sorted feature vectors to generate the final search results.

[0201] In this embodiment, S7 specifically includes:

[0202] S71. Obtain user retrieval behavior data and feedback data, use them as part of the training data, and generate a real-time feedback dataset D. feedback ;

[0203] S72. Transfer the real-time feedback dataset D feedback Compared with the original training dataset D train The datasets are merged to form an expanded training dataset D. ext ;

[0204] S73. Using the extended training dataset D extOnline fine-tuning of the loss function is performed on a large-scale pre-trained language model to adjust the model parameters θ:

[0205]

[0206] Among them, y i For actual labels, The model predicts the probability, where N is the number of training samples;

[0207] S74. Use gradient descent to dynamically update the parameters of deep learning networks:

[0208]

[0209] Where, θ t Let be the parameters for the t-th iteration, and η be the learning rate. The loss function L with respect to parameter θ t The gradient;

[0210] S75. Utilize information such as user clicks and dwell time from real-time feedback data to adjust the model's weighting coefficients and optimize the relevance of search results:

[0211] α new =α old +β·Δfeedback;

[0212] Where, α new and α old These are the weighting coefficients before and after adjustment, respectively; β is the adjustment rate; and Δfeedback is the feedback change.

[0213] S76. Through periodic online learning and real-time feedback mechanisms, the parameters of large-scale pre-trained language models and deep learning networks are continuously updated and optimized.

[0214] In this embodiment, S8 specifically includes:

[0215] S81. Cache frequently queried and high-frequency access data, and store the data with the highest cache hit rate in the high-speed cache, using the Least Recently Used algorithm for cache management:

[0216] C new =C old +△t·F;

[0217] Among them, C new and C old These represent the cache state before and after adjustment, respectively, where △ is the time increment and F is the access frequency;

[0218] S82. For different data characteristics, construct optimized index structures, using a hybrid index structure based on inverted indexes and B-trees to improve index retrieval efficiency:

[0219] I = (K, P);

[0220] Where I represents the index structure, K represents the keyword, and P represents the data location;

[0221] S83. Regularly update and maintain cached data and index structures to ensure data up-to-dateness and consistency; adopt a timestamp-based update strategy.

[0222] T new =T old +△t;

[0223] Among them, T new and T old These are the timestamps before and after the update, respectively, and △t is the time increment;

[0224] S84. During the retrieval process, the cache is checked first. If the cache is hit, the result is returned directly; otherwise, an index retrieval is performed.

[0225]

[0226] Where H is the cache hit rate, and C is the cache hit rate. hit To cache hit count, C total This represents the total number of queries.

[0227] S85. Utilizing parallel computing and distributed processing technologies, the index retrieval process is accelerated, enabling efficient operation in large-scale data environments:

[0228]

[0229] Where S is the acceleration ratio, T serial For serial computation time, T parallel For parallel computing time;

[0230] S86. Combining caching strategies and index optimization techniques, output the most relevant search results to the user's query.

[0231] Example 1:

[0232] To verify the feasibility of this invention in practice, we applied it to an e-commerce platform. First, we preprocessed all product data on the platform. Through data cleaning, we removed redundant and noisy data, ensuring data quality. Next, we annotated the product data in detail, such as adding category tags, brand tags, and function tags to each product. Then, we used advanced indexing techniques to build an efficient retrieval index from this data, laying the foundation for subsequent query processing.

[0233] When a user enters the query "portable and efficient power bank" in the search bar, our method first uses a large-scale pre-trained language model to perform semantic analysis on the user's query. The pre-trained language model can understand the deep semantics of the user's query; for example, it can identify the user's need for portability and efficiency in power banks, rather than simply matching the keyword "power bank." By generating a deep semantic representation of the query, we can more accurately understand the user's intent.

[0234] Next, we utilize deep learning networks to extract features from the preprocessed product data, generating high-dimensional feature vectors. These feature vectors capture various aspects of the product, such as capacity, charging speed, and brand. Then, we construct a distributed retrieval architecture to store these feature vectors in a distributed database for efficient retrieval.

[0235] After a user submits a query, we perform semantic matching in a distributed database based on the deep semantic representation of the query. By calculating the similarity between the feature vector of the user's query and the feature vector of the products, we can retrieve the most relevant products. In this process, we use a deep learning network to classify and rank the matched high-dimensional feature vectors, ensuring that the most relevant products rank at the top of the search results.

[0236] To continuously optimize search results, we combined online learning with real-time feedback mechanisms. The system can dynamically adjust the parameters of the pre-trained language model and deep learning network based on user clicks and purchase behavior, enabling the system to adaptively improve performance. Furthermore, we employed caching strategies and index optimization techniques to improve data retrieval response speed, ensuring users can quickly obtain search results.

[0237] In practical application, we conducted a three-month test on an e-commerce platform from May to July 2023. During this period, we collected user search and behavioral data and compared the effectiveness of traditional keyword matching methods with our proposed method.

[0238] The test results comparing the traditional keyword matching method and our proposed method are shown in Table 1 below:

[0239] Table 1 Data Comparison Table

[0240]

[0241] As shown in Table 1 above, using our method increased user search satisfaction by 30% and reduced average search response time by 50%. Specific data is as follows: In the first month of testing, the search response time using the traditional method was 1.2 seconds, while using our method, the search response time decreased to 0.6 seconds. In the second month, the click-through rate increased from 15% to 20%, and the purchase conversion rate increased from 5% to 8%. In the third month, user search satisfaction increased from 60% to 90%.

[0242] Based on the test data above, we can see that the data intelligent retrieval optimization method based on large models and deep learning significantly improves the user's search experience in practical applications on e-commerce platforms. It solves problems inherent in traditional methods, such as insufficient semantic understanding, limited ability to handle complex queries, low efficiency in processing high-dimensional data, and lack of dynamic adjustment mechanisms. This method not only improves the accuracy and efficiency of retrieval but also provides users with more personalized and accurate product recommendations, greatly enhancing user satisfaction and the platform's competitiveness.

[0243] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A data intelligent retrieval optimization method based on large models and deep learning, characterized in that: The following steps are involved: S1. Perform data preprocessing on the original data, wherein the data preprocessing includes data cleaning, data labeling and index construction; S2. Use a large-scale pre-trained language model to perform semantic analysis on user queries and generate deep semantic representations of queries; S3. Use the deep learning network to extract features from the preprocessed data to obtain a high-dimensional feature vector of the data; S4. Build a distributed retrieval architecture and store feature vectors in a distributed database; S5. According to the deep semantic representation of the user query, semantic matching is performed in the distributed database to retrieve relevant high-dimensional feature vectors; S6. Classify and sort the matched high-dimensional feature vectors through a deep learning network to filter out relevant data; S7. Combine online learning and real-time feedback mechanisms to dynamically adjust the parameters of large-scale pre-trained language models and deep learning networks; S8. Use cache strategies and index optimization techniques for data retrieval to output the retrieval results that are most relevant to the user's query.

2. According to claim 1, a data intelligent retrieval optimization method based on large models and deep learning is characterized in that: The S1 specifically includes: S11. Data cleaning, removing noise data and outliers from the original data, and using standardization and normalization methods to process numerical data: Among them, X is the original data, μ is the mean of the data, and σ is the standard deviation of the data; fill in the missing data, use interpolation or make missing value predictions: Among them, x i is a missing data point, x i-1 and x i+1 are adjacent data points; S12. Data labeling: labeling the data using a predefined label set; S13. Index construction: Use inverted index technology to build data index, vectorize data features, and generate feature vector matrix V, where each feature vector v i Represents the feature set of data item i.

3. According to claim 1, a data intelligent retrieval optimization method based on large models and deep learning is characterized in that: The S2 specifically includes: S21. Perform word segmentation on the user query to generate a word sequence Q = {q1, q2, ..., q n }, where q i is the i-th word in the query; S22. Use a large-scale pre-trained language model to encode the word sequence Q and generate a context embedding vector E = {e1, e2, …, e n }, where e i For word q i The context embedding vector of S23. Aggregate the context embedding vector E through a multi-layer attention mechanism to generate a global semantic representation vector S of the query: Among them, A i is the attention weight, Q is the query matrix, K is the key matrix, V is the value matrix, d k is the dimension of the key matrix; S24. Input the global semantic representation vector S into the deep neural network for further processing to generate the final query deep semantic representation S′, and the specific processing includes forward propagation and back propagation steps; S25. Output the deep semantic representation vector S′ of the query for subsequent retrieval and matching process.

4. According to claim 1, a data intelligent retrieval optimization method based on large models and deep learning is characterized in that: The S3 specifically includes: S31. Use inverted index technology to build a data index, where the inverted index structure includes document ID and keyword mapping; S32. Perform feature vectorization on the preprocessed data to generate a feature vector matrix V, where each feature vector v i Represents the feature set of data item i; S33. Use the local sensitive hashing method to hash the feature vector: Among them, a and b are randomly selected constants, w is the segment width, and v is the feature vector; S34. Use an optimized index structure B-tree or Trie tree to store and manage the hashed feature vectors; S35. Dynamically update the constructed index, combining the addition of new data and the deletion of old data to keep the index up to date and valid; S36. When searching, use the inverted index structure and the hashed feature vector to calculate the cosine similarity for matching: Among them, v i and v j are two eigenvectors, · represents the vector dot product, ||v|| represents the vector modulus; S37. Sort the search results according to the similarity scores and output the data most relevant to the user query.

5. According to claim 1, a data intelligent retrieval optimization method based on large model and deep learning is characterized in that: The S4 specifically includes: S41. Construct an efficient distributed retrieval architecture, wherein the distributed architecture includes a plurality of nodes, each node stores a portion of the data feature vector and can independently perform retrieval operations; S42. distribute the feature vector to each node according to a certain sharding strategy, which is based on the hash function: Among them, v i is the i-th component of the eigenvector, p i is the corresponding weight, N is the number of nodes; S43. In each node, a tree-structured indexing method KD tree or R tree is used to store and manage feature vectors; S44. Perform consistency processing on the data between distributed nodes, and use a consistent hashing algorithm to ensure uniform distribution and load balancing of data between nodes; S45. When performing a search, the user query is routed to the corresponding node set, and each node in the node set independently calculates its local search result; S46. The local search results of each node are merged and deduplicated, and the merged results are sorted using a global sorting algorithm, where the sorting criteria include the similarity of feature vectors and node response time: Among them, sim(v i ,q) represents the feature vector v i Similarity with query q, t i is the response time of node i, α and β are weighting coefficients; S47. Output the final sorted search results.

6. The data intelligent retrieval optimization method based on large model and deep learning according to claim 1 is characterized in that: The S5 specifically includes: S51. Calculate the similarity between the query vector S′ and the high-dimensional feature vector V stored in the database using cosine similarity; S52. Sort all feature vectors according to the similarity score and select the first k feature vectors {V1, V2, ..., V k } as the preliminary matching result; S53. Refine the preliminary matching results, use a large-scale pre-trained language model to perform secondary semantic analysis on the matching results and the query, and generate a similarity score: sim2(S′,V i )=α·sim(S′,V i )+β·BERT(S′,V i ); Among them, α and β are weighted coefficients, BERT(S′,V i ) represents the similarity score calculated by the large-scale pre-trained language model; S54. Combined with the secondary similarity score, the matching results are finally sorted to select the high-dimensional feature vector {V′1, V′2, …, V′ m }, where m≤k; S55. Output the final matching high-dimensional feature vector for subsequent data classification and sorting.

7. The data intelligent retrieval optimization method based on large model and deep learning according to claim 1 is characterized in that: The S6 specifically includes: S61. The matched high-dimensional feature vector {V′1, V′2, …, V′ m }Input into the pre-trained deep neural network; S62. Use adaptive sparse coding to process the feature vector in the output layer of the deep neural network to extract sparse features: Among them, X is the input feature vector, D is the dictionary matrix, α is the sparse representation, and λ is the regularization parameter; S63. Input the feature vector processed by adaptive sparse coding into the transformer model based on the attention mechanism for further feature extraction: Among them, Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key matrix; S64. Input the output of the self-attention mechanism into the multi-core support vector machine to capture the complex nonlinear relationship between features: Among them, K(x i ,x j ) is the kernel function, k m (x i ,x j ) is the mth basic kernel function, β m is the weight of the kernel function, M is the number of kernel functions; S65. Input the output of the multi-core support vector machine into the adaptive boosting model for further classification and feature extraction: Among them, F(x) is the final output of the classifier, α t is the weight of the tth weak classifier, h t (x) is the tth weak classifier, T is the total number of weak classifiers; S66. Input the output of the adaptive lifting model into the Gaussian process model for probability classification and uncertainty assessment: Where k(x,x′) is the covariance function of input x and x′, σ f is the signal standard deviation, l is the length scale; S67. According to the classification results of the Gaussian process classification model, the high-dimensional feature vectors are preliminarily sorted, and the first n feature vectors {V″1, V″2, …, V″ n } for further processing; S68. Refine the sorting of the initially sorted feature vectors, and calculate the comprehensive score by combining the semantic similarity and classification probability of the feature vectors: score(V″ i )=γ·sim(V″ i ,q)+δ·P(y=j|V″ i ); Among them, γ and δ are weighting coefficients, sim(V″ i ,q) is the eigenvector V″ i Semantic similarity with query q, P(y=j|V″ i ) is the eigenvector V″ i The probability of belonging to category j; S68. finally sorting the feature vectors according to the comprehensive scores, and selecting the high-dimensional feature vectors most relevant to the user query; S69. Output the sorted feature vector to generate the final retrieval result.

8. The data intelligent retrieval optimization method based on large model and deep learning according to claim 1 is characterized in that: The S7 specifically includes: S71. Obtain user retrieval behavior data and feedback data, use them as part of the training data, and generate a real-time feedback data set D feedback ; S72. The real-time feedback data set D feedback Compared with the original training data set D train Merge to form an extended training data set D ext ; S73. Using the extended training dataset D ext Perform online fine-tuning of the loss function on the large-scale pre-trained language model and adjust the model parameters θ: Among them, y i is the actual label, is the model prediction probability, N is the number of training samples; S74. Use gradient descent method to dynamically update the parameters of deep learning network: Among them, θ t is the parameter of the tth iteration, η is the learning rate, is the loss function L with respect to parameter θ t The gradient of S75. Use information such as user clicks and dwell time in real-time feedback data to adjust the weighting coefficient of the model and optimize the relevance of the search results: a new =a old +β·Δfeedback; Among them, α new and α old are the weighting coefficients before and after adjustment, β is the adjustment rate, and Δfeedback is the feedback change; S76. Continuously update and optimize the parameters of large-scale pre-trained language models and deep learning networks through periodic online learning and real-time feedback mechanisms.

9. The data intelligent retrieval optimization method based on large model and deep learning according to claim 1 is characterized in that: The S8 specifically includes: S81. Cache frequently-queried and frequently-accessed data, save the data with the highest cache hit rate in the cache, and use the least recently used algorithm for cache management: C new =C old +△t·F; Among them, C new and C old They are the cache states before and after adjustment, △ is the time increment, and F is the access frequency; S82. Build an optimized index structure based on different data features, and use a hybrid index structure based on inverted index and B-tree to improve index retrieval efficiency: I=(K,P); Among them, I is the index structure, K is the keyword, and P is the data position; S83. Regularly update and maintain cache data and index structures to ensure data freshness and consistency; adopt a timestamp-based update strategy: T new =T old +△t; Among them, T new and T old are the timestamps after and before the update, respectively, and △t is the time increment; S84. During the retrieval process, the cache is queried first. If the cache hits, the result is returned directly. Otherwise, an index search is performed: Among them, H is the cache hit rate, C hit is the number of cache hits, C total is the total number of queries; S85. Use parallel computing and distributed processing technology to accelerate the index retrieval process and run efficiently in a large-scale data environment: Among them, S is the acceleration ratio, T serial is the serial computation time, T parallel The parallel computing time; S86. Combine caching strategies and index optimization techniques to output the retrieval results that are most relevant to the user's query.

Images