A big data text information retrieval method based on AI

Abstract

The present invention discloses a big data text information retrieval method based on AI, which relates to the technical field of text information retrieval, and the steps include comparing text key information with pre-stored text key information of each pre-stored text stored in a database to obtain a first output text list, comparing the TF-IDF vector of the text to be retrieved with the pre-stored TF-IDF vector of each pre-stored text stored in the database to obtain a second output text list, merging and sorting the first output text list and the second output text list to obtain a sorted text list to be output, and adjusting and sorting each sorted text to be output in the sorted text list to be output based on a text information data set, and then outputting a final text list. The present invention extracts key information from texts through AI technology, improves the accuracy and relevance of information, and reduces the need for manual annotation through automated processing, thereby improving retrieval efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of text information retrieval, and in particular to a big data text information retrieval method based on AI. Background Art

[0002] In the field of big data text information retrieval, artificial intelligence technology plays an important role. Through machine learning and natural language processing, AI can extract key information from massive data and achieve fast and accurate retrieval. In addition, AI technology can dynamically understand user intentions, provide personalized recommendations, reduce information redundancy, and improve user retrieval efficiency. AI also continuously optimizes retrieval models and improves the ability to understand context through methods such as deep learning, making search results more relevant and providing users with a better retrieval experience.

[0003] However, the application of existing AI technology in text information retrieval still has some shortcomings. On the one hand, relying on a single feature for comparison leads to inaccurate matching and inability to fully capture the similarities between texts, thus affecting the relevance and accuracy of the retrieval results. On the other hand, failure to fully utilize the user's historical retrieval data will result in the use of outdated data, resulting in a poor user experience and inability to provide personalized recommendations based on the user's historical behavior, resulting in inefficiency in user information acquisition and a more serious information overload. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides an AI-based big data text information retrieval method, which extracts key information in the text through AI technology, improves the accuracy and relevance of the information, and reduces the need for manual labeling through automated processing, thereby improving retrieval efficiency.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] A big data text information retrieval method based on AI, comprising the following steps:

[0007] Obtain the text to be searched entered by the user, and extract key information of the text based on AI technology, including key topic information, key entity information, key relationship information, and key context information;

[0008] Comparing the text key information with the pre-stored text key information of each pre-stored text stored in the database to obtain a first output text list, the pre-stored text key information includes pre-stored text key subject information, pre-stored text key entity information, pre-stored text key relationship information, and pre-stored text key scenario information, the first output text list includes a plurality of first pre-stored texts obtained by first matching;

[0009] Obtaining a TF-IDF vector of the text to be searched, comparing the TF-IDF vector of the text to be searched with the pre-stored TF-IDF vectors of each pre-stored text stored in the database, and obtaining a secondary output text list, wherein the secondary output text list includes a plurality of second pre-stored texts obtained by secondary matching;

[0010] Merge and sort the first output text list and the second output text list to obtain a sorted list of texts to be output, wherein the sorted list of texts to be output includes a plurality of sorted texts to be output;

[0011] Obtaining a text information data set of each sorted text to be output in the sorted text to be output list, including historical search information data and text background data;

[0012] Based on the text information data set, each sorted text to be output in the sorted text list to be output is adjusted and sorted, and then the final text list is output. The historical retrieval information data includes the number of historical query records, average visit time, click rate, and bounce rate. The text background data includes the number of times the text is cited, the number of text references, and the frequency of text data updates.

[0013] Preferably, obtaining the first output text list comprises the following steps:

[0014] Compare the key topic information with the pre-stored text key topic information in each pre-stored text to obtain the similarity of each topic;

[0015] Compare the key entity information with the pre-stored text key entity information in each pre-stored text to obtain the similarity of each entity;

[0016] Compare the key relationship information with the pre-stored text key relationship information in each pre-stored text to obtain the similarity of each relationship;

[0017] Comparing the key scenario information with the pre-stored text key scenario information in each pre-stored text to obtain the similarity of each scenario;

[0018] Comprehensively analyze the topic similarity, entity similarity, relationship similarity and scenario similarity corresponding to each pre-stored text to obtain the first comprehensive similarity of each text;

[0019] Compare the first comprehensive similarity of each text with the first comprehensive similarity threshold of the text stored in the database in turn:

[0020] If the first comprehensive similarity of the text is less than the first comprehensive similarity threshold of the text, the pre-stored text corresponding to the first comprehensive similarity of the text is not selected;

[0021] If the first comprehensive similarity of the text is not less than the first comprehensive similarity threshold of the text, the pre-stored text corresponding to the first comprehensive similarity of the text is recorded as the first pre-stored text;

[0022] The obtained multiple first pre-stored texts are recorded as a first output text list.

[0023] Preferably, the calculation formula for the first comprehensive similarity of texts is:

[0024] ;

[0025] In the formula, is the first comprehensive similarity between the text to be retrieved and the text corresponding to the i-th pre-stored text, is the topic similarity between the text to be retrieved and the i-th pre-stored text, is the entity similarity between the text to be retrieved and the i-th pre-stored text, is the relationship similarity between the text to be retrieved and the i-th pre-stored text, is the scene similarity between the text to be retrieved and the i-th pre-stored text, and i is the number of each pre-stored text stored in the database.

[0026] Preferably, obtaining the secondary output text list comprises the following steps:

[0027] Compare the TF-IDF vector of the text to be retrieved with the pre-stored TF-IDF vectors of each pre-stored text to obtain the secondary comprehensive similarity of the text;

[0028] Compare the secondary comprehensive similarity of each text with the secondary comprehensive similarity threshold of the text stored in the database in turn:

[0029] If the secondary comprehensive similarity of the text is less than the secondary comprehensive similarity threshold of the text, the pre-stored text corresponding to the secondary comprehensive similarity of the text is not selected;

[0030] If the secondary comprehensive similarity of the text is not less than the secondary comprehensive similarity threshold of the text, the pre-stored text corresponding to the secondary comprehensive similarity of the text is recorded as the second pre-stored text;

[0031] The obtained multiple second pre-stored texts are recorded as a secondary output text list.

[0032] Preferably, obtaining the sorted list of texts to be output comprises the following steps:

[0033] The first output text list and the second output text list are merged to obtain an unsorted list of texts to be output, wherein the unsorted list of texts to be output includes a plurality of unsorted texts to be output;

[0034] Obtaining pre-stored text key information of each unsorted text to be output stored in a database, comparing the text key information of the text to be retrieved with the pre-stored text key information of each unsorted text to be output, and obtaining a first comprehensive similarity corresponding to each unsorted text to be output;

[0035] The obtained pre-stored TF-IDF vectors of the unsorted texts to be output are stored in the database, and the TF-IDF vectors of the texts to be retrieved are compared with the pre-stored TF-IDF vectors of the unsorted texts to be output, so as to obtain the second comprehensive similarity corresponding to the unsorted texts to be output;

[0036] Comprehensively analyzing the first comprehensive similarities and the second comprehensive similarities corresponding to each unsorted to-be-output text to obtain each third comprehensive similarity;

[0037] The third comprehensive similarities are sorted from large to small to obtain corresponding sorted texts to be output, and each sorted text to be output is output as a sorted text list to be output.

[0038] Preferably, the calculation formula of the third comprehensive similarity is:

[0039] ;

[0040] In the formula, is the bth third comprehensive similarity, is the first comprehensive similarity of the bth unsorted text to be output, is the second comprehensive similarity of the bth unsorted text to be output, Stored in the database The weight factor of Stored in the database The weight factor of is, and b is the number of each unsorted text to be output in the list of texts to be output.

[0041] Preferably, adjusting and sorting each sorted text to be output in the sorted text list to be output based on the text information data set and then outputting the final text list comprises the following steps:

[0042] Obtaining historical search information data of each sorted text to be output, analyzing each historical search information data, and obtaining a search feature value of each text;

[0043] Obtaining text background data of each sorted text to be output, analyzing each text background data, and obtaining each text background feature value;

[0044] Combining the text retrieval feature value and the text background feature value corresponding to each sorted text to be output, to obtain the priority evaluation value corresponding to each sorted text to be output;

[0045] The priority rating values ​​are arranged from large to small, the corresponding sorted texts to be output are sorted again, and the final text list is output.

[0046] Preferably, the calculation formula of the text retrieval feature value is:

[0047] ;

[0048] In the formula, Retrieve the feature value for the mth sorted output text, is the number of historical query records for the mth sorted text to be output. is the access time of the mth sorted text to be output, is the click rate of the mth sorted text to be output, is the bounce rate of the mth sorted text to be output, where m is the number of the sorted text to be output.

[0049] Preferably, the calculation formula of the text background feature value is:

[0050] ;

[0051] In the formula, is the text background feature value of the mth sorted text to be output, is the number of citations of the mth sorted text to be output. is the number of text references of the mth sorted text to be output, is the text data update frequency of the mth sorted text to be output, and m is the number of the sorted text to be output.

[0052] Preferably, the calculation formula of the priority rating value is:

[0053] ;

[0054] In the formula, is the priority rating value of the mth sorted text to be output, Retrieve the feature value for the mth sorted output text, is the text background feature value of the mth sorted text to be output, Stored in the database The weight factor of Stored in the database The weight factor of , e is a natural constant, and m is the number of the sorted text to be output.

[0055] The present invention has the following beneficial effects:

[0056] The present invention uses AI technology to extract key information from texts, greatly improving the accuracy and relevance of information. Automated processing reduces the need for manual annotation and improves retrieval efficiency. The reliability of retrieval results is enhanced through two different matching steps. The multi-dimensional comparison method ensures that users can obtain more relevant and accurate texts. The historical retrieval information and text background data are integrated, and the sorting of retrieval results can be dynamically adjusted, thereby improving user satisfaction and retrieval efficiency. Personalized recommendations can be provided based on user historical behavior, making it easier for users to find the information they need, reducing information overload, making users more efficient in obtaining information, and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings.

[0059] like Figure 1 As shown, a big data text information retrieval method based on AI includes the following steps:

[0060] Obtain the text to be retrieved input by the user, and extract key information of the text based on AI technology, including key topic information, key entity information, key relationship information, and key contextual information.

[0061] Key topic information is obtained by using a topic modeling algorithm (LDA) to identify potential topics in the text. LDA assumes that the text to be retrieved is composed of multiple topics, and each topic is represented by a set of word distributions. Generative models (such as BERT, GPT, etc.) are used to generate text summaries and extract important topics or concepts from them. Key entity information is extracted by using pre-trained named entity recognition (NER) models (such as SpaCy, BERT, etc.) to identify entities in the text to be retrieved, such as names, places, organizations, etc. Key relationship information is obtained by using deep learning models (such as relationship classification models) to identify the relationship between different entities. Methods such as Siamese networks and graph neural networks can be used. Key contextual information evaluates the context of the text through contextual information by applying sentiment analysis and contextual understanding models. Use contextual analysis tools to extract background information such as time, place, and environment.

[0062] The text key information is compared with the pre-stored text key information of each pre-stored text stored in the database to obtain a first output text list, wherein the pre-stored text key information includes pre-stored text key subject information, pre-stored text key entity information, pre-stored text key relationship information, and pre-stored text key scenario information. The first output text list includes multiple first pre-stored texts obtained by the first match.

[0063] Compare the key topic information with the pre-stored text key topic information in each pre-stored text to obtain each topic similarity; compare the key entity information with the pre-stored text key entity information in each pre-stored text to obtain each entity similarity; compare the key relationship information with the pre-stored text key relationship information in each pre-stored text to obtain each relationship similarity; compare the key scenario information with the pre-stored text key scenario information in each pre-stored text to obtain each scenario similarity; comprehensively analyze the topic similarity, entity similarity, relationship similarity and scenario similarity corresponding to each pre-stored text to obtain the first comprehensive similarity of each text.

[0064] The first comprehensive similarity of each text is compared with the first comprehensive similarity threshold of the text stored in the database in turn: if the first comprehensive similarity of the text is less than the first comprehensive similarity threshold of the text, the pre-stored text corresponding to the first comprehensive similarity of the text is not selected; if the first comprehensive similarity of the text is not less than the first comprehensive similarity threshold of the text, the pre-stored text corresponding to the first comprehensive similarity of the text is recorded as the first pre-stored text; the multiple first pre-stored texts obtained are recorded as the first output text list.

[0065] The calculation formula for the first comprehensive similarity of text is:

[0066] ;

[0067] In the formula, is the first comprehensive similarity between the text to be retrieved and the text corresponding to the i-th pre-stored text, is the topic similarity between the text to be retrieved and the i-th pre-stored text, is the entity similarity between the text to be retrieved and the i-th pre-stored text, is the relationship similarity between the text to be retrieved and the i-th pre-stored text, is the scene similarity between the text to be retrieved and the i-th pre-stored text, and i is the number of each pre-stored text stored in the database.

[0068] Among them: The calculation formula for topic similarity is:

[0069] ;

[0070] In the formula, is the topic vector representation of the text to be retrieved, which is used to represent key topic information. is the topic vector representation of the i-th pre-stored text stored in the database, which is used to represent the key topic information of the pre-stored text. for The model, for Model.

[0071] The calculation formula for entity similarity is:

[0072] ;

[0073] In the formula, is a collection of entity features of the text to be retrieved, used to represent key entity information. It is a set of entity features of the i-th pre-stored text stored in the database, used to represent the key entity information of the pre-stored text.

[0074] The calculation formula for relationship similarity is:

[0075] ;

[0076] In the formula, is a set of relational features of the text to be retrieved, used to represent key relational information. It is a set of relational features of the i-th pre-stored text stored in the database, used to represent the key relational information of the pre-stored text.

[0077] The calculation formula for scenario similarity is:

[0078] ;

[0079] In the formula, is the context vector representation of the text to be retrieved, which is used to represent key context information. is the context vector representation of the i-th pre-stored text stored in the database, used to represent the key context information of the pre-stored text. for The model, for Model.

[0080] Both topic similarity and context similarity are calculated using cosine similarity. By using cosine similarity to compare topic vectors, it reflects the similarity between the text to be retrieved and the i-th pre-stored text in terms of topic expression, which can effectively capture the similarity between the texts in terms of keywords and main concepts. Calculating context similarity using cosine similarity emphasizes the similarity between texts in paragraph structure and context, which is an extremely important feature in many content review and recommendation systems.

[0081] Both entity similarity and relationship similarity are calculated using set theory methods. The intersection and union of sets are used to evaluate the similarity between the text to be retrieved and the i-th pre-stored text entity, emphasizing the degree of sharing between the entities mentioned in the text, which is crucial for identifying whether the content can resonate with users. The intersection and union of sets are used to evaluate the similarity between the text to be retrieved and the i-th pre-stored text relationship, emphasizing the matching degree of relationship information in the text to evaluate the accuracy and logical coherence of the information.

[0082] The comprehensive similarity calculation formula calculates the comprehensive similarity between the text to be retrieved and the i-th pre-stored text, showing how the calculation of comprehensive similarity combines similarities of different dimensions (topic, entity, relationship, context), reflecting the multi-dimensional consideration of the text similarity between the text to be retrieved and the i-th pre-stored text, and can fully capture the similarity between texts. It helps to improve the accuracy of the search and ensure that the retrieval results are not just literal keyword matches. For example, relying only on topic similarity may ignore important relationships between entities or contexts.

[0083] Comprehensive calculations can reflect textual relationships at multiple levels. This structure can dynamically adapt to new textual data and optimize similarity calculations by continuously updating vector representations of topics, entities, relationships, and contexts. This enables the model to continuously learn and improve, which is particularly important for real-time changing information environments.

[0084] The TF-IDF vector of the text to be searched is obtained, and the TF-IDF vector of the text to be searched is compared with the pre-stored TF-IDF vectors of each pre-stored text stored in the database to obtain a secondary output text list, which includes multiple second pre-stored texts obtained by secondary matching.

[0085] The TF-IDF vector is a commonly used weighting technique for information retrieval and text mining that reflects the importance of a word in a document, taking into account both the frequency of the word in the document (TF) and its rarity in the entire document collection (IDF). The components of TF-IDF include:

[0086] TF (Term Frequency) indicates the frequency of a word appearing in a document; IDF (Inverse Document Frequency) measures the importance of a word in the entire document collection. The rarer the word, the higher the IDF value. For a document, the TF-IDF value of each word can be calculated and represented as a vector, which is the TF-IDF vector.

[0087] The calculation formula of the TF-IDF vector of the text to be retrieved is:

[0088] ;

[0089] In the formula, is the TF-IDF vector representation of the text to be retrieved, For specific words The TF-IDF value of For specific words The TF-IDF value of For specific words The TF-IDF value of is the ath specific word in the file to be searched, and a is each specific word in the text to be searched The number of is the text to be searched, For background documentation.

[0090] in:

[0091] ;

[0092] In the formula, For specific words The frequency of occurrence in the text to be searched, For specific words exist The inverse document frequency in .

[0093] ;

[0094] In the formula, For specific words The number of times it appears in the text to be searched. is the total number of words in the text to be searched.

[0095] ;

[0096] In the formula, For background documents containing specific words The number of source documents, is the cth source document in the background document, and c is the number of each source document stored in the database.

[0097] The calculation method of the pre-stored TF-IDF vector of each pre-stored text is the same as that of the TF-IDF vector of the text to be retrieved, except that Replace with the total stored document containing all stored text.

[0098] The TF-IDF model converts the text to be retrieved into a vector by calculating the TF-IDF value of a specific word, which is convenient for comparison with the pre-stored text in the database. It can ensure that the main meaning and key information of the text are effectively represented in the vector, making the text retrieval process standardized and efficient. Each word in the text to be retrieved is quantified into a numerical value, so that the similarity can be quickly calculated through a mathematical algorithm, and finally a secondary output text list is formed to find the pre-stored text similar to the text to be retrieved, which is used for subsequent integration with the first output text list above.

[0099] TF-IDF can effectively reduce the influence of common words in similarity calculations and emphasize the importance of unique words. This weight adjustment can significantly improve the accuracy of similarity matching, thereby ensuring that the returned text is more relevant to the user's needs. Using TF-IDF vectors can quickly process large-scale data sets and organize efficient information retrieval processes. When a request is made, the system can quickly calculate the similarity and provide relevant text, avoiding the inefficiency and tediousness of manual search.

[0100] The TF-IDF vector of the text to be retrieved is compared with the pre-stored TF-IDF vector of each pre-stored text to obtain the secondary comprehensive similarity of the text; the secondary comprehensive similarity of each text is compared with the secondary comprehensive similarity threshold of the text stored in the database in turn: if the secondary comprehensive similarity of the text is less than the secondary comprehensive similarity threshold of the text, the pre-stored text corresponding to the secondary comprehensive similarity of the text is not selected; if the secondary comprehensive similarity of the text is not less than the secondary comprehensive similarity threshold of the text, the pre-stored text corresponding to the secondary comprehensive similarity of the text is recorded as the second pre-stored text; and the obtained multiple second pre-stored texts are recorded as a secondary output text list.

[0101] The calculation formula of the secondary comprehensive similarity of text is:

[0102] ;

[0103] In the formula, is the secondary comprehensive similarity between the TF-IDF vector of the text to be retrieved and the pre-stored TF-IDF vector of the i-th pre-stored text, is the TF-IDF vector representation of the text to be retrieved, is the pre-stored TF-IDF vector representation of the i-th pre-stored text stored in the database, for The model, for Model.

[0104] This formula provides a quantitative indicator for the content relevance between the text to be retrieved and each pre-stored text. Through this quantitative analysis, it is possible to determine which texts are closest to the user's query, thereby achieving efficient information retrieval and avoiding the time-consuming process of users screening through large amounts of information. By comparing the calculated similarity value with the threshold, it is possible to effectively filter out those pre-stored texts that have a low similarity with the text to be retrieved, ensuring that the output content is highly relevant, which helps to improve the user's retrieval experience. Generate personalized recommendation lists based on the user's specific needs and interests to adapt to the preferences of different users. It can be flexibly applied to a variety of natural language processing scenarios, including search engines, text classification, sentiment analysis, etc., providing a consistent basis for text processing in different scenarios.

[0105] The first output text list and the second output text list are merged and sorted to obtain a sorted list of texts to be output, wherein the sorted list of texts to be output includes a plurality of sorted texts to be output.

[0106] The first output text list and the second output text list are merged to obtain an unsorted list of texts to be output, wherein the unsorted list of texts to be output includes a plurality of unsorted texts to be output; pre-stored text key information of each unsorted text to be output stored in a database is obtained, and the text key information of the text to be retrieved is compared with the pre-stored text key information of each unsorted text to be output to obtain a first comprehensive similarity corresponding to each unsorted text to be output; the pre-stored TF-IDF vector of each unsorted text to be output stored in the database is obtained, and the TF-IDF vector of the text to be retrieved is compared with the pre-stored TF-IDF vector of each unsorted text to be output to obtain a second comprehensive similarity corresponding to each unsorted text to be output; the first comprehensive similarity corresponding to each unsorted text to be output is comprehensively analyzed with the second comprehensive similarity to obtain each third comprehensive similarity; the third comprehensive similarities are sorted from large to small to obtain the corresponding sorted text to be output, and each sorted text to be output is output as a sorted list of texts to be output.

[0107] The calculation formula of the third comprehensive similarity is:

[0108] ;

[0109] In the formula, is the bth third comprehensive similarity, is the first comprehensive similarity of the bth unsorted text to be output, is the second comprehensive similarity of the bth unsorted text to be output, Stored in the database The weight factor of Stored in the database The weight factor of is, and b is the number of each unsorted text to be output in the list of texts to be output.

[0110] By combining the two similarities of content and TF-IDF, we focus on the theme and semantics of the text as well as the degree of vocabulary matching, and combine the similarities obtained by different methods to achieve a comprehensive analysis of text similarity. It can solve the details that may be lost by a single similarity indicator, provide users with more relevant text results, and make the evaluation of text more comprehensive and less susceptible to interference from certain specific factors. Let the most relevant texts in the list be ranked first, and the texts with low relevance be ranked at the back, to ensure the relevance and quality of the search results, so that users can locate the required information more quickly. Through adjustable weight factors, similarity calculations can be optimized for different application scenarios, so that the system can adapt to different user needs. Accurate similarity calculations can effectively filter out more relevant and higher-quality texts, improve the efficiency of users in obtaining information, and thus enhance user experience. The weight parameters are continuously adjusted according to actual usage, and the recommendation algorithm is improved using user interaction data, thereby forming a self-optimization mechanism and enhancing the intelligence level of the system.

[0111] The above setting , The first comprehensive similarity of the bth unsorted text to be output, the second comprehensive similarity of the bth unsorted text to be output and the bth third comprehensive similarity obtained by historical calculation are established from the database according to the historical data, and a mapping set of the first comprehensive similarity of the bth unsorted text to be output, the second comprehensive similarity of the bth unsorted text to be output and their corresponding weight factors is established to obtain the current , . , They are also obtained through a mapping set of historical data and weight factors established in the database, that is, the corresponding weight factors are obtained according to the current data.

[0112] Obtain a text information data set of each sorted text to be output in the sorted text list to be output, including historical search information data and text background data; adjust and sort each sorted text to be output in the sorted text list to be output based on the text information data set, and then output a final text list, the historical search information data includes the number of historical query records, average access time, click rate, and bounce rate, and the text background data includes the number of times the text is cited, the number of text references, and the frequency of text data update.

[0113] Obtain the historical search information data of each sorted text to be output, analyze each historical search information data, and obtain each text search feature value; obtain the text background data of each sorted text to be output, analyze each text background data, and obtain each text background feature value; combine the text search feature value corresponding to each sorted text to be output with the text background feature value to obtain the priority assessment value corresponding to each sorted text to be output; arrange the priority assessment values ​​from large to small, sort the corresponding sorted texts to be output again, and output the final text list.

[0114] The calculation formula of text retrieval feature value is:

[0115] ;

[0116] In the formula, Retrieve the feature value for the mth sorted output text, is the number of historical query records for the mth sorted text to be output. is the access time of the mth sorted text to be output, is the click rate of the mth sorted text to be output, is the bounce rate of the mth sorted text to be output, where m is the number of the sorted text to be output.

[0117] The number of historical query records indicates the total number of queries that users have made for a specific text, reflecting the popularity of the text. It is obtained through log records or database statistics. The average visit duration indicates the average length of time that users stay when viewing the text, which can indicate the attractiveness and depth of the content. It is monitored through web analytics tools (Google Analytics, Matomo, etc.) and user behavior tracking systems. The click-through rate indicates the ratio of the number of times users click on the retrieved text relative to the total number of searches. A high click-through rate usually indicates that the text is highly relevant. It is obtained by analyzing website or database access logs. The bounce rate indicates the proportion of users who leave without further interaction after visiting the text. A high bounce rate may indicate that the content fails to meet user expectations. It is obtained through web analytics tools.

[0118] Using a logarithmic function can reduce the range of the data and make the data vary at a level that is easier to understand and process. Adding 1 can prevent the negative infinity problem when the numerator is 0. The square root helps to balance the impact of different factors, weakening the impact of extreme values ​​and reflecting the comprehensive performance of these indicators. Summarizing the three historical performances of text popularity, user access time, and click-through rate reflects the attractiveness of the text content to users. The bounce rate is an inverse indicator. The higher the bounce rate, the more dissatisfied or uninterested the user is in the content. Putting it in the denominator is to reduce the retrieval feature value of the text, reflecting its negative impact on the results.

[0119] This formula is designed to provide a comprehensive evaluation index for the sorted text to be output, helping the system to select more relevant and valuable content when matching and sorting a large amount of text. By reasonably sorting and displaying more attractive and relevant content, it ensures that users can quickly find the information they need. Historical query records, access time, click-through rate and other data are all quantitative indicators, which can provide a more consistent and objective text quality evaluation method to avoid the influence of subjective factors. As user behavior changes (such as an increase in click-through rate or a change in access time), the text retrieval feature value will also change accordingly, ensuring the real-time relevance of the output text and adapting to the changing needs of users.

[0120] The calculation formula of the text background feature value is:

[0121] ;

[0122] In the formula, is the text background feature value of the mth sorted text to be output, is the number of citations of the mth sorted text to be output. is the number of text references of the mth sorted text to be output, The text data update frequency of the mth sorted text to be output.

[0123] The number of citations of a text indicates how often the text is cited in other documents, which can reflect the academic influence of the text. The data is queried through academic databases and citation indexes (such as Google Scholar and Web of Science). The number of references in a text indicates the total number of references cited in the text, which may reflect the depth and breadth of the text in research. It is obtained from the text itself or through document management software. The frequency of text data updates indicates the time interval between text updates, which reflects the timeliness of the text. Update records are tracked through databases or content management systems.

[0124] Through Applying the square root indicates the importance of this factor to the overall evaluation, but reduces the high impact of extreme values ​​on the results and enhances stability. Using the logarithmic function can convert data of different scales into comparable values, and can also reduce the impact of excessive values ​​on the results, while adding 1 to avoid the logarithm being negative infinity. The numerator combines multiple background features of the text to calculate a more comprehensive text relevance evaluation. This formula is designed to evaluate the richness of cultural, social and background information of the sorted output text, so as to provide users with more in-depth and understanding content. By increasing the value of background features, it can ensure that the text content is not limited to surface information, but contains enough depth and context to better meet the needs of users.

[0125] By considering the background features of the text, the system will prioritize texts with richer information, which helps users understand information in multiple dimensions and improves the user experience. Background features can help the system recommend relevant content to users more accurately through indicators in different dimensions, thereby enhancing user interest and engagement. This ensures the continuous output of high-quality text and the real-time relevance of content recommendations.

[0126] The priority rating value is calculated as follows:

[0127] ;

[0128] In the formula, is the priority rating value of the mth sorted text to be output, Retrieve the feature value for the mth sorted output text, is the text background feature value of the mth sorted text to be output, Stored in the database The weight factor of Stored in the database The weight factor of , e is a natural constant.

[0129] Applying the square root to the linearly combined values ​​has the effect of narrowing the range to ensure the smoothness of the final rating value and reduce the impact of extreme values. The newly constructed feature values ​​show the combined impact of the text retrieval feature values ​​and the text background feature values. The use of exponential functions allows even small changes to have a significant impact on the priority rating value, reflecting its importance to the recommendation strategy. The core function of this formula is to calculate the priority for the text to be output, helping the system decide which texts should be displayed to the user first. By combining retrieval relevance and text background depth, the system can generate content recommendations that better meet user needs and improve user experience.

[0130] The priority rating value calculated by the above formula can evaluate the priority of each sorted text to be output, and the system can effectively sort it, so as to give priority to displaying the most valuable information to users and improve the overall user experience. The calculation of the priority value takes into account the feedback of user behavior, so that the text output is not only based on past data, but also can flexibly respond to current changes in demand. Combining retrieval features with background features can more accurately reflect the actual needs of users, reduce the display of invalid information, and thus improve the efficiency of information retrieval. The priority rating value promotes the system to recommend higher quality content to users and promotes interaction and participation between users and text. In practical applications, it can be widely used in fields such as literature retrieval, document recommendation, and online learning platforms, and gradually improve the accuracy of information retrieval and the user experience.

[0131] In a specific example, when the text to be retrieved matches the i-th pre-stored text for the first time, To simplify the calculation, we get is 0.96, To simplify the calculation, we get is 0.51, To simplify the calculation, we get is 0.43, To simplify the calculation, we get is 0.98, calculated is 2.88, and the first comprehensive similarity threshold of the text is 1.3. If the text is not less than the first comprehensive similarity threshold, the The corresponding pre-stored text is recorded as the first pre-stored text, and other pre-stored texts are similarly calculated to obtain the first output text list.

[0132] The text to be retrieved is matched with the i-th pre-stored text twice. To simplify the calculation, we get is 0.72, and the text secondary comprehensive similarity threshold is 0.33. If the value is not less than the secondary comprehensive similarity threshold of the text, the pre-stored text corresponding to the secondary comprehensive similarity of the text is recorded as the second pre-stored text, and the similarity of other pre-stored texts is calculated to obtain a secondary output text list, and the secondary output text list is combined with the first output text list to obtain an unsorted list of texts to be output.

[0133] The text to be retrieved is compared with the unsorted text to be output in the bth unsorted text to be output list. To simplify the calculation, we get is 2.53, To simplify the calculation, we get is 0.75, is 0.47, is 0.53, and we get The value is 1.59. The other unsorted text lists to be output are similarly calculated to obtain the third comprehensive similarities, and are sorted from large to small to obtain the corresponding sorted text list to be output.

[0134] In the historical search information data, the mth sorted text to be output For 150 times, 80 seconds, 5%, is 10%, calculated It is 27.79.

[0135] In the text background data, the mth sorted text to be output For 200 times, For 50 articles, 2 times / year, calculate is 4.21, is 0.6, is 0.4, and the calculated is 72.24. According to its size in each priority rating value, the corresponding Adjust the position of the sorted text to be output.

[0136] The algorithm involved in this embodiment can be executed by an electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The above algorithm calculation is implemented by executing the software by the processor.

Claims

1. A big data text information retrieval method based on AI, characterized in that: The following steps are involved: Obtain the text to be searched entered by the user, and extract key information of the text based on AI technology, including key topic information, key entity information, key relationship information, and key context information; Comparing the text key information with the pre-stored text key information of each pre-stored text stored in the database to obtain a first output text list, the pre-stored text key information includes pre-stored text key subject information, pre-stored text key entity information, pre-stored text key relationship information, and pre-stored text key scenario information, the first output text list includes a plurality of first pre-stored texts obtained by first matching; Obtaining a TF-IDF vector of the text to be searched, comparing the TF-IDF vector of the text to be searched with the pre-stored TF-IDF vectors of each pre-stored text stored in the database, and obtaining a secondary output text list, wherein the secondary output text list includes a plurality of second pre-stored texts obtained by secondary matching; Merge and sort the first output text list and the second output text list to obtain a sorted list of texts to be output, wherein the sorted list of texts to be output includes a plurality of sorted texts to be output; Obtaining a text information data set of each sorted text to be output in the sorted text to be output list, including historical search information data and text background data; Based on the text information data set, each sorted text to be output in the sorted text list to be output is adjusted and sorted, and then the final text list is output. The historical retrieval information data includes the number of historical query records, average visit time, click rate, and bounce rate. The text background data includes the number of times the text is cited, the number of text references, and the frequency of text data updates.

2. According to the AI-based big data text information retrieval method of claim 1, it is characterized in that: Getting the first output text list includes the following steps: Compare the key topic information with the pre-stored text key topic information in each pre-stored text to obtain the similarity of each topic; Compare the key entity information with the pre-stored text key entity information in each pre-stored text to obtain the similarity of each entity; Compare the key relationship information with the pre-stored text key relationship information in each pre-stored text to obtain the similarity of each relationship; Comparing the key scenario information with the pre-stored text key scenario information in each pre-stored text to obtain the similarity of each scenario; Comprehensively analyze the topic similarity, entity similarity, relationship similarity and scenario similarity corresponding to each pre-stored text to obtain the first comprehensive similarity of each text; Compare the first comprehensive similarity of each text with the first comprehensive similarity threshold of the text stored in the database in turn: If the first comprehensive similarity of the text is less than the first comprehensive similarity threshold of the text, the pre-stored text corresponding to the first comprehensive similarity of the text is not selected; If the first comprehensive similarity of the text is not less than the first comprehensive similarity threshold of the text, the pre-stored text corresponding to the first comprehensive similarity of the text is recorded as the first pre-stored text; The obtained multiple first pre-stored texts are recorded as a first output text list.

3. The AI-based big data text information retrieval method according to claim 2 is characterized in that: The calculation formula for the first comprehensive similarity of text is: ； In the formula, is the first comprehensive similarity between the text to be retrieved and the text corresponding to the i-th pre-stored text, is the topic similarity between the text to be retrieved and the i-th pre-stored text, is the entity similarity between the text to be retrieved and the i-th pre-stored text, is the relationship similarity between the text to be retrieved and the i-th pre-stored text, is the scene similarity between the text to be retrieved and the i-th pre-stored text, and i is the number of each pre-stored text stored in the database.

4. The AI-based big data text information retrieval method according to claim 1, characterized in that: Obtaining the secondary output text list includes the following steps: Compare the TF-IDF vector of the text to be retrieved with the pre-stored TF-IDF vectors of each pre-stored text to obtain the secondary comprehensive similarity of the text; Compare the secondary comprehensive similarity of each text with the secondary comprehensive similarity threshold of the text stored in the database in turn: If the secondary comprehensive similarity of the text is less than the secondary comprehensive similarity threshold of the text, the pre-stored text corresponding to the secondary comprehensive similarity of the text is not selected; If the secondary comprehensive similarity of the text is not less than the secondary comprehensive similarity threshold of the text, the pre-stored text corresponding to the secondary comprehensive similarity of the text is recorded as the second pre-stored text; The obtained multiple second pre-stored texts are recorded as a secondary output text list.

5. The AI-based big data text information retrieval method according to claim 1, characterized in that: Obtaining a sorted list of text to be output includes the following steps: The first output text list and the second output text list are merged to obtain an unsorted list of texts to be output, wherein the unsorted list of texts to be output includes a plurality of unsorted texts to be output; Obtaining pre-stored text key information of each unsorted text to be output stored in a database, comparing the text key information of the text to be retrieved with the pre-stored text key information of each unsorted text to be output, and obtaining a first comprehensive similarity corresponding to each unsorted text to be output; The obtained pre-stored TF-IDF vectors of the unsorted texts to be output are stored in the database, and the TF-IDF vectors of the texts to be retrieved are compared with the pre-stored TF-IDF vectors of the unsorted texts to be output, so as to obtain the second comprehensive similarity corresponding to the unsorted texts to be output; Comprehensively analyzing the first comprehensive similarities and the second comprehensive similarities corresponding to each unsorted to-be-output text to obtain each third comprehensive similarity; The third comprehensive similarities are sorted from large to small to obtain corresponding sorted texts to be output, and each sorted text to be output is output as a sorted text list to be output.

6. The AI-based big data text information retrieval method according to claim 5, characterized in that: The calculation formula of the third comprehensive similarity is: ； In the formula, is the bth third comprehensive similarity, is the first comprehensive similarity of the bth unsorted text to be output, is the second comprehensive similarity of the bth unsorted text to be output, Stored in the database The weight factor of Stored in the database The weight factor of is, and b is the number of each unsorted text to be output in the list of texts to be output.

7. The AI-based big data text information retrieval method according to claim 1, characterized in that: Adjusting and sorting each sorted text to be output in the sorted text list to be output based on the text information data set and then outputting the final text list includes the following steps: Obtaining historical search information data of each sorted text to be output, analyzing each historical search information data, and obtaining a search feature value of each text; Obtaining text background data of each sorted text to be output, analyzing each text background data, and obtaining each text background feature value; Combining the text retrieval feature value and the text background feature value corresponding to each sorted text to be output, to obtain the priority evaluation value corresponding to each sorted text to be output; The priority rating values ​​are arranged from large to small, the corresponding sorted texts to be output are sorted again, and the final text list is output.

8. The AI-based big data text information retrieval method according to claim 7, characterized in that: The calculation formula of text retrieval feature value is: ； In the formula, Retrieve the feature value for the mth sorted output text, is the number of historical query records for the mth sorted text to be output. is the access time of the mth sorted text to be output, is the click rate of the mth sorted text to be output, is the bounce rate of the mth sorted text to be output, where m is the number of the sorted text to be output.

9. The AI-based big data text information retrieval method according to claim 7, characterized in that: The calculation formula of the text background feature value is: ； In the formula, is the text background feature value of the mth sorted text to be output, is the number of citations of the mth sorted text to be output. is the number of text references of the mth sorted text to be output, is the text data update frequency of the mth sorted text to be output, and m is the number of the sorted text to be output.

10. The AI-based big data text information retrieval method according to claim 7, characterized in that: The priority rating value is calculated as follows: ； In the formula, is the priority rating value of the mth sorted text to be output, Retrieve the feature value for the mth sorted output text, is the text background feature value of the mth sorted text to be output, Stored in the database The weight factor of Stored in the database The weight factor of , e is a natural constant, and m is the number of the sorted text to be output.

Images