An information intelligent retrieval method and system based on big data processing

Abstract

The present application relates to the technical field of information retrieval, and discloses an information intelligent retrieval method and system based on big data processing, which acquires subject information and unstructured data, and performs semantic processing on the unstructured data according to the subject information to generate a structured information abstract. The information abstract is stored in a fast access area, thereby greatly improving the efficiency of subsequent queries. When a user submits a query intention, the system can quickly acquire query information from the fast access area and judge whether it meets the query requirement. If yes, an instruction for directional retrieval of original data is generated. The method effectively solves the problem that the traditional retrieval method is inefficient and difficult to meet the requirement of rapid decision-making when facing massive unstructured data and multi-source heterogeneous information in the prior art.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, and more specifically, to an intelligent information retrieval method and system based on big data processing. Background Technology

[0002] In enterprise-level data management, traditional retrieval methods are often inefficient and fail to meet the demands of rapid decision-making when faced with massive amounts of unstructured data and heterogeneous information from multiple sources. This is especially true when information sources become more diverse, containing numerous non-standardized expressions and out-of-domain terms. Existing systems encounter significant challenges in semantic understanding and query processing, leading to inaccurate search results, untimely information acquisition, and consequently impacting the enterprise's strategic judgment and market responsiveness.

[0003] Existing intelligent information retrieval systems perform well when processing standardized internal enterprise data, but their semantic understanding capabilities are significantly insufficient when faced with external market research data. This external data contains a large amount of internet slang, abbreviations, emerging industry terms, and non-standard grammatical structures, causing the system to be unable to accurately identify query intent. For example, it cannot understand the meaning of "Generation Z" or "metaverse" in a specific context, thus failing to translate user queries into precise search instructions.

[0004] Due to the limitations of semantic understanding, market researchers are forced to submit broad and vaguely intentd queries, such as "analyze competitors' recent marketing activities." Existing systems, in order to handle these queries, employ an "exhaustive" strategy, breaking them down into numerous sub-queries and performing cross-validation and matching across multiple heterogeneous data sources. This approach places enormous computational pressure on the server's CPU and memory, significantly increasing query parsing time and even causing timeouts, severely impacting the timeliness of information retrieval.

[0005] Furthermore, due to biases in the query parser's ability to decompose broad queries and understand the semantics of new data, coupled with performance degradation when processing large numbers of subqueries and association matches, the final generated search instruction set often contains errors or omissions. This directly results in a significant discrepancy between the documents located by the backend system in the inverted index structure and the user's actual needs, greatly reducing the accuracy of search results and failing to provide market researchers with comprehensive and accurate insights.

[0006] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0007] This invention provides an intelligent information retrieval method and system based on big data processing, aiming to solve the problems of low efficiency, insufficient semantic understanding, high query processing pressure, and inaccurate retrieval results of traditional retrieval methods in existing enterprise-level data management.

[0008] The technical solution of this application is as follows: Firstly, this application discloses an intelligent information retrieval method based on big data processing, including: Obtain topic information and unstructured data; Based on the topic information, semantic processing is performed on unstructured data to generate structured information summaries; Store structured information summaries in a fast-access area; Retrieve query information from the quick access area based on the user's submitted query intent; Determine whether the query information meets the query requirements; If so, then generate instructions to perform a targeted retrieval of the original data.

[0009] This technical solution enables the effective processing of massive amounts of unstructured data. By generating structured information summaries through semantic processing and storing them in a fast access area, it significantly improves the efficiency and accuracy of information retrieval, solving the problems of low efficiency and inability to meet the needs of rapid decision-making in traditional retrieval methods.

[0010] Furthermore, based on the above, semantic processing is performed on external unstructured data according to thematic information to generate structured information summaries, including: Identify novel representations in external unstructured data; Analyze the contextual information of the new expression to infer its meaning and obtain the inferred meaning; Obtain feedback information on the semantic transformation results of the new expression; Update the semantic transformation rules based on the inferred meaning and feedback information; Based on the updated semantic transformation rules, semantic processing is performed on external unstructured data to generate structured information summaries.

[0011] Through this technical solution, this application can dynamically identify and learn the meaning of new expressions, and continuously optimize semantic transformation rules based on feedback information, thereby significantly improving the system's ability to understand non-standardized expressions and out-of-domain terms in external market research data, and overcoming the challenges of existing systems in semantic understanding.

[0012] Furthermore, based on the above, query information is retrieved from the fast access area according to the user-submitted query, including: The key expressions that are traced are obtained by tracking the key expressions that appear in the query and the key expressions contained in the information summary in the fast access area; When a change in the meaning of a key expression being tracked is detected, a contextual analysis process is initiated, which includes: Extract contextual information of the key expressions from the original text fragments in which they appear; By combining contextual information and the frequency of use and co-occurring words of the tracked key expressions in different time periods and community contexts, the meaning of the tracked key expressions in the current context is inferred, and the inferred meaning is obtained. By combining historical feedback from the tracked key expressions, the inferred meaning is calibrated to obtain the calibrated meaning of the key expressions; Using the calibrated key expression meanings, the semantic matching degree between the query intent and the information summary is calculated; When the semantic matching degree reaches a preset threshold, a message summary is obtained.

[0013] Through this technical solution, this application can dynamically track the semantic changes of key expressions and initiate a contextualized analysis process. By combining multi-dimensional information, it can infer and calibrate the meaning of key expressions, thereby significantly improving the accuracy of understanding query intent and the accuracy of semantic matching, and avoiding the problem of broad queries caused by the limitations of semantic understanding.

[0014] In some preferred embodiments, when a change in the meaning of a key expression being tracked is detected, a contextual analysis process is initiated, including: Prioritize the monitoring of semantic changes for each key expression being tracked; Adjust the allocation of computational resources for the semantic change recognition task based on the priority of semantic change monitoring; Analyze the semantic patterns of high-priority key expressions and compare them with historical semantic patterns; When the comparison results show that the semantic pattern of high-priority key expressions deviates significantly from the historical semantic pattern, the contextual analysis process is initiated.

[0015] Through this technical solution, this application can intelligently allocate computing resources based on the priority of semantic change monitoring, and prioritize the analysis of semantic pattern changes of high-priority key expressions, thereby more efficiently and accurately identifying the meaning changes of key expressions, avoiding unnecessary computing overhead, and improving system response speed.

[0016] As a further improvement, the semantic patterns of high-priority key expressions are analyzed and compared with historical semantic patterns, including: Contextualize the text in the latest data stream that contains high-priority key expressions; For each specific context, we extract the co-occurrence word distribution, sentiment distribution, and topic distribution of high-priority key expressions to obtain the semantic patterns of high-priority key expressions in specific contexts. The semantic patterns in each specific context are compared with the historical semantic patterns of high-priority key expressions; If a significant deviation exists, analyze whether the deviation is a semantic drift in the subdivided context; Update the overall semantic pattern of high-priority key expressions based on semantic drift in the subdivided context.

[0017] Through this technical solution, this application is able to perform detailed contextual segmentation and semantic pattern analysis of high-priority key expressions, and identify semantic drift, thereby more accurately understanding the meaning changes of key expressions in different contexts, ensuring the comprehensiveness and dynamism of semantic understanding.

[0018] To improve the solution, if significant deviations exist, the analysis will determine whether the deviations are semantic drifts within specific contexts, including: Obtain the associated expressions of key expressions that show significant deviations from historical patterns in the semantic drift of the subdivided context; By analyzing the co-occurrence frequency, semantic distance, and mention status of related expressions in historical data, a potential impact range score of semantic drift is obtained. Downstream analysis tasks related to identifying key representations associated with semantic drift; Simulate the data processing results before and after semantic drift to obtain the differences in data processing results; Based on the differences in data processing results, suggestions are provided for adjusting downstream analysis task parameters or data sources; Based on the potential impact score, the priority of subsequent information processing procedures will be adjusted.

[0019] Through this technical solution, this application can deeply analyze the potential impact of semantic drift and provide targeted adjustment suggestions, thereby effectively avoiding the negative impact of semantic drift on downstream analysis tasks and ensuring the accuracy and reliability of data processing results.

[0020] Based on the above, and considering the differences in data processing results, suggestions are provided for adjusting downstream analysis task parameters or data sources, including: Identify data features or semantic elements that lead to differences in data processing results; Based on a pre-defined adjustment strategy library, a set of candidate adjustment schemes is generated, which includes parameter adjustment ranges, data source switching options, or model retraining suggestions. Record the adoption results of market researchers' recommendations for candidate adjustments; Based on the adoption results, adjust the priority and recommendation logic of the solutions in the strategy library.

[0021] Through this technical solution, this application can intelligently generate and optimize adjustment plans based on differences in data processing results, and continuously improve the adjustment strategy library based on the adoption results of market researchers, thereby realizing the intelligence and personalization of adjustment plans, and further improving the system's adaptability and user satisfaction.

[0022] As a technological improvement, the method also includes: The expected impact of each candidate adjustment scheme on the key indicators of downstream analysis tasks is evaluated, as well as the computational resources and time costs required for each candidate adjustment scheme, to obtain the evaluation results.

[0023] Through this technical solution, this application can comprehensively evaluate candidate adjustment schemes, taking into account their impact on key indicators and resource costs, thereby providing market researchers with a more scientific and comprehensive basis for decision-making and avoiding blind adjustments.

[0024] Furthermore, the expected impact of each candidate adjustment scheme on key indicators of downstream analysis tasks is evaluated, along with the computational resources and time costs required for each candidate adjustment scheme, yielding evaluation results, including: Obtain the identity information of market researchers and the identifier of current research projects; Based on identity information, the preference weights of market researchers for different types of adjustment schemes are obtained from historical interaction records. These preference weights include the relative importance attached to expected impact, computing resources, and time costs. Based on the research project identifier, retrieve the priority settings for the research project's expected impact, computational resources, and time costs from the project configuration; By combining preference weights and priority settings, a weighted evaluation criterion is obtained; Using a weighted evaluation criterion, the expected impact, required computational resources, and time cost of each candidate adjustment plan are weighted and calculated to obtain a comprehensive evaluation score; Based on the comprehensive evaluation scores, the candidate adjustment plans are ranked and screened to generate evaluation results that adapt to the decision-making preferences and project priorities of market researchers.

[0025] Through this technical solution, this application can combine the preferences of market researchers and project priorities to conduct personalized evaluation and ranking of candidate adjustment solutions, thereby generating evaluation results that better meet user needs and project goals, and significantly improving the accuracy and effectiveness of decision support.

[0026] Secondly, this application also discloses an intelligent information retrieval system based on big data processing, comprising: The input end is used to obtain topic information and unstructured data; The processing unit performs semantic processing on unstructured data based on topic information to generate a structured information digest; stores the structured information digest in a fast access area; and retrieves query information from the fast access area based on the user's submitted query intent. The output end is used to determine whether the query information meets the query requirements; if so, it generates instructions to perform targeted retrieval of the original data.

[0027] This application provides a system that can effectively process big data and perform intelligent retrieval through this technical solution. Through modular design, it realizes the automation of information acquisition, processing and output, and solves the efficiency and accuracy problems of traditional systems when processing massive amounts of unstructured data. Beneficial effects

[0028] This application discloses an intelligent information retrieval method and system based on big data processing. It acquires topic information and unstructured data, performs semantic processing on the unstructured data according to the topic information, and generates a structured information summary. This information summary is stored in a fast access area, thereby greatly improving the efficiency of subsequent queries. When a user submits a query intent, the system can quickly retrieve query information from the fast access area and determine whether it meets the query requirements. If it does, it generates an instruction to perform a targeted retrieval of the original data. This method effectively solves the problems of low efficiency and inability to meet the needs of rapid decision-making in traditional retrieval methods when facing massive amounts of unstructured data and multi-source heterogeneous information. By introducing semantic processing and a fast access area, this application significantly improves the efficiency and accuracy of information retrieval, avoiding the huge computational pressure and significantly increased query parsing time caused by traditional "exhaustive" retrieval strategies. This provides market researchers with timely and accurate information acquisition capabilities, overcoming the challenges of semantic understanding and query processing in existing systems. Attached Figure Description

[0029] Figure 1 This is a flowchart of an intelligent information retrieval method based on big data processing provided by an embodiment of the present invention; Figure 2 This is a flowchart of a method for semantic processing of external unstructured data provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an intelligent information retrieval system based on big data processing provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Reference Figure 1 , Figure 1This is a flowchart of an intelligent information retrieval method based on big data processing provided by an embodiment of the present invention, including: S11, obtain topic information and unstructured data; S12, based on the topic information, perform semantic processing on the unstructured data to generate a structured information summary; S13, store the structured information summary in a fast access area; S14, retrieve query information from the fast access area based on the query intent submitted by the user; S15, determine whether the query information meets the query requirements; S16, if so, then generate an instruction to perform a targeted retrieval of the original data.

[0032] This application introduces a semantic processing mechanism for unstructured data and combines it with storage and query optimization for fast access areas, effectively improving the accuracy and efficiency of information retrieval. This breaks through the limitations of existing technologies in semantic understanding and query processing, providing enterprises with more accurate and timely information support.

[0033] To make the technical solution of this application easier and clearer to understand, some key terms involved will be explained first.

[0034] "Topic information" refers to predefined or user-specified information categories, fields, or areas of interest, such as market trends, competitor analysis, and product reviews. Its purpose is to provide a clear context and direction for subsequent semantic processing, ensuring the system focuses on the data most relevant to the user's needs.

[0035] Unstructured data refers to data that does not have a predefined data model or fixed format, such as text files, web page content, social media posts, emails, and reports. This type of data usually contains rich semantic information, but it is difficult to query and analyze directly through traditional databases.

[0036] "Semantic processing" refers to the deep understanding and analysis of text data through technologies such as natural language processing (NLP) and machine learning. This includes lexical analysis, syntactic analysis, entity recognition, sentiment analysis, and topic modeling. The aim is to extract meaningful information from unstructured data and transform it into a structured form that machines can understand.

[0037] "Structured information summaries" refer to key information extracted from unstructured data after semantic processing and represented in a structured form (such as key-value pairs, tables, graphs, etc.). These summaries typically contain core content such as topics, entities, relationships, and viewpoints, facilitating rapid storage, indexing, and retrieval.

[0038] A "fast access zone" refers to an optimized data storage layer characterized by high read / write speeds and short response times, such as in-memory databases, caching systems, or high-performance distributed file systems. Its function is to store structured information summaries to support efficient retrieval of query information.

[0039] "Query intent" refers to the true information needs or goals implied by a user when submitting a query. Accurately understanding query intent is key to achieving intelligent retrieval, as it determines how the system parses the query, matches information, and generates search instructions.

[0040] "Query information" refers to a structured information summary obtained from the quick access area that is relevant to the user's query intent.

[0041] "Query requirements" refers to users' expectations for search results, including the relevance, completeness, and timeliness of the information.

[0042] "Raw data" refers to unstructured data that has not been processed in any way and is initially acquired.

[0043] This application provides an intelligent information retrieval method based on big data processing.

[0044] There are various methods for acquiring topic information and unstructured data. For example, topic information can be manually entered through a user interface or automatically loaded through a preset configuration file. Unstructured data can be obtained from various data sources, such as web crawlers scraping web content from the internet, API interfaces retrieving posts from social media platforms, or documents reading from an enterprise's internal file server. As a preferred implementation, a data acquisition module can be configured to periodically scan specified data sources, automatically identify and acquire newly added unstructured data, and automatically associate it with corresponding topic information according to preset classification rules.

[0045] To generate structured information summaries from unstructured data through semantic processing based on topic information, the following approaches can be adopted. For example, Natural Language Processing (NLP) toolkits, such as NLTK or SpaCy, can be used to perform operations such as word segmentation, part-of-speech tagging, and named entity recognition on the unstructured data. Subsequently, combined with topic information, core concepts and key facts are extracted from the processed text through keyword extraction, topic modeling (such as LDA), or text summarization techniques, and organized into a structured form, such as a JSON object or an XML document. As a preferred implementation, a deep learning-based semantic understanding model can be constructed. This model can perform end-to-end semantic analysis of unstructured data based on topic information, automatically identifying and extracting entities, events, and relationships highly relevant to the topic, and transforming them into standardized structured information summaries.

[0046] The following methods can be used to store structured message digests in a fast-access area. For example, the generated structured message digests can be written directly to an in-memory database, such as Redis or Memcached, to achieve extremely low read / write latency. Alternatively, they can be stored in a distributed caching system, such as Apache Ignite, to support large-scale concurrent access.

[0047] To determine whether query information meets the query requirements, the following methods can be used. For example, a simple keyword matching threshold can be set; if the obtained query information contains all the keywords in the user's query, it is considered to meet the query requirements. Alternatively, the text similarity between the query information and the query intent can be calculated; when the similarity exceeds a preset threshold, it is determined that the requirements are met. As a preferred implementation, a machine learning-based matching model can be constructed. This model can comprehensively consider multiple dimensions such as the semantic relevance, timeliness, and source reliability of the query information, quantitatively evaluate the degree of matching between the query information and the query requirements, and determine whether the requirements are met based on the evaluation results.

[0048] In generating instructions for targeted retrieval of raw data, the following methods can be used. For example, if the query information meets the query requirements, the system can generate an instruction pointing to the raw data storage location based on the raw data identifier or storage path contained in the query information. This instruction can be a file path, a database query statement, or an API call. As a preferred implementation, an instruction set containing detailed retrieval parameters can be generated, such as specifying the file type, time range, and specific fields of the raw data, to ensure that the backend retrieval system can accurately locate and extract the raw data fragments required by the user.

[0049] This application presents an intelligent information retrieval method based on big data processing. By transforming unstructured data into structured information summaries and storing them in a fast-access area, it significantly improves the efficiency and accuracy of information retrieval. When a user submits a query, the system can quickly retrieve the query information from the preprocessed summary and intelligently determine whether it meets the query requirements. Once met, the system can generate precise targeted search instructions that directly point to the original data, avoiding blind scanning of massive amounts of raw data.

[0050] In some embodiments described above, this application proposes semantic processing of unstructured data based on topic information to generate structured information summaries. However, in practical applications, unstructured data often contains a large number of constantly evolving new expressions, industry terms, or context-specific vocabulary. If the semantic processing fails to effectively identify and adapt to the meanings of these new expressions, the accuracy and completeness of the information summary may decrease, thereby affecting the subsequent intelligent retrieval performance. Therefore, this application further proposes a method to optimize the aforementioned semantic processing, aiming to improve the ability to identify and process new expressions in unstructured data, thereby generating more accurate and timely structured information summaries.

[0051] In this regard, refer to Figure 2 , Figure 2 This is a flowchart of a method for semantic processing of external unstructured data provided in an embodiment of the present invention, S12: S121, Identify new expressions in the external unstructured data; S122, Analyze the contextual information of the new expression to infer the meaning of the new expression and obtain the inferred meaning; S123, obtain feedback information on the semantic conversion result of the new expression, and obtain feedback information; S124, Update the semantic conversion rules based on the inferred meaning and the feedback information; S125, according to the updated semantic transformation rules, perform semantic processing on the external unstructured data to generate a structured information digest.

[0052] Specifically, identifying new expressions in external unstructured data refers to automatically detecting and labeling new words, phrases, or expressions that are not included in existing dictionaries or knowledge bases, or whose semantics have changed significantly in a specific context, using natural language processing (NLP) techniques such as lexical analysis, named entity recognition, or new word discovery algorithms based on statistical models. The aim is to capture the dynamic changes in language and ensure the comprehensiveness of semantic processing.

[0053] This process involves analyzing the contextual information of new expressions to infer their meaning. This inferred meaning can be understood as follows: after recognizing a new expression, the system extracts contextual information such as the preceding and following words, syntactic structure, and paragraph topic from the text in which the new expression appears. Using deep learning models or methods based on co-occurrence statistics, this contextual information is analyzed to preliminarily infer the possible meaning of the new expression in the current context. The aim is to provide a preliminary semantic explanation for the new expression.

[0054] In practical applications, obtaining feedback information on the semantic transformation results of new expressions involves collecting user evaluations, corrections, or annotations of these summaries after the system performs semantic transformation on the new expressions based on the inferred meaning and generates information summaries. Alternatively, feedback on the accuracy of the semantic transformation can be obtained through other automated evaluation mechanisms (such as comparison with known high-quality summaries). The purpose is to introduce an external verification mechanism to improve the reliability of semantic processing.

[0055] Furthermore, updating semantic conversion rules based on the inferred meaning and feedback information means that the system comprehensively analyzes the initially inferred new meaning and the collected feedback information. If the feedback information confirms or corrects the inferred meaning, the system will adjust or add semantic conversion rules accordingly, such as updating the dictionary, adjusting semantic mapping relationships, or optimizing semantic model parameters. The aim is to continuously learn and adapt to language changes, improving the accuracy and robustness of semantic processing.

[0056] Therefore, semantic processing of external unstructured data to generate structured information summaries based on updated semantic transformation rules means that after the semantic transformation rules are updated, the system will use these latest rules to reprocess or continue processing the unstructured data. This ensures that the subsequently generated information summaries can reflect the latest language understanding and semantic mapping, thereby improving the accuracy and timeliness of the information summaries.

[0057] This application's solution effectively addresses the limitations of traditional semantic processing methods in the face of dynamic language evolution by introducing a closed-loop mechanism for identifying new expressions in unstructured data, inferring meaning, obtaining feedback, and updating rules. Through this technical solution, the application significantly improves the ability to identify and process new expressions in unstructured data, effectively solving the problem of inaccurate or lagging semantic understanding in traditional methods when dealing with dynamic language environments. By continuously learning and adaptively updating semantic transformation rules, the system can adapt to language changes in a timely manner, ensuring that the generated information summaries are not only structured but also semantically accurate and complete. This not only improves the efficiency and accuracy of intelligent information retrieval but also enhances the system's robustness and adaptability in processing massive amounts of real-time unstructured data, providing users with more valuable and timely information services.

[0058] In some preferred embodiments, suppose market researchers need to intelligently retrieve user feedback on an emerging technology product (such as a "quantum computing phone") from massive amounts of social media data. Traditional methods may fail to accurately identify the semantics of "quantum computing phone" because it is a novel expression, leading to the omission of relevant information. The solution in this application is implemented as follows: First, after acquiring social media data, the system uses lexical analysis and new word discovery algorithms to identify the new expression "quantum computing phone".

[0059] Next, the system analyzes the context in which "quantum computing phone" appears in the text, such as "the battery life of this quantum computing phone" and "the processor performance of the quantum computing phone," and infers that it means a new type of smartphone that incorporates quantum computing technology.

[0060] Subsequently, the system submits the initially generated information summary containing "quantum computing phone" to market researchers for review and collects their feedback on the summary's accuracy. If researchers correct any inaccurate semantic transformations, these corrections are recorded as feedback by the system. Based on the inferred meaning and the researchers' feedback, the system updates its internal semantic transformation rules, such as adding "quantum computing phone" and its related attributes to the dictionary and establishing semantic associations with concepts like "smartphone" and "quantum technology."

[0061] Ultimately, the system utilizes updated semantic transformation rules to perform semantic processing on subsequent incoming social media data. This allows it to accurately summarize all information mentioning "quantum computing phones" and related discussions and store it in a quick access area, enabling market researchers to efficiently retrieve comprehensive and accurate user feedback on this emerging product.

[0062] Traditional intelligent information retrieval methods often rely on static keyword matching or pre-defined semantic models to retrieve query information from quick access areas based on user-submitted query intent. However, in a dynamic environment of rapid information iteration and constantly evolving language, the meaning of key expressions may drift over time, depending on community context, or specific situation. If this problem is not addressed, the system may fail to accurately understand user intent when the semantics of words in the user's query or the content of the information summary change, leading to reduced relevance of the retrieved information and impacting retrieval efficiency and user experience.

[0063] To address this, this application proposes a more refined method for obtaining query information. This method aims to more accurately understand user query intent and improve the precision of information retrieval by dynamically tracking semantic changes in key expressions and performing contextual analysis. The aforementioned method of obtaining query information from the quick access area based on the user-submitted query includes: By tracking the key expressions appearing in the query and the key expressions contained in the information summary in the fast access area, the tracked key expressions are obtained; When a change in the meaning of the tracked key expression is detected, a contextual analysis process is initiated, which includes: Extract the contextual information of the key expressions being tracked from the original text fragments in which they appear; By combining the contextual information and the frequency of use and co-occurring words of the tracked key expressions in different time periods and community contexts, the meaning of the tracked key expressions in the current context is inferred, and the inferred meaning is obtained. By combining the historical feedback of the tracked key expressions, the inferred meaning is calibrated to obtain the calibrated meaning of the key expressions; Using the calibrated key expression meanings, calculate the semantic matching degree between the query intent and the information summary; When the semantic matching degree reaches a preset threshold, the information digest is obtained.

[0064] Specifically, tracking key expressions refers to the system continuously monitoring words or phrases appearing in user queries, as well as the core vocabulary or concepts contained in the information summaries stored in the quick access area. This tracking can be based on word frequency, TF-IDF values, or pre-trained word embedding models to identify expressions with high informational value. The goal is to build a dynamic list of key expressions for subsequent monitoring of their semantic changes.

[0065] When the system detects a change in the meaning of a key expression being tracked—for example, if the usage context of a word in recent data streams differs significantly from its historical context, or if its co-occurring vocabulary has shifted noticeably—it initiates a contextual analysis process. This process aims to gain a deeper understanding of the true meaning of key expressions within a specific context.

[0066] Extracting contextual information from the original text fragments where the tracked key expressions appear means that the system traces back to the complete text content where the key expression first or most recently appeared, and extracts the sentences, paragraphs, or adjacent word sequences containing the key expression. This contextual information is the basis for understanding the current meaning of the key expression.

[0067] Furthermore, by combining contextual information and the frequency of use and co-occurring words of the tracked key expressions in different time periods and community contexts, the meaning of the tracked key expressions in the current context can be inferred. Specifically, the system can utilize natural language processing techniques, such as word vector models (Word2Vec, GloVe, BERT, etc.), topic models (LDA), or deep learning models, to analyze the occurrence patterns of key expressions in specific time periods (e.g., the past 24 hours, the past week) and specific communities (e.g., a forum, a social media group). By comparing these patterns with historical patterns, the new or evolved meanings of the key expressions in the current context can be inferred. For example, a word might refer to a financial product in a financial community, while in a technology community it might refer to a technological concept.

[0068] Furthermore, the inferred meaning is calibrated by incorporating historical feedback from tracked key expressions. Historical feedback can include user satisfaction with previous search results, explicit user annotations of the meanings of specific words, or expert suggestions for adjusting the semantic model. This feedback is used to correct or validate the meanings automatically inferred by the system, thereby improving the accuracy and reliability of meaning inference.

[0069] Therefore, using the calibrated key expression meanings, the semantic matching degree between the query intent and the information summary is calculated. The semantic matching degree can be calculated using various methods, such as cosine similarity based on word embeddings, semantic similarity models based on deep learning, or reasoning combined with ontology knowledge graphs. The aim is to quantify the relevance of the user's true query intent to the information summary content in the quick access area.

[0070] When the semantic matching degree reaches a preset threshold, for example, a similarity score exceeding 0.7, the information summary is considered highly relevant to the user's query, and thus the information summary is obtained. The preset threshold can be adjusted according to the actual application scenario and the requirements for retrieval accuracy.

[0071] This application's solution effectively addresses the limitations of traditional methods in handling dynamic semantic changes by introducing dynamic semantic tracking and contextual analysis mechanisms. Through these technical solutions, this application significantly improves the accuracy and user satisfaction of intelligent information retrieval. Compared to traditional methods that rely solely on static semantic matching, this application, by dynamically tracking the semantic changes of key expressions and performing contextual analysis, can more accurately capture subtle changes and evolutions in user query intent. Especially when dealing with semantic drift in emerging vocabulary, internet slang, or domain-specific terminology, this solution can promptly adjust its understanding of these expressions, thereby avoiding low-quality search results due to semantic misunderstandings. Furthermore, by combining historical feedback to calibrate semantic inference, the system's adaptive learning ability and the reliability of semantic understanding are further enhanced. This allows the system to provide highly relevant query information even in complex and ever-changing linguistic environments, effectively solving the problem of insufficient retrieval accuracy in dynamic semantic environments using traditional methods.

[0072] In some preferred embodiments, suppose a user submits a query for "Apple product launch". The system first tracks the key expressions "Apple" and "product launch". If, at some point, the word "Apple" suddenly appears frequently alongside "Vision Pro" in tech news and social media, and the contextual information (e.g., news headlines, discussion posts) clearly points to a new mixed reality device rather than a traditional phone or computer, the system recognizes that the meaning of the key expression "Apple" may have changed from referring generally to "Apple Inc." or its traditional products to specifically referring to its new mixed reality product line.

[0073] At this point, the contextual analysis process is initiated. The system extracts context from recent text snippets about "Apple," for example, extracting "Apple unveiled its first spatial computing device, Vision Pro, at WWDC" from a tech news report. Combining this contextual information with the high frequency of "Apple" appearing in tech communities over the past month alongside terms like "Vision Pro," "MR," and "spatial computing," the system infers that in the current context, "Apple" more likely refers to its mixed reality products. If historical feedback shows that users' queries related to "Apple" are more likely to seek the latest and most cutting-edge product information, the system will further refine this inference.

[0074] Subsequently, the system uses the calibrated meaning of "Apple" (i.e., "Apple's mixed reality products") and the meaning of "new product launch" to calculate the semantic match between the user's query "Apple new product launch" and various information summaries stored in the quick access area (e.g., "iPhone 15 launch recap," "Vision Pro technology analysis," "MacBook Air M3 launch"). If the information summary of "Vision Pro technology analysis" matches the calibrated query intent to the highest degree and reaches a preset threshold, the system will retrieve that information summary and generate instructions for targeted retrieval of the original data, thereby providing the user with more accurate search results that better fit the current semantic context.

[0075] In some embodiments described above, a contextual analysis process is initiated when a change in the meaning of a tracked key expression is detected. However, in practical applications, simply identifying a meaning change and immediately initiating contextual analysis may lead to excessive consumption of computational resources, especially when dealing with massive amounts of data and numerous key expressions. Not all meaning changes are of equal importance or urgency; processing them indiscriminately may result in system inefficiency and potentially delay a timely response to key semantic shifts.

[0076] In this regard, this application further proposes the following steps for initiating the contextual analysis process when the meaning of the key expressions being tracked changes: Prioritize the monitoring of semantic changes for each key expression being tracked; Adjust the allocation of computational resources for the semantic change recognition task based on the priority of semantic change monitoring; Analyze the semantic patterns of high-priority key expressions and compare them with historical semantic patterns; When the comparison results show that the semantic pattern of high-priority key expressions deviates significantly from the historical semantic pattern, the contextual analysis process is initiated.

[0077] Specifically, prioritizing the semantic changes of each tracked key expression refers to assigning an importance or sensitivity level to each tracked key expression. This priority can be determined based on various factors, such as the importance of the key expression in a specific domain, the frequency of its meaning changes in historical data, its potential impact on downstream analysis tasks, or a level of concern preset by the user or system administrator. For example, key expressions related to core products, major competitors, or high-risk events can be given higher priority.

[0078] The adjustment of computational resource allocation for the meaning change recognition task based on semantic change monitoring priority can be understood as the system dynamically allocating computational resources for monitoring meaning changes based on the priority of key expressions. For example, for high-priority key expressions, more processor time, memory, or more frequent monitoring cycles can be allocated to ensure that their meaning changes are captured promptly and meticulously. Conversely, for low-priority key expressions, less resources or a lower monitoring frequency can be used, thereby optimizing the overall system's resource utilization.

[0079] In practical applications, analyzing the semantic patterns of high-priority key expressions and comparing them with historical semantic patterns means that the system will conduct in-depth analysis of the semantic performance of those high-priority key expressions in the latest data stream. Semantic patterns can include, but are not limited to, co-occurring word distribution, sentiment tendency, and topic distribution. These latest semantic patterns are compared with the historical semantic patterns of the key expression periodically or in real time. Historical semantic patterns can be stored as average patterns, trend patterns, or a series of discrete snapshots over a period of time.

[0080] When the comparison results show a significant deviation between the semantic patterns of high-priority key expressions and historical semantic patterns, a contextual analysis process is initiated. A significant deviation refers to a change in semantic pattern that exceeds a preset statistical threshold or an expert-defined semantic drift standard. For example, if the sentiment of a key expression changes from positive to negative, or if its main co-occurring words undergo a fundamental change, this might be considered a significant deviation. Only when such a significant and meaningful deviation is detected will a more in-depth and resource-intensive contextual analysis process be triggered to avoid unnecessary processing of unimportant semantic fluctuations.

[0081] This application's solution effectively addresses the resource waste and response delays that may exist in the aforementioned basic solutions by introducing semantic change monitoring priorities. Through this technical solution, this application achieves intelligent and efficient monitoring of semantic changes in key expressions. Compared to indiscriminately identifying meaning changes and initiating contextual analysis, this application significantly reduces unnecessary computational overhead and optimizes system resource utilization by introducing priority management and dynamic resource allocation. Furthermore, by focusing on semantic pattern analysis of high-priority key expressions and only initiating contextual analysis when significant deviations are detected, this application can capture truly important semantic drifts more promptly and accurately, thereby ensuring a more accurate and efficient understanding of user query intent and acquisition of information summaries, avoiding misjudgments or delays caused by minor semantic fluctuations.

[0082] In some preferred embodiments, suppose a market research team is monitoring discussions on social media about multiple products and competitors. The team sets "Product A" and "Competitor X" as high-priority key expressions because they have the greatest impact on market strategy; while "Product B," "Product C," and "Competitor Y" are set as medium- to low-priority.

[0083] The system will allocate more computing resources to the semantic change recognition task for "Product A" and "Competitor X," for example, performing semantic pattern analysis once per hour and using a more complex natural language processing model. For "Product B," "Product C," and "Competitor Y," the system may perform analysis once every four hours and use a relatively lightweight model.

[0084] Specifically, the system continuously analyzes the distribution of co-occurring words, sentiment, and themes related to "Product A" in the latest social media data stream, comparing it with the average semantic pattern over the past six months. One day, the system detected a sudden surge of negative words related to "recall" and "defect" in the co-occurring words of "Product A," with the overall sentiment shifting significantly from "positive" to "negative," which was deemed a significant deviation from historical semantic patterns. At this point, the system immediately initiates a contextualized analysis process to delve into the source, content, and scope of these negative discussions to infer the true meaning of "Product A" in the current context and provide an early warning to the market research team.

[0085] At the same time, if the semantic pattern of "Product C" also fluctuates slightly but does not reach the threshold of significant deviation, the system will not immediately initiate contextual analysis, but will continue to monitor at a lower frequency, thereby avoiding unnecessary resource consumption and ensuring priority processing of the most critical information.

[0086] In some embodiments described above in this application, a method is proposed for analyzing the semantic patterns of high-priority key expressions and comparing them with historical semantic patterns. Specifically, the steps of analyzing the semantic patterns of high-priority key expressions and comparing them with historical semantic patterns include: Contextual segmentation is performed on the text in the latest data stream where the high-priority key expressions appear; For each sub-context, the co-occurrence word distribution, sentiment distribution, and topic distribution of the high-priority key expressions are extracted to obtain the semantic pattern of the high-priority key expressions in the specific sub-context. The semantic patterns in each sub-context are compared with the historical semantic patterns of the high-priority key expressions; If a significant deviation exists, analyze whether the deviation is a semantic drift of the subdivided context; The overall semantic pattern of the high-priority key expressions is updated based on the semantic drift of the subdivided context.

[0087] Specifically, context segmentation refers to dividing high-priority key expressions in the latest data stream into multiple independent sub-contexts with specific semantic environments, based on features such as context, topic, or source. For example, a key expression may appear in a news report or in a social media discussion; these different scenarios constitute different contexts.

[0088] Specifically, for each sub-context, the co-occurrence word distribution, sentiment distribution, and topic distribution of high-priority key expressions are extracted to construct the semantic pattern of the key expression in that specific context. The co-occurrence word distribution reflects the frequency and relevance of other words appearing alongside the key expression; the sentiment distribution reveals the positive, negative, or neutral sentiment carried by the key expression in that context; and the topic distribution indicates the core issues addressed by the key expression in that context. Through these multi-dimensional features, the semantic characteristics of the key expression in a specific context can be comprehensively and meticulously depicted.

[0089] In practical applications, the semantic patterns of each sub-context are compared with the historical semantic patterns of high-priority key expressions. The purpose is to identify the semantic evolution or changes of key expressions in different contexts. The comparison can employ various statistical or machine learning methods, such as calculating semantic distance, similarity, or difference.

[0090] Furthermore, if the comparison results show significant deviations, it is necessary to analyze whether these deviations are semantic drift due to sub-contextual variations. Semantic drift refers to the evolution of the meaning of a word or expression over time, in context, or within a community. Identifying semantic drift helps distinguish between temporary contextual changes in key expressions and substantial semantic evolution.

[0091] Therefore, the overall semantic pattern of high-priority key expressions is updated based on semantic drift in specific contexts. This means that when semantic drift of a key expression is detected in a specific context, the drift information will be integrated into the overall semantic model of the key expression to ensure the accuracy and timeliness of its semantic representation.

[0092] This application's solution performs fine-grained contextual segmentation of high-priority key expressions in the latest data stream and extracts multi-dimensional semantic features for each sub-context, thereby capturing the semantic performance of key expressions in different contexts more accurately and comprehensively. Through this technical solution, this application achieves refined analysis and dynamic updating of the semantic patterns of high-priority key expressions. Compared to simply comparing overall semantic patterns, this solution significantly improves the accuracy and robustness of semantic change recognition by introducing contextual segmentation and semantic drift analysis. Specifically, by constructing and comparing semantic patterns at the sub-contextual level, the meaning evolution of key expressions in specific domains or contexts can be captured earlier and more accurately, avoiding subtle but important semantic changes that might be overlooked due to macro-level comparisons. Furthermore, the semantic drift recognition and update mechanism enables the system to continuously learn and adapt to dynamic language changes, thus ensuring the real-time nature and effectiveness of information retrieval and providing users with more accurate and context-appropriate query results.

[0093] In some of the embodiments described above in this application, although it is possible to identify significant deviations between the semantic patterns of high-priority key expressions in specific sub-contexts and historical patterns, and to analyze whether semantic drift exists and thus update the overall semantic pattern, simply identifying and updating the semantic pattern may not be sufficient to fully address the challenges posed by semantic drift. In complex and dynamically changing big data environments, semantic drift not only affects the understanding of individual key expressions, but may also have a chain reaction on related concepts and downstream analysis tasks that depend on these expressions. If timely assessment and measures are not taken, it may lead to a decrease in the accuracy of subsequent information processing results, or even cause erroneous decisions.

[0094] In response, this application further proposes a more refined semantic drift analysis method, which aims to deeply assess the potential impact of semantic drift and provide targeted adjustment suggestions to ensure that the intelligent information retrieval system can remain efficient and accurate in a semantically dynamic environment.

[0095] If significant deviations exist, the analysis will determine whether the deviations represent semantic drift within the subdivided context, including: Obtain the associated expressions of key expressions that show significant deviations between semantic drift and historical patterns in the subdivided context; By analyzing the co-occurrence frequency, semantic distance, and mention status of the associated expressions in historical data, a potential impact range score for the semantic drift is obtained. Identify downstream analysis tasks related to key representations of the semantic shift; Simulate the data processing results before and after the semantic shift to obtain the differences in data processing results; Based on the differences in the data processing results, suggestions are provided for adjusting the parameters of the downstream analysis task or the data source; The priority of subsequent information processing procedures will be adjusted based on the potential impact range score.

[0096] Specifically, acquiring the associated expressions of key expressions whose semantic drift in the subdivided context significantly deviates from historical patterns means that after identifying a semantic drift in a key expression, the system further explores other expressions that are semantically closely related to that key expression. These associated expressions can be words that frequently co-occur with the key expression, its synonyms, near-synonyms, or concepts that have a specific relationship with it in a specific knowledge graph. The purpose is to comprehensively understand the scope of semantic drift, rather than being limited to the drift itself.

[0097] The analysis of the co-occurrence frequency, semantic distance, and mention status of the associated expressions in historical data, along with their relevance in the current data stream, yields a potential impact score for semantic drift. This can be understood as assessing the breadth and depth of the potential impact of semantic drift on the entire information ecosystem by quantitatively analyzing the relationship between these associated expressions and the key drifting expressions. Co-occurrence frequency reflects the statistical strength of the association between words; semantic distance measures the similarity of words in the semantic space; and mention status in the current data stream reflects the activity of these associated expressions in the latest context. By combining these indicators, a potential impact score can be calculated. The higher the score, the wider and more profound the potential impact of semantic drift.

[0098] In practical applications, identifying downstream analysis tasks related to the key expressions involved in the semantic shift specifically means that the system determines, based on predefined task dependencies or by analyzing task metadata, which subsequent data analysis or processing flows will directly or indirectly use the key expressions that have experienced semantic shift. For example, if the semantics of the word "cloud" shifts from "weather phenomenon" to "cloud computing," then all tasks that rely on "cloud" for weather forecast analysis and IT industry trend analysis need to be identified. The purpose is to clarify the affected business processes in order to implement targeted interventions.

[0099] Furthermore, simulating the data processing results before and after the semantic drift to obtain the differences in data processing results means that the system uses a representative set of data to process the data using both the semantic model before and after the drift, and compares the differences between the two processing results. For example, in sentiment analysis tasks, this involves comparing the changes in the sentiment polarity or intensity assigned to the same text before and after semantic drift. The purpose is to quantify the degree of impact of semantic drift on specific analysis results, providing data support for subsequent adjustments.

[0100] Therefore, providing suggestions for adjusting the parameters of the downstream analysis task or data source based on the differences in the data processing results means that the system intelligently generates a series of suggestions based on the differences observed in the simulation results to help the downstream task adapt to the new semantic environment. These suggestions may include adjusting model parameters (e.g., classifier thresholds, feature weights), switching to a data source more suitable for the new semantics, or suggesting retraining the relevant model. The purpose is to guide users or automated systems to intervene effectively to maintain the accuracy and robustness of the downstream task.

[0101] Furthermore, adjusting the priority of subsequent information processing flows based on the potential impact score means that the system dynamically adjusts the execution priority of affected downstream tasks or related information processing flows based on the potential impact score of semantic drift. For example, if a semantic drift with a high impact range is detected, the key downstream tasks related to that drift may be given higher priority to ensure they can be reviewed and adjusted as soon as possible, thereby minimizing potential negative impacts. The aim is to optimize system resource allocation and ensure that semantic drifts with the greatest impact on critical business operations are processed first.

[0102] This application's solution achieves a more comprehensive and in-depth understanding of semantic drift by introducing association expression analysis, potential impact range assessment, and downstream task impact simulation. Through these technical solutions, this application significantly improves the robustness and adaptability of intelligent information retrieval systems when handling dynamic semantic changes. Compared to existing methods that merely identify semantic drift and update semantic patterns, this application, by deeply analyzing the association expression of semantic drift and its potential impact range, can more comprehensively assess the potential risks of semantic drift to the entire information processing chain. Furthermore, by simulating differences in data processing results and providing targeted adjustment suggestions, this application enables the system to proactively optimize and adjust downstream analysis tasks, effectively avoiding analysis result deviations or erroneous decisions caused by semantic changes. Simultaneously, adjusting the priority of subsequent processing flows based on the potential impact range score ensures that critical business tasks receive priority attention and processing, optimizes system resource allocation, and improves the overall efficiency and accuracy of information processing, thereby maintaining high quality and high reliability of intelligent information retrieval in complex and ever-changing big data environments.

[0103] Based on the differences in data processing results, the above provides suggestions for adjusting downstream analysis task parameters or data sources, including: Identify data features or semantic elements that cause differences in the data processing results; Based on a pre-defined adjustment strategy library, a set of candidate adjustment schemes is generated, including parameter adjustment ranges, data source switching options, or model retraining suggestions. Record the adoption results of market researchers regarding the candidate adjustment proposals; Based on the adoption results, the priority and recommendation logic of the solutions in the adjustment strategy library are adjusted.

[0104] Specifically, identifying data features or semantic elements that lead to differences in data processing results means that after simulating the data processing results before and after semantic drift and obtaining the differences, the system further analyzes the specific manifestations of these differences. For example, is the semantic change of a specific word causing the deviation in the classification results, or is the decline in the quality of a data source affecting the accuracy of the analysis? This can be achieved through feature engineering, statistical analysis, or interpretive analysis using machine learning models on the discrepancies in the data. The goal is to accurately pinpoint the root cause of the problem and provide a basis for subsequent adjustments.

[0105] The generation of a set of candidate adjustment schemes based on a pre-established adjustment strategy library means that after identifying the discrepancies in data features or semantic elements, the system generates a series of possible solutions based on a pre-built strategy library. This adjustment strategy library can contain various types of adjustment suggestions. For example, for parameter adjustment, it can provide different parameter adjustment ranges, such as adjusting thresholds, weights, or model hyperparameters; for data sources, it can provide options to switch to an alternative data source, introduce a new data source, or clean and preprocess existing data sources; for models, it can suggest model retraining, including using a new dataset, adjusting the model architecture, or updating the training algorithm. Its purpose is to provide diverse and selectable solutions to address different types of semantic drift problems.

[0106] In practical applications, recording the adoption results of candidate adjustment solutions by market researchers means that the system tracks and records which candidate adjustment solutions market researchers selected in actual operation, as well as the feedback on the effects of these solutions in actual application. This can be achieved through selection recording on the user interface, log analysis, or direct user feedback mechanisms. The purpose is to collect real user behavior data to provide a basis for subsequent strategy optimization.

[0107] Furthermore, adjusting the priority of solutions in the strategy library and the recommendation logic based on the adoption results means that the system dynamically updates the priority of each solution in the strategy library according to the adoption status of different candidate solutions by market researchers. For example, if a solution is frequently adopted and performs well, its priority will be increased; conversely, if a solution is rarely adopted or performs poorly, its priority will be decreased. Simultaneously, the recommendation logic will also be optimized based on the adoption results, enabling the system to more intelligently recommend solutions that best meet user needs and actual effects when providing suggestions in the future. The aim is to achieve adaptive learning and continuous optimization of the adjustment strategies, improving the accuracy and practicality of the suggestions.

[0108] This application's solution effectively addresses the lack of specificity and adaptability in providing adjustment suggestions by introducing in-depth identification of the underlying causes of differences in data processing results, generation of diverse candidate solutions, and a feedback learning mechanism based on user adoption results. Through the aforementioned technical solutions, this application significantly improves the intelligence and adaptability of adjustment suggestions for downstream analysis tasks after semantic drift occurs. Compared to traditional methods that only provide suggestions based on preset rules or simple difference judgments, this application provides more accurate and targeted adjustment solutions by deeply analyzing the root causes of differences in data processing results. Furthermore, by establishing a candidate solution library containing multiple adjustment types and dynamically learning and optimizing based on actual adoption results from market researchers, the provided suggestions are not only diversified but also better adapted to different user preferences and project needs, thereby effectively improving the adoption rate and actual effect of adjustment suggestions. Ultimately, this ensures that downstream analysis tasks maintain high accuracy and reliability even in complex and ever-changing semantic environments.

[0109] In some of the embodiments described above in this application, although a set of candidate adjustment schemes is generated based on the differences in data processing results, and the adoption results of market researchers on these schemes are recorded, in practical applications, market researchers often need to weigh the potential impact and cost of multiple candidate adjustment schemes. A lack of systematic evaluation of each candidate scheme may lead to the selection of a suboptimal scheme, thereby affecting the efficiency and accuracy of downstream analysis tasks and even causing unnecessary waste of resources.

[0110] In this regard, this application further proposes that the method also includes: evaluating the expected impact of each of the candidate adjustment schemes on the key indicators of the downstream analysis task, as well as the computational resources and time costs required for each of the candidate adjustment schemes, to obtain the evaluation results.

[0111] Specifically, assessing the expected impact of each candidate adjustment scheme on key metrics of downstream analytics tasks refers to predicting and quantifying the potential impact of each candidate adjustment scheme on key performance indicators (e.g., accuracy, recall, F1 score, conversion rate, user satisfaction) of downstream analytics tasks (e.g., sentiment analysis, topic identification, user profiling). This can be achieved through simulations, small-scale experiments, historical data analysis, or predictive models based on expert knowledge. For example, for model retraining suggestions, the extent to which they improve model accuracy can be predicted; for data source switching options, the improvement in data coverage and timeliness can be predicted.

[0112] Evaluating the computational resources and time costs required for each candidate adjustment scheme can be understood as estimating the hardware resources (such as CPU, GPU, memory, and storage), software resources (such as specific libraries and frameworks), and the time required to complete the scheme (such as data preparation time, model training time, deployment time, and validation time) needed to implement each candidate scheme. This helps market researchers fully consider the feasibility and cost-effectiveness of implementation schemes when making decisions.

[0113] Therefore, by comprehensively considering the expected impact, computing resources, and time costs, an evaluation result can be obtained. This result can be expressed as a comprehensive score, priority ranking, or detailed evaluation report for each candidate solution, providing a basis for market researchers' decision-making.

[0114] This application's solution effectively addresses the aforementioned limitations by introducing a systematic evaluation mechanism for candidate adjustment schemes. Through this technical solution, this application provides market researchers with a more scientific and quantitative decision support tool. This solution not only identifies downstream analysis tasks requiring adjustment but also further evaluates the potential benefits and costs of different adjustment schemes, thereby significantly improving the quality and efficiency of decision-making. This ensures that the selected adjustment scheme maximizes the optimization of downstream analysis task performance while minimizing resource consumption and time costs, thereby improving the adaptability and robustness of the entire intelligent information retrieval system and enabling it to more effectively cope with dynamic changes in data semantics.

[0115] In some embodiments described above, this application proposes to evaluate the expected impact of each candidate adjustment scheme on key indicators of downstream analysis tasks, as well as the computational resources and time costs required for each candidate adjustment scheme, to obtain evaluation results. However, in practical applications, market researchers' decision-making preferences and the priority settings of different research projects often differ. If only a uniform evaluation standard is used, the generated evaluation results may not fully match the actual needs or project objectives of market researchers, thereby affecting the adoption efficiency and final effect of the adjustment scheme. To address this, this application further proposes a more refined evaluation method that combines the personalized preferences of market researchers and the specific priorities of research projects to generate more adaptive evaluation results.

[0116] The above assessment evaluates the expected impact of each candidate adjustment scheme on key indicators of downstream analysis tasks, as well as the computational resources and time costs required for each candidate adjustment scheme, yielding the evaluation results, which specifically include: The system retrieves the identity information of market researchers and the identifier of the current research project. The identity information of the market researchers can be obtained through user login credentials, unique user IDs, etc., to identify the specific operator; the identifier of the current research project is used to distinguish different market research projects, such as project number and project name.

[0117] Based on the identified information, the system retrieves the market researcher's preference weights for different types of adjustment options from historical interaction records. These preference weights include the relative importance placed on expected impact, computational resources, and time costs. Specifically, the system records whether the researcher previously preferred the option that maximized the effect or the option with the lowest cost or shortest time when selecting adjustment options. This historical preference data can be quantified into weight values; for example, if a researcher frequently chooses high-impact options, their preference weight for "expected impact" will be higher.

[0118] Based on the identification of the research project, the prioritization settings for expected impact, computational resources, and time cost of the research project are obtained from the project configuration. For example, an urgent project might prioritize "time cost," while a long-term strategic project might place greater emphasis on "expected impact." These prioritization settings reflect strategic considerations at the project level.

[0119] By combining the preference weights and priority settings, a weighted evaluation criterion is obtained. This combination process can be achieved through various methods such as weighted averaging and multiplication factors, aiming to comprehensively consider individual preferences and project requirements to form a customized evaluation criterion.

[0120] Using the weighted evaluation criteria, the expected impact, required computational resources, and time cost of each candidate adjustment scheme are weighted and calculated to obtain a comprehensive evaluation score. For example, if a scheme has a high expected impact, low computational resources, and low time cost, and these indicators have high weights in the weighted evaluation criteria, then the scheme will receive a higher comprehensive evaluation score.

[0121] Based on the comprehensive evaluation score, the candidate adjustment schemes are ranked and filtered to generate evaluation results that adapt to the decision-making preferences and project priorities of market researchers. Ranking clearly shows which schemes best meet the current needs of users and projects; filtering eliminates schemes that do not meet minimum requirements or have low priority, thus providing more accurate decision support.

[0122] This application's solution, by incorporating the identity information of market researchers and the identifiers of research projects, can acquire and integrate personalized decision-making preferences and project-level priority settings. Through this technical solution, the application overcomes the limitations of traditional evaluation methods when faced with diverse market researcher preferences and project priorities. Specifically, by introducing the identity information of market researchers and the identifiers of research projects, the system can dynamically acquire and integrate personalized preference weights and project priority settings, thereby constructing a highly customized weighted evaluation standard. This makes the evaluation of candidate adjustment solutions no longer based on a single general standard, but can accurately reflect the actual needs of specific decision-makers and specific projects. Consequently, the generated evaluation results can more accurately guide market researchers in making decisions, significantly improving the adoption rate and implementation effectiveness of adjustment solutions, avoiding resource waste and inefficiency caused by evaluation results not matching actual needs, and thus enhancing the adaptability and practicality of the entire intelligent information retrieval method.

[0123] refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of an intelligent information retrieval system based on big data processing provided in an embodiment of the present invention, including: The input end is used to obtain topic information and unstructured data; The processing unit is used to perform semantic processing on the unstructured data according to the topic information to generate a structured information digest; store the structured information digest in a fast access area; and retrieve query information from the fast access area according to the query intent submitted by the user. The output terminal is used to determine whether the query information meets the query requirements; if so, it generates an instruction to perform a targeted retrieval of the original data.

[0124] This application's system, through a modular design, assigns information acquisition, semantic processing, information storage, query processing, and result output to the input, processing, and output ends respectively, forming an efficient and intelligent information retrieval architecture. This system aims to address the inefficiencies and insufficient semantic understanding of traditional retrieval methods when handling massive amounts of unstructured data and complex query intents. Through the collaborative work of its components, it achieves accurate information acquisition and rapid response.

[0125] The specific steps and implementation details of the intelligent information retrieval method have been described in the above embodiments and will not be repeated here. It should be emphasized that this application further provides a system architecture for implementing the above method. This system, through specific functional module division, ensures the coordinated and efficient operation of each link.

[0126] Specifically, the system of this application includes an input terminal, a processing terminal, and an output terminal.

[0127] The input terminal is configured to acquire topic information and unstructured data. In one implementation, the input terminal can be a data acquisition module that connects to external data sources (such as the internet, social media platforms, or internal enterprise databases) via a network interface and can receive topic information manually entered by the user through a graphical user interface. This input terminal can be a standalone server or a submodule integrated into a larger system, its primary responsibility being to ensure the reliable and efficient inflow of raw data and topic information. In some embodiments, the input terminal can be a hardware unit configured with multiple data connectors (such as HTTP / HTTPS connectors, FTP connectors, and database connectors), capable of actively or passively acquiring unstructured data in different formats according to a preset data source configuration.

[0128] The processing unit is configured to perform semantic processing on unstructured data based on topic information to generate a structured information digest; store the structured information digest in a fast access region (QAR); and retrieve query information from the QAR based on the user's submitted query intent. The processing unit can be a high-performance computing cluster with multiple processing units deployed internally. One semantic processing unit is responsible for performing natural language processing tasks, such as word segmentation, entity recognition, and relation extraction, to transform unstructured data into a structured digest. Another storage management unit is responsible for writing the generated structured information digest into a QAR, which can be an in-memory database or a high-speed cache system. Furthermore, the processing unit includes a query parsing and matching unit to receive the user's query intent and retrieve matching query information from the QAR. Alternatively, the processing unit can be a distributed service cluster composed of multiple virtual machines or containers, each service responsible for a specific processing task, such as a data cleaning service, a semantic analysis service, a digest storage service, and a query matching service, communicating and coordinating through a message queue.

[0129] The output end is configured to determine whether the query information meets the query requirements; if so, it generates an instruction to perform a targeted retrieval of the original data. The output end can be a decision and instruction generation module that receives the query information and query requirements provided by the processing end and executes matching judgment logic. When the judgment result indicates that the requirements are met, this module will generate a specific retrieval instruction based on the original data identifier or path contained in the query information. This instruction can be an API call, a database query statement, or a file system path, used to guide the backend system to accurately locate and extract the original data. In some embodiments, the output end can be an independent microservice that focuses on evaluating the query matching degree and, when the conditions are met, sends a targeted retrieval request to the original data storage system through a standardized interface.

[0130] This application presents an intelligent information retrieval system based on big data processing. Through its unique modular architecture, it effectively solves the core problems of traditional information retrieval systems, such as low efficiency, insufficient semantic understanding, and inaccurate search results when processing massive amounts of unstructured data. Traditional information retrieval systems often employ a tightly coupled design, making it difficult to flexibly cope with the diversity of data sources and the complexity of query intents. For example, existing systems demonstrate significantly insufficient semantic understanding capabilities when faced with external market research data containing a large amount of internet slang and emerging industry terminology, resulting in an inability to accurately identify query intents.

[0131] This application's system achieves specialization and optimization of each stage by clearly defining the functions of the input, processing, and output ends. The input end efficiently aggregates multi-source, heterogeneous unstructured data and topical information, providing high-quality input for subsequent processing. The processing end, as the core of the system, integrates advanced semantic processing capabilities, transforming complex unstructured data into structured information summaries and storing them in a fast-access area, greatly improving information processability and query efficiency. This preprocessing and caching mechanism allows the system to directly retrieve query information from efficient summaries when receiving user queries, rather than performing real-time parsing from raw, massive unstructured data, significantly reducing computational load during queries and improving response speed.

[0132] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An intelligent information retrieval method based on big data processing, characterized in that, include: Obtain topic information and unstructured data; Based on the topic information, the unstructured data is semantically processed to generate a structured information summary; The structured information digest is stored in a fast-access area; Query information is obtained from the fast access area based on the user's submitted query intent; Determine whether the query information meets the query requirements; If so, then generate instructions to perform a targeted retrieval of the original data.

2. The intelligent information retrieval method based on big data processing according to claim 1, characterized in that, The step of performing semantic processing on the external unstructured data based on the topic information to generate a structured information summary includes: Identify new expressions in the external unstructured data; The meaning of the new expression is inferred by analyzing the contextual information of the new expression. Obtain feedback information on the semantic transformation result of the new expression; Update the semantic transformation rules based on the inferred meaning and the feedback information; Based on the updated semantic transformation rules, the external unstructured data is semantically processed to generate a structured information digest.

3. The intelligent information retrieval method based on big data processing according to claim 1, characterized in that, The step of obtaining query information from the fast access area based on the user-submitted query includes: By tracking the key expressions appearing in the query and the key expressions contained in the information summary in the fast access area, the tracked key expressions are obtained; When a change in the meaning of the tracked key expression is detected, a contextual analysis process is initiated, which includes: Extract the contextual information of the key expressions being tracked from the original text fragments in which they appear; By combining the contextual information and the frequency of use and co-occurring words of the tracked key expressions in different time periods and community contexts, the meaning of the tracked key expressions in the current context is inferred, and the inferred meaning is obtained. By combining the historical feedback of the tracked key expressions, the inferred meaning is calibrated to obtain the calibrated meaning of the key expressions; Using the calibrated key expression meanings, calculate the semantic matching degree between the query intent and the information summary; When the semantic matching degree reaches a preset threshold, the information digest is obtained.

4. The intelligent information retrieval method based on big data processing according to claim 3, characterized in that, When a change in the meaning of the tracked key expression is detected, a contextual analysis process is initiated, including: Obtain the semantic change monitoring priority for each of the key expressions being tracked; Based on the semantic change monitoring priority, adjust the allocation of computing resources for the meaning change recognition task; Analyze the semantic patterns of high-priority key expressions and compare them with historical semantic patterns; When the comparison results show that the semantic pattern of high-priority key expressions deviates significantly from the historical semantic pattern, the contextual analysis process is initiated.

5. The intelligent information retrieval method based on big data processing according to claim 4, characterized in that, The analysis of semantic patterns of high-priority key expressions and comparison with historical semantic patterns includes: Contextual segmentation is performed on the text in the latest data stream where the high-priority key expressions appear; For each sub-context, the co-occurrence word distribution, sentiment distribution, and topic distribution of the high-priority key expressions are extracted to obtain the semantic pattern of the high-priority key expressions in the specific sub-context. The semantic patterns in each sub-context are compared with the historical semantic patterns of the high-priority key expressions; If a significant deviation exists, analyze whether the deviation is a semantic drift of the subdivided context; The overall semantic pattern of the high-priority key expressions is updated based on the semantic drift of the subdivided context.

6. The intelligent information retrieval method based on big data processing according to claim 5, characterized in that, If a significant deviation exists, the analysis will determine whether the deviation is a semantic shift within the subdivided context, including: Obtain the associated expressions of key expressions that show significant deviations between semantic drift and historical patterns in the subdivided context; By analyzing the co-occurrence frequency, semantic distance, and mention status of the associated expressions in historical data, a potential impact range score for the semantic drift is obtained. Identify downstream analysis tasks related to key representations of the semantic shift; Simulate the data processing results before and after the semantic shift to obtain the differences in data processing results; Based on the differences in the data processing results, suggestions are provided for adjusting the parameters of the downstream analysis task or the data source; The priority of subsequent information processing procedures will be adjusted based on the potential impact range score.

7. The intelligent information retrieval method based on big data processing according to claim 6, characterized in that, The suggestion to adjust the parameters of the downstream analysis task or the data source based on the differences in the data processing results includes: Identify data features or semantic elements that cause differences in the data processing results; Based on a pre-defined adjustment strategy library, a set of candidate adjustment schemes is generated, including parameter adjustment ranges, data source switching options, or model retraining suggestions. Record the adoption results of market researchers regarding the candidate adjustment proposals; Based on the adoption results, the priority and recommendation logic of the solutions in the adjustment strategy library are adjusted.

8. The intelligent information retrieval method based on big data processing according to claim 7, characterized in that, The method further includes: The expected impact of each candidate adjustment scheme on key indicators of downstream analysis tasks is evaluated, along with the computational resources and time costs required for each candidate adjustment scheme, to obtain the evaluation results.

9. The method according to claim 8, characterized in that, The evaluation results, obtained by assessing the expected impact of each candidate adjustment scheme on key indicators of downstream analysis tasks, and the computational resources and time costs required for each candidate adjustment scheme, include: Obtain the identity information of market researchers and the identifier of current research projects; Based on the identity information, the market researchers' preference weights for different types of adjustment schemes are obtained from historical interaction records. The preference weights include the relative importance attached to expected impact, computing resources, and time costs. Based on the identifier of the research project, obtain the priority settings of the research project for expected impact, computing resources and time cost from the project configuration; By combining the preference weights and the priority settings, a weighted evaluation criterion is obtained; Using the weighted evaluation criteria, the expected impact, required computational resources, and time cost of each candidate adjustment scheme are weighted and calculated to obtain a comprehensive evaluation score; Based on the comprehensive evaluation score, the candidate adjustment schemes are ranked and screened to generate evaluation results that adapt to the decision-making preferences and project priorities of market researchers.

10. An intelligent information retrieval system based on big data processing, characterized in that, include: The input end is used to obtain topic information and unstructured data; The processing unit is used to perform semantic processing on the unstructured data according to the topic information to generate a structured information digest; store the structured information digest in a fast access area; and retrieve query information from the fast access area according to the query intent submitted by the user. The output terminal is used to determine whether the query information meets the query requirements; if so, it generates an instruction to perform a targeted retrieval of the original data.

Images