Intelligent data analysis method and system based on semantic interaction

Abstract

The present invention discloses an intelligent data analysis method and system based on semantic interaction, which belongs to the technical field of big data processing and analysis. The technical problem to be solved by the present invention is the limitations and insufficient analysis depth of traditional data analysis methods when processing large-scale, multi-dimensional and semantically complex data sets. The technical solution adopted is: multimodal data acquisition and preprocessing data: efficiently collect data from multi-source heterogeneous data sources, and perform distributed cleaning on the collected multimodal data by removing duplicate data, supplementing missing values and replacing erroneous data. The cleaned data is labeled and classified to construct a corpus; the pre-trained language model is trained using the corpus to generate a semantic interaction model with semantic intelligent parsing capabilities; the user input content is analyzed and processed based on the trained semantic interaction model, and finally a data result set that meets the user's requirements is output.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing and analysis, and specifically to an intelligent data analysis method and system based on semantic interaction. Background Art

[0002] With the rapid development of big data technology, businesses and research institutions have accumulated massive amounts of data resources in their daily operations and decision-making processes. These resources are not only massive in volume but also diverse in form, including structured data (such as database records), semi-structured data (such as log files and XML documents), and unstructured data (such as text, images, and audio). However, how to quickly and accurately extract valuable information from this vast amount of data has become a major challenge in the current data analysis field.

[0003] Traditional data analysis methods primarily rely on pre-set algorithmic models and rules, applying statistical, mining, and visualization operations to reveal hidden patterns and patterns within the data. However, these methods often face numerous limitations when working with large-scale, multi-dimensional, and semantically complex datasets. For example, they struggle to automatically understand the semantic meaning of the data, resulting in analysis results that may not accurately reflect the data's true intent. Furthermore, traditional methods often require users to possess a high level of technical expertise to effectively explore and query data, undoubtedly increasing the barrier to entry and the cost of data analysis.

[0004] In recent years, with the rapid development of natural language processing (NLP) and artificial intelligence technologies, semantic interaction technology has gradually emerged. By simulating human language comprehension and communication capabilities, semantic interaction technology enables computer systems to more accurately understand user query intent and contextual information, thereby providing more intelligent and personalized data analysis services. However, applying semantic interaction technology to actual data analysis scenarios still faces many technical challenges. For example, how to efficiently build and maintain large-scale data semantic models to support complex data query and understanding, how to design intuitive and easy-to-use interactive interfaces to reduce users' technical barriers, and how to combine domain knowledge and machine learning algorithms to achieve intelligent data analysis and decision support. Summary of the Invention

[0005] The technical task of the present invention is to provide an intelligent data analysis method and system based on semantic interaction to solve the limitations and insufficient analysis depth of traditional data analysis methods when processing large-scale, multi-dimensional and semantically complex data sets.

[0006] The technical task of the present invention is achieved in the following manner: an intelligent data analysis method based on semantic interaction, the method is as follows:

[0007] Multimodal data acquisition and preprocessing: Efficiently collect data from multiple heterogeneous data sources, analyze the number of data sources to be collected, the data processing complexity coefficient, the processing power per core, and the configuration parameters of a single physical machine, calculate the number of nodes in the data collection cluster, and perform distributed cleaning of the collected multimodal data by removing duplicate data, supplementing missing values, and replacing erroneous data. The cleaned data is then labeled and classified to construct a corpus. The annotation content includes the semantic categories and contextual information of the data.

[0008] Use the corpus to train the pre-trained language model to generate a semantic interaction model with semantic intelligent parsing capabilities;

[0009] Analyze and process user input content based on the trained semantic interaction model, and ultimately output a data result set that meets user requirements;

[0010] Use low-code technology to build a user interaction interface, collect user feedback through the user interaction interface, and optimize and adjust the semantic interaction model.

[0011] As a preference, the number of nodes in the data collection cluster is calculated as follows:

[0012] Calculate the total number of virtual cores required (vCores) using the following formula:

[0013] vCores = (total data volume × processing complexity coefficient) / unit core processing capacity;

[0014] Calculate the total required memory vMemorys using the following formula:

[0015] vMemorys = total data volume × memory consumption coefficient;

[0016] The cluster size is calculated based on the total number of virtual cores (vCores) and the total memory required (vMemorys). The number of nodes in the data collection cluster, nums, is then calculated using the following formula:

[0017] nums = MAX (total required number of virtual cores vCore / number of virtual cores on a single node, total required memory vMemorys / memory on a single node).

[0018] As a preference, the pre-trained language model is trained using the corpus to generate a semantic interaction model with semantic intelligent parsing capabilities as follows:

[0019] Based on the needs of data analysis scenarios, the vocabulary embedding layer of the pre-trained language model is fine-tuned according to the specific vocabulary of professional terminology and industry terms in the data analysis scenario;

[0020] Using annotated corpora, the pre-trained language model is further trained to enable it to accurately understand natural language instructions in specific fields, improving the accuracy and adaptability of semantic understanding. The pre-trained language model is then trained with a large amount of annotated data to enable it to automatically learn the semantic features and contextual relationships of the language, obtaining a semantic interaction model with intelligent semantic parsing capabilities, and improving the ability to understand complex semantics.

[0021] The trained semantic interaction model is evaluated through cross-validation method, and the semantic interaction model is optimized and adjusted based on the evaluation results. Among them, the optimization indicators focus on the accuracy, recall rate and F1 score of semantic understanding to ensure the performance of the semantic interaction model in real application scenarios of intelligent analysis.

[0022] As a preferred method, the user input content is analyzed and processed based on the trained semantic interaction model, and the final output data result set that meets the user's requirements is as follows:

[0023] Based on the trained semantic interaction model (SI-Model), the natural language query input by the user is semantically parsed and converted into a semantic vector;

[0024] Vector retrieval technology is used to match the parsed semantic vector with the semantic vector in the dataset to quickly retrieve data related to the user's query semantics;

[0025] Sort search results based on semantic matching, and optimize search results based on user history and preferences to improve relevance and accuracy.

[0026] Using the K-Means algorithm, the retrieved data is clustered and analyzed, and the data is divided into multiple clusters according to business type;

[0027] The Apriori algorithm is used to identify frequent co-occurrence relationships between data items. By mining association rules, potential associations and trends between data, as well as patterns, trends and anomalies in the data, are automatically identified, and finally a data result set that meets the user's ideas is output.

[0028] Preferably, the user interface has the following functions:

[0029] ① Support users to interact through natural language input, voice input, drag and drop operations or visual charts;

[0030] ② Present the results of intelligent analysis of user semantics to users in an intuitive and visual way through charts and reports;

[0031] ③ Provide intelligent prompts and automatic completion functions, record user query history and feedback information, and automatically adjust the interactive interface and recommended content.

[0032] An intelligent data analysis system based on semantic interaction, the system comprising:

[0033] The Data Processor is used to efficiently collect data from multiple heterogeneous data sources and dynamically adjust the cluster size to cope with different data volumes through a load-based dynamic expansion algorithm. It also cleans the collected data to remove noise and duplicate data, optimize data quality, and annotate and classify the cleaned data to build a corpus.

[0034] The Semantic Interaction Model Trainer (SIMT) trains pre-trained language models using a corpus and optimizes their performance through a multi-head attention mechanism and regularization techniques. It then obtains a semantic interaction model, evaluates it through cross-validation, and adjusts parameters in real time to improve precision, recall, and F1 scores, ensuring that the semantic interaction model can accurately understand natural language instructions in real-world applications.

[0035] The semantic analyzer (SAD) is used to deeply analyze user input content based on the trained semantic interaction model (SI-Model), extract query intent, entity and relationship semantic information, and then convert natural language into executable semantic instructions. Through context-aware technology, it ensures that the analysis results accurately reflect the user's true intent and provides precise semantic guidance for data retrieval and analysis.

[0036] The Visual Interaction Layer (VIL) is used to use low-code technology to drag and drop components, configure properties, and optimize layout and style through a visual designer to quickly build a simple and easy-to-use visual interface, provide intuitive and easy-to-use visual operations, and present analysis results in the form of intuitive charts and reports, allowing users to deeply explore data through filtering and drilling interactive operations, making data analysis more convenient and efficient.

[0037] Preferably, the data acquisition processor includes:

[0038] The data acquisition engine (DEG) is used to efficiently collect data from multiple heterogeneous data sources, analyze the number of heterogeneous data sources to be collected, the data processing complexity coefficient, the unit core processing capacity, and the configuration parameters of a single physical machine, and calculate the number of nodes in the data acquisition cluster;

[0039] The Data Cleaning Engine (DCE) is used to perform distributed cleaning of the collected multimodal data by removing duplicate data, supplementing missing values, and replacing erroneous data. The cleaned data is then annotated and classified to build a corpus. The annotation content covers the semantic categories and contextual information of the data.

[0040] More optimally, the number of nodes in the data collection cluster is calculated as follows:

[0041] Calculate the total number of virtual cores required (vCores) using the following formula:

[0042] vCores = (total data volume × processing complexity coefficient) / unit core processing capacity;

[0043] Calculate the total required memory vMemorys using the following formula:

[0044] vMemorys = total data volume × memory consumption coefficient;

[0045] The cluster size is calculated based on the total number of virtual cores (vCores) and the total memory required (vMemorys). The number of nodes in the data collection cluster, nums, is then calculated using the following formula:

[0046] nums = MAX (total required number of virtual cores vCore / number of virtual cores on a single node, total required memory vMemorys / memory on a single node).

[0047] Preferably, the semantic interaction model trainer includes:

[0048] The fine-tuning module is used to fine-tune the vocabulary embedding layer of the pre-trained language model based on the specific vocabulary of professional and industry terms in data analysis scenarios.

[0049] The speech interaction model generation module is used to further train the pre-trained language model using the annotated corpus, enabling it to accurately understand natural language instructions in specific fields and improve the accuracy and adaptability of semantic understanding. The pre-trained language model is then trained with a large amount of annotated data, enabling it to automatically learn the semantic features and contextual relationships of the language, thereby obtaining a semantic interaction model with intelligent semantic parsing capabilities and improving the ability to understand complex semantics.

[0050] The model evaluation module is used to evaluate the trained semantic interaction model through cross-validation methods and optimize and adjust the semantic interaction model based on the evaluation results. Among them, the optimization indicators focus on the accuracy, recall rate and F1 score of semantic understanding to ensure the performance of the semantic interaction model in real application scenarios of intelligent analysis.

[0051] Preferably, the semantic parser includes:

[0052] The conversion module is used to perform semantic analysis on the natural language query input by the user based on the trained semantic interaction model (SI-Model) and convert it into a semantic vector;

[0053] The matching module is used to match the parsed semantic vector with the semantic vector in the dataset using vector retrieval technology to quickly retrieve data related to the user's query semantics;

[0054] The sorting module is used to sort the search results according to the semantic matching degree, and optimize the search results based on the user's historical query records and preferences to improve the relevance and accuracy of the search results;

[0055] The cluster analysis module is used to perform cluster analysis on the retrieved data using the K-Means algorithm and divide the data into multiple clusters according to business types;

[0056] The output module is used to use the Apriori algorithm to identify frequent co-occurrence relationships between data items. By mining association rules, it automatically identifies potential associations and trends between data as well as patterns, trends and anomalies in the data, and ultimately outputs a data result set that meets the user's ideas.

[0057] The intelligent data analysis method and system based on semantic interaction of the present invention have the following advantages:

[0058] (1) This invention achieves full automation from natural language query to intelligent analysis by integrating natural language processing, knowledge graph and machine learning technologies. Based on semantic understanding and intent recognition, it converts user queries into structured analysis tasks, and performs semantic expansion and reasoning through dynamic knowledge graphs, automatically associates relevant data dimensions, and intelligently selects the optimal algorithm and executes distributed computing tasks based on the adaptive analysis engine, ultimately presenting the results to data analysts in various forms such as visual charts, tables and reports.

[0059] (2) This invention achieves bidirectional mapping of semantics and data, context-aware interactive analysis, and visual presentation of results, significantly lowering the technical threshold for data analysis and improving analysis efficiency and depth. It can be widely applied in business intelligence, public opinion analysis, command and decision-making, and other fields, providing users with an intelligent data analysis experience where "what you want is what you get";

[0060] (3) This invention combines deep learning semantic understanding technology with domain knowledge graphs to achieve intelligent processing and analysis of multi-source heterogeneous data. By constructing four core technical modules: Data Processor, Semantic Interaction Model Trainer (SIMT), Semantic Parser (SAD), and Visual Interaction Layer (VIL), it addresses the limitations and insufficient analytical depth of traditional data analysis methods when processing large-scale, multi-dimensional, and semantically complex data sets, providing a new generation of data analysis infrastructure for the development of smart government.

[0061] (4) The present invention can automatically understand the semantic meaning of data, support users to query and explore data through intuitive means such as natural language, and simultaneously combine domain knowledge and machine learning algorithms to achieve deep mining and intelligent analysis of data. This can not only significantly improve the efficiency and accuracy of data analysis and lower the technical threshold for users, but also provide more intelligent and personalized data decision support services for various industries;

[0062] (V) In view of the challenges currently faced in the field of data analysis, especially when dealing with large-scale, multi-dimensional, and semantically complex data sets, the limitations of traditional data analysis methods are becoming increasingly prominent. The advantages of the present invention are mainly reflected in the following aspects:

[0063] ① Improving the intelligence level of data analysis: By introducing advanced natural language processing (NLP) technology and a semantic understanding engine, this invention aims to achieve a precise understanding of data semantics, encompassing not only the basic meaning of data items but also the relationships between data and contextual information, thereby providing users with more intelligent and personalized data analysis services. This increased intelligence will greatly enhance the depth and breadth of data analysis, helping users to uncover more valuable insights from massive amounts of data.

[0064] Optimizing the user interaction experience: Traditional data analysis methods often require users to have a high technical background to effectively explore and query data. However, this invention uses an intuitive and easy-to-use interactive data analysis interface to support users to interact with the system in various ways, such as natural language, drag-and-drop operations, or visual charts. This user-friendly design significantly lowers the technical threshold for users, allowing non-professionals to easily get started and enjoy the convenience brought by data analysis.

[0065] ③ Enhance the accuracy and efficiency of data analysis: Combining domain knowledge and machine learning algorithms, this invention aims to achieve deep mining and intelligent analysis of data. By automatically identifying patterns, trends, and anomalies in data, it can provide users with advanced functions such as predictive analysis, association rule mining, and sentiment analysis. This will not only improve the accuracy of data analysis, but also significantly enhance analysis efficiency, enabling users to make data-driven decisions faster.

[0066] ④ Promote innovation and development of data analysis technology: This invention not only solves practical problems currently facing the field of data analysis, but also provides new ideas and directions for the future development of data analysis technology. By integrating advanced technologies such as semantic interaction, natural language processing, machine learning, and domain knowledge graphs, this invention is expected to lead technological innovation in the field of data analysis and promote the sustainable development of the entire industry.

[0067] ⑤ Maximizing the use of data value: In the data-driven era, the value of data lies in its ability to be effectively analyzed and utilized. This invention provides intelligent data analysis methods to help businesses, research institutions, government departments, and other industries better explore and utilize data resources, thereby promoting business innovation, optimizing decision-making processes, and improving operational efficiency, ultimately maximizing the use of data value.

[0068] To sum up, the present invention solves the limitations of traditional data analysis methods when processing complex data, improves the intelligence level of data analysis, optimizes user interaction experience, enhances the accuracy and efficiency of data analysis, and at the same time promotes the innovation and development of data analysis technology, and promotes the maximum utilization of data value. It will provide more efficient and intelligent data analysis solutions for all walks of life, and help digital transformation and intelligent upgrading. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The present invention will be further described below with reference to the accompanying drawings.

[0070] Attachment Figure 1 This is a structural diagram of an intelligent data analysis system based on semantic interaction. DETAILED DESCRIPTION

[0071] The intelligent data analysis method and system based on semantic interaction of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0072] Example 1: This example provides an intelligent data analysis method based on semantic interaction, which is as follows:

[0073] S1. Multimodal Data Collection and Preprocessing: Efficiently collect data from multi-source heterogeneous data sources, analyze the number of data sources to be collected, the data processing complexity coefficient, the unit core processing capacity, and the configuration parameters of a single physical machine, calculate the number of nodes in the data collection cluster, and perform distributed cleaning of the collected multimodal data by removing duplicate data, supplementing missing values, and replacing erroneous data. The cleaned data is then labeled and classified to construct a corpus. The annotation content includes the semantic categories and contextual information of the data. Multi-source heterogeneous data sources include business data, IoT data, system logs, monitoring data, user behavior data, etc.

[0074] S2. Use the corpus to train the pre-trained language model to generate a semantic interaction model with semantic intelligent parsing capabilities;

[0075] S3: Analyze and process user input content based on the trained semantic interaction model, and ultimately output a data result set that meets user requirements;

[0076] S4. Use low-code technology to build a user interaction interface, collect user feedback through the user interaction interface, and optimize and adjust the semantic interaction model.

[0077] The number of nodes in the data collection cluster calculated in step S1 of this embodiment is as follows:

[0078] S101. Calculate the total number of virtual cores required (vCores) using the following formula:

[0079] vCores = (total data volume × processing complexity coefficient) / unit core processing capacity;

[0080] S102. Calculate the total required memory vMemorys using the following formula:

[0081] vMemorys = total data volume × memory consumption coefficient;

[0082] S103. Calculate the cluster size based on the total number of virtual cores (vCores) and the total memory required (vMemorys), and then calculate the number of nodes (nums) in the data collection cluster using the following formula:

[0083] nums = MAX (total required number of virtual cores vCore / number of virtual cores on a single node, total required memory vMemorys / memory on a single node).

[0084] In step S2 of this embodiment, the pre-trained language model is trained using the corpus to generate a semantic interaction model with semantic intelligent parsing capabilities as follows:

[0085] S201. Based on the data analysis scenario requirements, fine-tune the vocabulary embedding layer of the pre-trained language model according to the specific vocabulary of professional terms and industry terms in the data analysis scenario;

[0086] S202. Further train the pre-trained language model using the annotated corpus, so that the pre-trained language model can accurately understand natural language instructions in a specific field, thereby improving the accuracy and adaptability of semantic understanding; then train the pre-trained language model using a large amount of annotated data, so that the pre-trained language model can automatically learn the semantic features and contextual relationships of the language, obtain a semantic interaction model with intelligent semantic parsing capabilities, and improve the ability to understand complex semantics;

[0087] S203. Evaluate the trained semantic interaction model through cross-validation method, and optimize and adjust the semantic interaction model based on the evaluation results; among them, the optimization indicators focus on the accuracy, recall rate and F1 score of semantic understanding to ensure the performance of the semantic interaction model in the real application scenarios of intelligent analysis.

[0088] In step S3 of this embodiment, the user input content is analyzed and processed based on the trained semantic interaction model, and the data result set that meets the user's requirements is finally output as follows:

[0089] S301, semantically analyzing the natural language query input by the user based on the trained semantic interaction model (SI-Model) and converting it into a semantic vector;

[0090] S302, using vector retrieval technology to match the parsed semantic vector with the semantic vector in the data set, and quickly retrieve data related to the user's query semantics;

[0091] S303: Sort the search results based on semantic matching, and optimize the search results based on the user's historical query records and preferences to improve the relevance and accuracy of the search results;

[0092] S304: Perform cluster analysis on the retrieved data using the K-Means algorithm, and divide the data into multiple clusters according to business types;

[0093] S305. Use the Apriori algorithm to identify frequent co-occurrence relationships between data items. By mining association rules, automatically identify potential associations and trends between data as well as patterns, trends and anomalies in the data, and ultimately output a data result set that meets the user's requirements.

[0094] The user interaction interface in step S4 of this embodiment has the following functions:

[0095] ① Support users to interact through natural language input, voice input, drag and drop operations or visual charts;

[0096] ② Present the results of intelligent analysis of user semantics to users in an intuitive and visual way through charts and reports;

[0097] ③ Provide intelligent prompts and automatic completion functions, record user query history and feedback information, and automatically adjust the interactive interface and recommended content.

[0098] Example 2: As shown in the attached Figure 1 As shown, this embodiment provides an intelligent data analysis system based on semantic interaction, which includes:

[0099] The Data Processor is used to efficiently collect data from multiple heterogeneous data sources. It dynamically adjusts the cluster size to cope with different data volumes through a load-based dynamic expansion algorithm. It also cleans the collected data to remove noise and duplicate data, optimize data quality, and annotate and classify the cleaned data to build a corpus. Multi-source heterogeneous data sources include business data, IoT data, system logs, monitoring data, user behavior data, and more.

[0100] The Semantic Interaction Model Trainer (SIMT) trains pre-trained language models using a corpus and optimizes their performance through a multi-head attention mechanism and regularization techniques. It then obtains a semantic interaction model, evaluates it through cross-validation, and adjusts parameters in real time to improve precision, recall, and F1 scores, ensuring that the semantic interaction model can accurately understand natural language instructions in real-world applications.

[0101] The semantic analyzer (SAD) is used to deeply analyze user input content based on the trained semantic interaction model (SI-Model), extract query intent, entity and relationship semantic information, and then convert natural language into executable semantic instructions. Through context-aware technology, it ensures that the analysis results accurately reflect the user's true intent and provides precise semantic guidance for data retrieval and analysis.

[0102] The Visual Interaction Layer (VIL) is used to use low-code technology to drag and drop components, configure properties, and optimize layout and style through a visual designer to quickly build a simple and easy-to-use visual interface, provide intuitive and easy-to-use visual operations, and present analysis results in the form of intuitive charts and reports, allowing users to deeply explore data through filtering and drilling interactive operations, making data analysis more convenient and efficient.

[0103] The data acquisition processor in this embodiment includes:

[0104] The data acquisition engine (DEG) is used to efficiently collect data from multiple heterogeneous data sources, analyze the number of heterogeneous data sources to be collected, the data processing complexity coefficient, the unit core processing capacity, and the configuration parameters of a single physical machine, and calculate the number of nodes in the data acquisition cluster;

[0105] The Data Cleaning Engine (DCE) is used to perform distributed cleaning of the collected multimodal data by removing duplicate data, supplementing missing values, and replacing erroneous data. The cleaned data is then annotated and classified to build a corpus. The annotation content covers the semantic categories and contextual information of the data.

[0106] The number of nodes in the calculation data collection cluster in this embodiment is as follows:

[0107] Calculate the total number of virtual cores required (vCores) using the following formula:

[0108] vCores = (total data volume × processing complexity coefficient) / unit core processing capacity;

[0109] Calculate the total required memory vMemorys using the following formula:

[0110] vMemorys = total data volume × memory consumption coefficient;

[0111] The cluster size is calculated based on the total number of virtual cores (vCores) and the total memory required (vMemorys). The number of nodes in the data collection cluster, nums, is then calculated using the following formula:

[0112] nums = MAX (total required number of virtual cores vCore / number of virtual cores on a single node, total required memory vMemorys / memory on a single node).

[0113] The semantic interaction model trainer in this embodiment includes:

[0114] The fine-tuning module is used to fine-tune the vocabulary embedding layer of the pre-trained language model based on the specific vocabulary of professional and industry terms in data analysis scenarios.

[0115] The speech interaction model generation module is used to further train the pre-trained language model using the annotated corpus, enabling it to accurately understand natural language instructions in specific fields and improve the accuracy and adaptability of semantic understanding. The pre-trained language model is then trained with a large amount of annotated data, enabling it to automatically learn the semantic features and contextual relationships of the language, thereby obtaining a semantic interaction model with intelligent semantic parsing capabilities and improving the ability to understand complex semantics.

[0116] The model evaluation module is used to evaluate the trained semantic interaction model through cross-validation methods and optimize and adjust the semantic interaction model based on the evaluation results. Among them, the optimization indicators focus on the accuracy, recall rate and F1 score of semantic understanding to ensure the performance of the semantic interaction model in real application scenarios of intelligent analysis.

[0117] The semantic parser in this embodiment includes:

[0118] The conversion module is used to perform semantic analysis on the natural language query input by the user based on the trained semantic interaction model (SI-Model) and convert it into a semantic vector;

[0119] The matching module is used to match the parsed semantic vector with the semantic vector in the dataset using vector retrieval technology to quickly retrieve data related to the user's query semantics;

[0120] The sorting module is used to sort the search results according to the semantic matching degree, and optimize the search results based on the user's historical query records and preferences to improve the relevance and accuracy of the search results;

[0121] The cluster analysis module is used to perform cluster analysis on the retrieved data using the K-Means algorithm and divide the data into multiple clusters according to business types;

[0122] The output module is used to use the Apriori algorithm to identify frequent co-occurrence relationships between data items. By mining association rules, it automatically identifies potential associations and trends between data as well as patterns, trends and anomalies in the data, and ultimately outputs a data result set that meets the user's ideas.

[0123] The visual interaction layer in this embodiment has the following functions:

[0124] ① Support users to interact through natural language input, voice input, drag and drop operations or visual charts;

[0125] ② Present the results of intelligent analysis of user semantics to users in an intuitive and visual way through charts and reports;

[0126] ③ Provide intelligent prompts and automatic completion functions, record user query history and feedback information, and automatically adjust the interactive interface and recommended content.

[0127] The working process of the system is as follows:

[0128] Step 1: Build a data acquisition processor (DataProcessor) to collect and preprocess data: By building a distributed data acquisition processor (DataProcessor), the collection and preprocessing of multimodal data can be realized; the data acquisition processor (DataProcessor) mainly consists of two parts: the data acquisition engine (DEG) and the data cleaning engine (DCE); first, the data acquisition engine (DEG) is used to analyze the parameters such as the amount of data to be collected, the data processing complexity coefficient, the unit core processing power and the single physical machine configuration, and calculate the number of nodes in the data acquisition cluster. The first step is to calculate the total number of virtual cores (vCores) required: vCores = (total data volume × processing complexity coefficient) / unit core processing power, and the second step is to calculate the total required memory book (vMemorys): vMemorys = total data volume × memory consumption coefficient, and then the cluster size is calculated based on vCores and vMemorys. The number of nodes (nums) of the data acquisition cluster = MAX(total required vCore / single node vCore, Total required memory / single-node memory); then, using the Data Cleaning Engine (DCE), the collected large-scale data is distributed cleaned by removing duplicate data, supplementing missing values, and replacing erroneous data. The cleaned data is annotated and classified to build a corpus. The annotation content needs to cover the semantic categories and contextual information of the data.

[0129] Step 2: Construct a semantic interaction model trainer (SIMT) to train the semantic interaction model: By constructing a semantic interaction model trainer (SIMT), the collected data is used to train the semantic interaction model to generate a semantic interaction model (SI-Model) with semantic intelligent parsing capabilities; first, based on the scenario requirements of data analysis, the vocabulary embedding layer of the pre-trained model is fine-tuned according to the specific vocabulary in the data analysis scenario (such as professional terms, industry terms, etc.); the model is further trained using annotated corpora to enable it to accurately understand natural language instructions in specific fields and improve the accuracy and adaptability of semantic understanding; secondly, the model is trained with a large amount of annotated data to enable it to automatically learn the semantic features and contextual relationships of the language and improve the ability to understand complex semantics; thirdly, the trained model is evaluated through cross-validation, and the model is optimized and adjusted based on the evaluation results; the optimization indicators mainly focus on three aspects: the accuracy of semantic understanding, the recall rate, and the F1 score, to ensure the performance of the model in the real application scenarios of intelligent analysis.

[0130] Step 3: Build a semantic parser (SAD) for semantic parsing and vectorized storage: By building a semantic parser (SAD), semantic parsing of user input content is achieved and converted into vectorized data. First, based on the trained semantic interaction model (SI-Model), semantic parsing of the user's natural language query is performed and converted into semantic vectors. Vector retrieval technology is used to match the parsed semantic vectors with the semantic vectors in the dataset to quickly retrieve data related to the user's query semantics. Second, the retrieval results are sorted according to the semantic matching degree. Combined with the user's historical query records and preferences, the retrieval results are optimized to improve the relevance and accuracy of the retrieval results. Third, the retrieved data is clustered using the K-Means algorithm, and the data is divided into multiple clusters according to business type. The Apriori algorithm is then used to identify frequent co-occurrence relationships between data items. By mining association rules, potential associations and trends between data, as well as patterns, trends, and anomalies in the data, are automatically identified. The final output is a data result set that meets the user's requirements.

[0131] Step 4: Build a visual interaction layer (VIL) to provide users with a visual interactive interface: Leverage low-code technology and use its visual designer to drag components, configure properties, and optimize layout and style to quickly build a simple and easy-to-use interface that supports users to interact with the system through various forms such as natural language input, voice input, drag and drop operations, or visual charts; through the above-mentioned intelligent analysis of user semantics, the analysis results are presented to users in intuitive visual forms such as charts, reports, and reports; provide intelligent prompts and automatic completion functions, record user query history and feedback information, and automatically adjust the interactive interface and recommended content; collect user feedback through the user interaction interface, and optimize and adjust the semantic interaction model and data analysis algorithm.

[0132] At this point, after the above steps, combined with technologies such as machine learning and big models, the user's intelligent interactive experience in data analysis has been optimized, and the intelligence level of data analysis has been greatly improved. Through advanced natural language processing technology and semantic understanding engines, it can accurately understand the semantic meaning of data, provide users with more intelligent and personalized data analysis services, greatly enhance the depth and breadth of data analysis, and help users to dig out more valuable insights from massive data.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent data analysis method based on semantic interaction, characterized in that: The method is as follows: Multimodal data acquisition and preprocessing: Efficiently collect data from multiple heterogeneous data sources, analyze the number of data sources to be collected, the data processing complexity coefficient, the processing power per core, and the configuration parameters of a single physical machine, calculate the number of nodes in the data collection cluster, and perform distributed cleaning of the collected multimodal data by removing duplicate data, supplementing missing values, and replacing erroneous data. The cleaned data is then labeled and classified to construct a corpus. The annotation content includes the semantic categories and contextual information of the data. Use the corpus to train the pre-trained language model to generate a semantic interaction model with semantic intelligent parsing capabilities; Analyze and process user input content based on the trained semantic interaction model, and ultimately output a data result set that meets user requirements; Use low-code technology to build a user interaction interface, collect user feedback through the user interaction interface, and optimize and adjust the semantic interaction model.

2. The intelligent data analysis method based on semantic interaction according to claim 1 is characterized in that: The number of nodes in the data collection cluster is calculated as follows: Calculate the total number of virtual cores required (vCores) using the following formula: vCores = (total number of heterogeneous data sources to be collected × processing complexity coefficient) / unit core processing capacity; Calculate the total required memory vMemorys using the following formula: vMemorys = the total number of heterogeneous data sources to be collected × memory consumption coefficient; The cluster size is calculated based on the total number of virtual cores (vCores) and the total number of memories (vMemorys). The number of nodes in the data collection cluster, nums, is then calculated using the following formula: nums = MAX (total required number of virtual cores vCore / number of virtual cores on a single node, total required memory vMemorys / memory on a single node).

3. The intelligent data analysis method based on semantic interaction according to claim 1 is characterized in that: The pre-trained language model is trained using the corpus to generate a semantic interaction model with intelligent semantic parsing capabilities. The details are as follows: Based on the needs of data analysis scenarios, the vocabulary embedding layer of the pre-trained language model is fine-tuned according to the specific vocabulary of professional terminology and industry terms in the data analysis scenario; The pre-trained language model is further trained using the annotated corpus, enabling it to accurately understand natural language instructions in specific fields. The pre-trained language model is then trained using annotated data, enabling it to automatically learn the semantic features and contextual relationships of the language, thereby acquiring a semantic interaction model with intelligent semantic parsing capabilities. The trained semantic interaction model is evaluated through cross-validation method, and the semantic interaction model is optimized and adjusted based on the evaluation results. Among them, the optimization indicators focus on the accuracy, recall rate and F1 score of semantic understanding to ensure the performance of the semantic interaction model in real application scenarios of intelligent analysis.

4. The intelligent data analysis method based on semantic interaction according to claim 1 is characterized in that: The user input content is analyzed and processed based on the trained semantic interaction model, and the final output is a data result set that meets the user's requirements. The specific details are as follows: Based on the trained semantic interaction model, the natural language query input by the user is semantically parsed and converted into a semantic vector; Vector retrieval technology is used to match the parsed semantic vector with the semantic vector in the dataset to quickly retrieve data related to the user's query semantics; Sort search results based on semantic matching, and optimize search results based on user history and preferences to improve relevance and accuracy. Using the K-Means algorithm, the retrieved data is clustered and analyzed, and the data is divided into multiple clusters according to business type; The Apriori algorithm is used to identify frequent co-occurrence relationships between data items. By mining association rules, potential associations and trends between data, as well as patterns, trends and anomalies in the data, are automatically identified, and finally a data result set that meets the user's ideas is output.

5. The intelligent data analysis method based on semantic interaction according to any one of claims 1 to 4, characterized in that: The user interface has the following functions: ① Support users to interact through natural language input, voice input, drag and drop operations or visual charts; ② Present the results of intelligent analysis of user semantics to users in an intuitive and visual way through charts and reports; ③ Provide intelligent prompts and automatic completion functions, record user query history and feedback information, and automatically adjust the interactive interface and recommended content.

6. An intelligent data analysis system based on semantic interaction, characterized in that: The system includes: The data acquisition processor is used to efficiently collect data from multiple heterogeneous data sources and dynamically adjust the cluster size to cope with different data volumes through a load-based dynamic expansion algorithm. It also performs data cleaning processing on the collected data to remove noise and duplicate data, optimize data quality, and annotate and classify the cleaned data to build a corpus. The semantic interaction model trainer is used to train the pre-trained language model using the corpus, optimize the performance of the pre-trained language model through a multi-head attention mechanism and regularization technology, obtain the semantic interaction model, evaluate the semantic interaction model through cross-validation, and adjust parameters in real time to improve precision, recall, and F1 score, ensuring that the semantic interaction model can accurately understand natural language instructions in practical applications; The semantic parser is used to deeply analyze user input content based on a trained semantic interaction model, extract query intent, entity and relationship semantic information, and then convert natural language into executable semantic instructions. Context-aware technology is used to ensure that the parsing results accurately reflect the user's true intent, providing precise semantic guidance for data retrieval and analysis. The visual interaction layer is used to use low-code technology to drag components, configure properties, and optimize layout and style through the visual designer to quickly build a simple and easy-to-use visual interface, provide intuitive and easy-to-use visual operations, and present analysis results in the form of intuitive charts and reports, allowing users to deeply explore data through filtering and drilling interactive operations, making data analysis more convenient and efficient.

7. The intelligent data analysis system based on semantic interaction according to claim 6 is characterized in that: The data acquisition processor includes: The data collection engine is used to efficiently collect data from multiple heterogeneous data sources. It analyzes the number of heterogeneous data sources to be collected, the data processing complexity coefficient, the unit core processing capacity, and the configuration parameters of a single physical machine, and calculates the number of nodes in the data collection cluster. The data cleaning engine is used to perform distributed cleaning of the collected multimodal data by removing duplicate data, supplementing missing values, and replacing erroneous data, annotating and classifying the cleaned data, and building a corpus; the annotation content covers the semantic categories and contextual information of the data.

8. The intelligent data analysis system based on semantic interaction according to claim 7, characterized in that: The number of nodes in the data collection cluster is calculated as follows: Calculate the total number of virtual cores required (vCores) using the following formula: vCores = (total number of heterogeneous data sources to be collected × processing complexity coefficient) / unit core processing capacity; Calculate the total required memory vMemorys using the following formula: vMemorys = the total number of heterogeneous data sources to be collected × memory consumption coefficient; The cluster size is calculated based on the total number of virtual cores (vCores) and the total number of memories (vMemorys). The number of nodes in the data collection cluster, nums, is then calculated using the following formula: nums = MAX (total required number of virtual cores vCore / number of virtual cores on a single node, total required memory vMemorys / memory on a single node).

9. The intelligent data analysis system based on semantic interaction according to claim 6, characterized in that: The semantic interaction model trainer includes: The fine-tuning module is used to fine-tune the vocabulary embedding layer of the pre-trained language model based on the specific vocabulary of professional and industry terms in data analysis scenarios. The speech interaction model generation module is used to further train the pre-trained language model using the annotated corpus, enabling it to accurately understand natural language instructions in specific fields. The pre-trained language model is then trained using annotated data, enabling it to automatically learn the semantic features and contextual relationships of the language, thereby obtaining a semantic interaction model with intelligent semantic parsing capabilities. The model evaluation module is used to evaluate the trained semantic interaction model through cross-validation methods and optimize and adjust the semantic interaction model based on the evaluation results. Among them, the optimization indicators focus on the accuracy, recall rate and F1 score of semantic understanding to ensure the performance of the semantic interaction model in real application scenarios of intelligent analysis.

10. The intelligent data analysis system based on semantic interaction according to claim 6, characterized in that: The semantic parser includes: The conversion module is used to perform semantic analysis on the natural language query input by the user based on the trained semantic interaction model and convert it into a semantic vector; The matching module is used to match the parsed semantic vector with the semantic vector in the dataset using vector retrieval technology to quickly retrieve data related to the user's query semantics; The sorting module is used to sort the search results according to the semantic matching degree, and optimize the search results based on the user's historical query records and preferences to improve the relevance and accuracy of the search results; The cluster analysis module is used to perform cluster analysis on the retrieved data using the K-Means algorithm and divide the data into multiple clusters according to business types; The output module is used to use the Apriori algorithm to identify frequent co-occurrence relationships between data items. By mining association rules, it automatically identifies potential associations and trends between data as well as patterns, trends and anomalies in the data, and ultimately outputs a data result set that meets the user's ideas.

Images