Data mining analysis system and method based on hybrid model

Abstract

The invention relates to a data mining analysis system and method based on a hybrid model, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps that a data analysis system obtains a data set from a database through SQL query based on a user instruction; determining a target model according to the business scene demand; calculating the data set through the target model to generate a calculation result; automatically carrying out filtering processing on the calculation result; calling a large model to perform data mining analysis on the filtered calculation result to generate a data mining analysis result; and displaying the data mining analysis result at the front end of a data analysis system. According to the data mining analysis system and method based on the hybrid model, the electronic equipment, the computer readable medium and the computer program product, a user can automatically carry out big data analysis and processing more efficiently and more safely, and therefore the overall working efficiency and the data insight ability are greatly improved.

Description

Technical Field

[0001] This application relates to the field of computer information processing. Specifically, it relates to a data mining and analysis system, method, electronic device, computer-readable medium, and computer program product based on a hybrid model. Background Art

[0002] With the rapid growth of data volume and the increase in data complexity, the demand for data insights and decision support from modern enterprises and research institutions has become increasingly complex and diverse. However, traditional data analysis methods have revealed many limitations and drawbacks when dealing with these demands.

[0003] Traditional analysis methods have high requirements for data preprocessing and require a large amount of manual data cleaning, transformation, and formatting work. This not only consumes time and effort but is also prone to errors. The data preparation work requires a large amount of human intervention, leading to the complication of the analysis process and an increase in time costs. Using traditional data analysis tools usually requires users to have relatively high statistical, mathematical, and programming skills. As the complexity of data analysis requirements increases, users need higher professional skills to understand and apply complex statistical models and algorithms, which further increases the difficulty of use. Moreover, traditional data analysis tools usually rely on preset models and algorithms and lack the ability to adapt to complex business scenarios. In addition, these tools usually have poor scalability when dealing with large-scale data and are difficult to support efficient computing and storage in a big data environment.

[0004] Traditional data analysis methods have significant deficiencies when dealing with big data and complex data environments. With the rapid growth of data volume and the increase in complexity, enterprises and research institutions urgently need more efficient, accurate, and comprehensive data analysis tools and platforms.

[0005] Therefore, this application proposes a new data mining and analysis system, method, electronic device, computer-readable medium, and computer program product based on a hybrid model.

[0006] The above information disclosed in the background art section is only used to enhance the understanding of the background of this application. Therefore, it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0007] In view of this, this application provides a data mining and analysis system, method, electronic device, computer-readable medium, and computer program product based on a hybrid model, which can enable users to automate big data analysis and processing more efficiently and securely, thereby greatly improving the overall work efficiency and data insight ability.

[0008] Other features and advantages of this application will become apparent through the following detailed description, or will be partially learned through the practice of this application.

[0009] According to one aspect of the present application, a data mining and analysis method based on a hybrid model is proposed. The method includes: the data analysis system obtains a data set from the database through an SQL query based on a user instruction; determines a target model according to the requirements of the business scenario; calculates the data set through the target model to generate a calculation result; automatically filters the calculation result; invokes a large model to perform data mining and analysis on the calculation result after filtering to generate a data mining and analysis result; and displays the data mining and analysis result on the front end of the data analysis system.

[0010] Optionally, it further includes: the user applies to access the data analysis system through a client; the client is securely verified based on a reverse proxy server; after the security verification is passed, the user is allowed to access the data analysis system.

[0011] Optionally, the data analysis system obtains a data set from the database through an SQL query based on a user instruction, including: the data analysis system provides a front-end access interface through the progressive JavaScript framework Vue3; the user selects the data set to be used in the front-end interface; executes an SQL query to obtain the data set from the database; and preprocesses the data in the data set.

[0012] Optionally, preprocessing the data in the data set includes: exporting the data in the data set in CSV format; performing data cleaning on the data; and converting the data set into a Pandas data set.

[0013] Optionally, determining a target model according to the requirements of the business scenario includes: determining the type of algorithm model from the model library according to the requirements of the business scenario; and determining the target model from multiple models based on the type.

[0014] Optionally, calculating the data set through the target model to generate a calculation result includes: screening and extracting feature data from the data in the data set; and invoking the target model through a python algorithm engine to calculate the data set to generate a calculation result.

[0015] Optionally, automatically filtering the calculation result includes: automatically performing result value filtering on the calculation result; and / or automatically performing valid value extraction on the calculation result; and / or automatically performing data classification on the calculation result; and / or automatically performing sorting and assembly on the calculation result; and / or automatically performing data specification unification on the calculation result.

[0016] Optionally, a large model is invoked to perform data mining analysis on the calculation results after the filtering process, generating data mining analysis results, including: integrating the calculation results after the filtering process; invoking the large model and its corresponding knowledge base through the Java backend; performing data mining analysis on the calculation results after data integration through the large model and its corresponding knowledge base; generating an analysis result set, analysis text content, and visualization charts.

[0017] Optionally, integrating the calculation results after the filtering process includes: performing business scenario analysis on the calculation results after the filtering process; and / or performing data visualization classification on the calculation results after the filtering process; and / or performing data conversion and collection on the calculation results after the filtering process.

[0018] Optionally, presenting the data mining analysis results at the front end of the data analysis system includes: integrating the analysis result set, analysis text content, and visualization charts in the data mining analysis results to generate a data analysis report; presenting the data analysis report at the front end.

[0019] According to one aspect of the present application, a data mining analysis system based on a hybrid model is proposed. The device includes: a data module for obtaining a data set from a database through an SQL query based on a user instruction; a front-end module for determining a target model according to business scenario requirements; a model calculation module for calculating the data set through the target model to generate a calculation result; a data processing module for automatically filtering the calculation result; a large model module for invoking a large model to perform data mining analysis on the calculation result after the filtering process to generate a data mining analysis result; and a display module for presenting the data mining analysis result at the front end of the data analysis system.

[0020] According to one aspect of the present application, an electronic device is proposed. The electronic device includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0021] According to one aspect of the present application, a computer-readable medium is proposed, on which a computer program is stored. When the program is executed by a processor, the method as described above is implemented.

[0022] According to one aspect of the present application, a computer program product is proposed, including: a computer program / instructions, when the computer program / instructions are executed by a processor, the method as described above is implemented.

[0023] A data mining and analysis system, method, electronic device, computer-readable medium, and computer program product based on a hybrid model according to the present application obtain a data set from a database through an SQL query by a data analysis system based on a user instruction; determine a target model according to business scenario requirements; calculate the data set through the target model to generate a calculation result; automatically filter the calculation result; invoke a large model to perform data mining and analysis on the calculation result after filtering to generate a data mining and analysis result; and display the data mining and analysis result at the front end of the data analysis system, which can enable users to perform big data analysis and processing more efficiently and securely in an automated manner, thereby greatly improving the overall work efficiency and data insight ability.

[0024] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the present application. Brief Description of the Drawings

[0025] By referring to the accompanying drawings and describing its exemplary embodiments in detail, the above and other objectives, features, and advantages of the present application will become more apparent. The following described drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0026] Figure 1 is a flowchart of a data mining and analysis method based on a hybrid model shown according to an exemplary embodiment.

[0027] Figure 2 is a schematic diagram of a data mining and analysis method based on a hybrid model shown according to another exemplary embodiment.

[0028] Figure 3 is a schematic diagram of a data mining and analysis method based on a hybrid model shown according to another exemplary embodiment.

[0029] Figure 4 is a schematic diagram of a data mining and analysis method based on a hybrid model shown according to another exemplary embodiment.

[0030] Figure 5 is a block diagram of a data mining and analysis system based on a hybrid model shown according to an exemplary embodiment.

[0031] Figure 6 is a block diagram of an electronic device shown according to an exemplary embodiment. Detailed Description of the Specific Embodiment

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. Like reference numerals in the figures denote like or similar parts, and thus their repetitive description will be omitted.

[0033] Figure 1 FIG. is a flowchart of a data mining analysis method based on a hybrid model shown according to an exemplary embodiment. The data mining analysis method 10 based on the hybrid model at least includes steps S102 to S112.

[0034] As Figure 1 shown, in S102, the data analysis system obtains a data set from the database through an SQL query based on a user instruction. The data analysis system can provide a front-end access interface through the progressive JavaScript framework Vue3; the user selects the data set to be used in the front-end interface; executes an SQL query to obtain the data set from the database; and preprocesses the data in the data set. In the initial stage of the data mining analysis method, the data analysis system first obtains the data set required for analysis. This process includes the user submitting a request through the front-end interface, obtaining the data set from the database based on an SQL query, and preprocessing the data.

[0035] More specifically, the data analysis system can use a progressive JavaScript framework (such as Vue3) to provide a user-friendly front-end access interface, and the user can select the data set to be analyzed in the interface. The system extracts the data set from the database through an SQL query. By a database that supports complex queries (such as MySQL, PostgreSQL, or a NoSQL database), various structured and unstructured data can be effectively processed.

[0036] More specifically, preprocessing the data in the data set may include: exporting the data in the data set to CSV format; cleaning the data; and converting the data set to a Pandas data set. Before data analysis, the system preprocesses the obtained data. The preprocessing process may include: exporting to CSV format: exporting the data extracted from the database to the standard CSV file format for further processing and analysis of the data. Further, outliers, duplicate values, missing values, and noise data in the data can be removed to ensure data quality. Further still, the Pandas library in Python can be used to convert the CSV format data to a DataFrame object for further data processing and analysis operations.

[0037] Among them, before the data analysis system obtains a data set from a database through an SQL query based on a user instruction, it further includes: the user applies to access the data analysis system through a client; the reverse proxy server performs a security verification on the client; after the security verification is passed, the user is allowed to access the data analysis system. Before the user accesses the data analysis system, the client needs to perform a security verification through a reverse proxy server (such as Nginx or Apache). The reverse proxy server can perform authentication, access control, and firewall protection on requests to ensure that only authorized users can access the data analysis system. After the verification is passed, the user can access the system.

[0038] In S104, determine the target model according to the business scenario requirements. For example, determine the type of algorithm model from the model library according to the business scenario requirements; determine the target model from multiple models based on the type.

[0039] The system has a built-in library of multiple algorithm models, including various models such as linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), K-means clustering, neural networks (such as CNN, RNN), etc. According to the business scenario requirements (such as prediction, classification, clustering, etc.), select a suitable model type from the model library.

[0040] The user can first determine the model type to be used on the page provided by the front end. After determining the model type, the user can also select the target model from multiple models of the same type. For example, for time series analysis, the system may provide multiple models of the LSTM category for the user to choose; for text classification, the system may provide multiple models of the BERT category for the user to choose.

[0041] In S106, calculate the data set through the target model to generate a calculation result. For example, perform feature data screening and feature extraction on the data in the data set; call the target model through a python algorithm engine to calculate the data set to generate a calculation result.

[0042] First, the data set can be screened for features, removing irrelevant or redundant features, and only retaining the feature data closely related to the business scenario. At the same time, use feature engineering (such as standardization, normalization, encoding, dimensionality reduction, etc.) techniques to further process the data to improve the accuracy and efficiency of the model. Then, call the target model through a Python algorithm engine (such as a model framework based on TensorFlow, PyTorch) to calculate the data set after preprocessing and feature extraction to generate a calculation result.

[0043] In S108, the calculation result is automatically filtered. For example, the result value of the calculation result can be automatically filtered; for another example, the valid value of the calculation result can be automatically extracted; for another example, the data classification of the calculation result can be automatically performed; for another example, the sorting and assembly of the calculation result can be automatically performed; for another example, the data specification unification of the calculation result can be automatically performed.

[0044] In S110, a large model is retrieved to perform data mining analysis on the calculation result after the filtering process, and a data mining analysis result is generated. For example, the calculation result after the filtering process is integrated; the large model and its corresponding knowledge base are retrieved through the Java backend; the calculation result after the data integration is subjected to data mining analysis through the large model and its corresponding knowledge base; an analysis result set, analysis text content, and visualization chart are generated.

[0045] After the filtering process, the system will call a large model (such as GPT-4, BERT, XLNet, etc.) and its corresponding knowledge base to perform in-depth data mining analysis and generate a data mining analysis result.

[0046] The large model and its knowledge base are called through the Java backend service (such as Spring Boot, Quarkus). The large model uses deep learning and natural language processing (NLP) technologies to deeply analyze the integrated data.

[0047] Through the inference and analysis of the large model, the system generates a series of analysis result sets, analysis text content, and visualization charts. For example, a text summary of user purchase behavior analysis, a time series chart of market trends, etc.

[0048] Among them, integrating the calculation result after the filtering process includes: performing a business scenario analysis on the calculation result after the filtering process; it may also include: performing data visualization classification on the calculation result after the filtering process; it may also include: performing data conversion and collection on the calculation result after the filtering process. The integration process can be adjusted according to the business scenario, such as scenario analysis and data classification visualization for specific user requirements.

[0049] In S112, the data mining analysis result is displayed at the front end of the data analysis system. The analysis result set, analysis text content, and visualization charts in the data mining analysis result can be integrated to generate a data analysis report, and the data analysis report is displayed at the front end. Integrating the analysis result set, text content, and visualization charts to generate a comprehensive data analysis report. For example, a progressive JavaScript framework (such as Vue3) can be used to dynamically display the analysis report at the front end, where users can interactively browse, filter, and export the analysis results, supporting the export of reports in formats such as PDF and Excel.

[0050] According to the data mining analysis method based on a hybrid model of the present application, a data set is obtained from a database through SQL query by a data analysis system based on a user instruction; a target model is determined according to business scenario requirements; the data set is calculated by the target model to generate a calculation result; the calculation result is automatically filtered; a large model is invoked to perform data mining analysis on the calculation result after filtering to generate a data mining analysis result; the manner of displaying the data mining analysis result at the front end of the data analysis system can enable users to perform big data analysis and processing more efficiently and securely in an automated manner, thus greatly improving the overall work efficiency and data insight ability.

[0051] According to the data mining analysis method based on a hybrid model of the present application, the data mining analysis method based on a hybrid model integrates multiple technical steps such as SQL data query, feature extraction, machine learning model calculation, and large model data mining analysis, and can efficiently and accurately handle data analysis requirements in a big data environment. This method greatly improves the automation and intelligence of data analysis, helps users extract valuable information from massive data, optimize decision support, and enhance enterprise competitiveness.

[0052] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teachings of the content disclosed in the present application, these principles can be applied to many other embodiments.

[0053] Figure 2 It is a schematic diagram of a data mining analysis method based on a hybrid model shown according to another exemplary embodiment. Figure 2 The system architecture of the system of the present application is shown. As Figure 2 shown, the system architecture of the present application includes a client, a front-end service container, a server container, and a virtual machine. Among them, the front-end service container may include multiple servers for resolving the front-end and back-end interface domain names, and the virtual machine and the server container are used to respond to the user's processing requests and invoke models to process actual calculations. More specifically, in an actual embodiment of the present application, the configuration in the system architecture may be as follows:

[0054]

[0055] The present application provides a data analysis platform integrating multiple models and algorithms. This platform integrates a variety of advanced analysis techniques and algorithms, including but not limited to LLM AI hybrid analysis, model algorithm analysis, big data analysis, big data mining, correlation analysis, model weight analysis, IV (Information Value) analysis, WOE (Weight of Evidence) analysis, KS (Kolmogorov - Smirnov) algorithm, data insight, data anomaly detection, variance binning calculation, box plot anomaly detection, data overview, and big data batch analysis, etc.

[0056] By using hybrid AI technology and large language models (LLMs) to comprehensively analyze and summarize data, this platform can effectively obtain accurate algorithm results. At the same time, the platform utilizes the analysis and summary capabilities of AI to greatly accelerate the data processing process and avoid the randomness problem of AI models. The platform can also automatically generate summary analysis reports based on the AI large model, providing users with in - depth data insights and actionable analysis results.

[0057] This platform gives full play to the advantages of various analysis techniques, enabling users to quickly and efficiently extract valuable information from massive data and then make informed decisions.

[0058] Figure 3 It is a schematic diagram of a data mining and analysis method based on a hybrid model shown according to another exemplary embodiment. Figure 3 It shows the call relationship between the data of the system of the present application. In the actual application process, users access the system through the self - service query platform provided by the system. After accessing the system, the AI data analysis platform supports the computing requirements of users. The AI data analysis platform has a front - end algorithm platform, a back - end algorithm platform, a Python algorithm engine, and a data source reading module built - in. The AI data analysis platform is connected to a database, and the database can include: MySQL database, Redis database. The AI data analysis platform can call the MySQL database through parameter configuration and can also cache data into the Redis database.

[0059] The AI data analysis platform can also be connected to a data set. The data set can include data sets of Doris type and SCV type. The AI data analysis platform can directly call the above - mentioned data sets.

[0060] The AI data analysis platform can also call the large model system through a specific port. The large model system includes a GPT model and its corresponding knowledge base. In the present application, the knowledge base can be created separately according to different business characteristics to make the analysis of the large model more accurate.

[0061] The system platform provided by this application can offer the following solutions:

[0062] Solve complex data processing procedures: Users do not need to possess programming skills. They only need to simply select the data range on the platform to perform complex data analysis, which is easy to operate and user-friendly.

[0063] Provide accurate results: Through comprehensive analysis using a hybrid algorithm and an AI large language model (LLM), the platform can provide stable and consistent analysis results, avoiding the randomness problem in model calculations.

[0064] Support larger data volumes: Through the data query shunting function, combined with LLM AI hybrid analysis and big data technology, the platform can efficiently process and analyze massive data, meet the requirements of real-time data analysis, solve the problem of slow processing speed of traditional statistical tools, and can achieve a computing power of up to one million levels, breaking through the computing limitations of traditional tools such as Excel.

[0065] Automate data analysis: The platform automatically completes data cleaning, filtering, classification, and preparation work, and generates in-depth data insights and analysis reports, greatly reducing manual intervention and time costs, and solving the problems of high complexity and low automation degree of existing tools.

[0066] Data security and simplified operations: Data can remain within the relevant data source platform, ensuring data security; at the same time, the platform simplifies the data processing process, reducing the steps of data cleaning, code writing, manual filtering, and extraction that users need to perform, and solving the problem of cumbersome data preparation in BI tools.

[0067] Batch processing: The platform supports batch processing of big data, significantly reducing the time for data processing, improving work efficiency, and solving the problem of the lack of automated analysis in existing tools.

[0068] Source data processing, calculation, and visualization: The platform integrates functions of source data processing, calculation, visual display, and report generation and export. Users can easily complete the entire workflow from data processing to result display, solving the problems of limited analysis depth and insufficient automated analysis in traditional BI tools.

[0069] Generally speaking, the platform of this application provides users with an efficient, secure, and easy-to-use comprehensive data analysis solution by integrating various advanced analysis technologies and automated processing functions, enabling it to quickly extract valuable information from massive data and enhance decision-making support capabilities.

[0070] Figure 4 It is a schematic diagram of a data mining and analysis method based on a hybrid model shown according to another exemplary embodiment.Figure 4 It shows the data analysis algorithm processing flow of the system of this application.

[0071] In the actual application process, execute an SQL query operation according to the user's instruction to obtain the source data. After that, export the source data as data in CSV format, and then, through processes such as data cleaning, desensitization, filling in blanks, and numerical standardization, convert it into data in a Pandas dataset.

[0072] The system can perform feature screening on the dataset, screen out irrelevant or redundant features, and only retain the feature data closely related to the business scenario. At the same time, use feature engineering (such as standardization, normalization, encoding, dimensionality reduction, etc.) techniques to further process the data to improve the accuracy and efficiency of the model.

[0073] After that, the target model selected by the user can be called to calculate the data in the dataset. After generating the preliminary calculation results, these results can be automatically filtered to extract valuable information for subsequent data mining and analysis. For example, remove the calculation results that do not meet specific conditions or thresholds. For example, extract the valid values in the calculation results, exclude noise and outliers. For example, group the calculation results according to specific classification criteria, such as by time, region, category, etc. For example, sort and assemble the results for further analysis. For example, perform standardization processing on the calculation results to unify the data format and specifications and ensure data consistency.

[0074] After that, the analysis results can be integrated, and the large model can be called to continue the calculation to generate the final analysis report. The analysis report can include contents such as the analysis result set, analysis text feedback, visualization charts, and so on.

[0075] In short, this application provides an efficient, accurate, and automated data analysis solution. The following are the advantages of this platform in multiple aspects:

[0076] Improve the data processing speed and efficiency, and solve the speed bottleneck of traditional tools**: By optimizing data query shunting and big data technology, the platform can efficiently process and analyze large-scale data to meet the needs of real-time data analysis. Compared with traditional statistical tools, the platform can easily handle data volume calculations in the millions. By converting the data into CSV files for analysis, the frequent queries to the database are reduced, and the response speed and performance of the database are improved.

[0077] Reducing the usage complexity, the platform enables code-free computing. Users do not need to have complex statistical and programming skills. They only need to simply select the data range to perform complex data analysis. This greatly lowers the threshold for ordinary users to use advanced analysis techniques. The platform automatically completes data cleaning, filtering, classification, and preparation work, simplifies the data preprocessing process, enables users to avoid writing complex cleaning code, improves development efficiency, and reduces human errors.

[0078] Automating the analysis process, the platform automatically executes data processing and analysis tasks, including data cleaning, feature extraction, analytical calculations, and visual display, significantly reducing users' manual operations and enhancing the overall work efficiency. The platform supports batch processing of big data, automatically extracts valid data results, reduces manual processing steps, simplifies the operation process, and enhances the analysis efficiency.

[0079] Enhancing the depth and breadth of analysis, by integrating large language models (LLMs) such as GPT models and their corresponding knowledge bases, the platform can provide deeper analysis and data mining capabilities than existing BI tools, avoiding the randomness problem of AI model calculations and ensuring the stability and consistency of results. The platform supports complex multi-dimensional data analysis, such as business growth analysis, risk control, market trend analysis, etc., helping users to deeply understand data from different perspectives and discover potential patterns and anomalies.

[0080] Simplifying data preparation work, the platform can automatically extract the valid value range of data, complete data preparation work such as cleaning, filtering, and classification, reducing users' operation steps and time costs, and solving the problem of cumbersome data preparation in BI tools.

[0081] Providing comprehensive visual display, the platform can automatically generate easy-to-understand visual charts to help users quickly and intuitively understand the analysis results and improve data insight capabilities. Users do not need to process the data again, and the platform automatically generates visual results, shortening the analysis time. The platform supports generating detailed analysis reports and can be exported in PDF and CSV formats for easy sharing by users and decision support.

[0082] Ensuring data security, the platform completes analysis operations within the data source, avoiding the risk of data leakage and ensuring data security. The system automatically performs desensitization and security processing on sensitive data to ensure data privacy and compliance.

[0083] Diversified application scenarios, the platform is widely used in scenarios such as risk control, market growth, sales analysis, model modeling, and promotion channel decision-making, supporting each department to optimize strategies and improve business efficiency and decision-making quality.

[0084] Those skilled in the art can understand that all or part of the steps for implementing the above embodiments are realized as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided in this application are executed. The program can be stored in a computer-readable storage medium, which can be a read-only memory, a magnetic disk, an optical disk, etc.

[0085] In addition, it should be noted that the above drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of this application, rather than for limiting purposes. It is easy to understand that the processes shown in the above drawings do not indicate or limit the chronological order of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0086] The following is the device embodiment of this application, which can be used to execute the method embodiment of this application. For details not disclosed in the device embodiment of this application, please refer to the method embodiment of this application.

[0087] Figure 5 is a block diagram of a data mining and analysis system based on a hybrid model shown according to an exemplary embodiment. As Figure 5 shown, the data mining and analysis system 50 based on the hybrid model includes: a data module 502, a front-end module 504, a model calculation module 506, a data processing module 508, a large model module 510, and a display module 512.

[0088] The data module 502 is used to obtain a data set by SQL query from the database based on a user instruction; the data module 502 is also used to provide a front-end access interface for the data analysis system through the progressive JavaScript framework Vue3; the user selects the data set to be used in the front-end interface; executes an SQL query to obtain the data set from the database; and preprocesses the data in the data set.

[0089] The front-end module 504 is used to determine a target model according to the requirements of the business scenario; the front-end module 504 is also used to determine the type of algorithm model from the model library according to the requirements of the business scenario; and determine the target model from multiple models based on the type.

[0090] The model calculation module 506 is used to calculate the data set through the target model to generate a calculation result; the model calculation module 506 is also used to screen and extract feature data from the data in the data set; and call the target model through the python algorithm engine to calculate the data set to generate a calculation result.

[0091] The data processing module 508 is used to automatically filter the calculation results; the data processing module 508 is also used to automatically filter the result values of the calculation results; the data processing module 508 is also used to automatically extract the valid values of the calculation results; the data processing module 508 is also used to automatically classify the calculation results; the data processing module 508 is also used to automatically sort and assemble the calculation results; the data processing module 508 is also used to automatically unify the data specifications of the calculation results.

[0092] The large model module 510 is used to retrieve the large model to perform data mining analysis on the calculation results after filtering, and generate data mining analysis results; the large model module 510 is also used to integrate the calculation results after filtering; the large model and its corresponding knowledge base are retrieved through the java backend; data mining analysis is performed on the calculation results after data integration through the large model and its corresponding knowledge base; an analysis result set, analysis text content, and visualization chart are generated.

[0093] The display module 512 is used to display the data mining analysis results on the front end of the data analysis system. The display module 512 is also used to integrate the analysis result set, analysis text content, and visualization chart in the data mining analysis results to generate a data analysis report; the data analysis report is displayed on the front end.

[0094] According to the data mining analysis system based on the hybrid model of the present application, a data set is obtained by SQL query from the database through the data analysis system based on a user instruction; a target model is determined according to the business scenario requirements; the data set is calculated through the target model to generate calculation results; the calculation results are automatically filtered; the large model is retrieved to perform data mining analysis on the calculation results after filtering to generate data mining analysis results; the data mining analysis results are displayed on the front end of the data analysis system, which can enable users to perform big data analysis and processing more efficiently and safely, thereby greatly improving the overall work efficiency and data insight ability.

[0095] As Figure 6 shown, an embodiment of the present application provides an electronic device, including a processor 610, a memory 620, and a bus. Among them, the processor 610 and the memory 620 complete communication with each other through the bus 640;

[0096] The memory 620 is used to store computer programs;

[0097] When the processor 610 is used to execute the program stored on the memory 620, it implements the data mining analysis method based on the hybrid model in any of the above embodiments.

[0098] The communication interface 620 is used for communication between the above-mentioned electronic device and other devices.

[0099] The memory 620 may include a random access memory 620 (Random Access Memory, abbreviated as RAM), or may also include a non-volatile memory 620 (non-volatile memory), such as at least one disk memory 620. Optionally, the memory 620 may also be at least one storage device located far from the aforementioned processor 610.

[0100] If the above method in this application is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned embodiment methods of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc.

[0101] The embodiments of this application provide a computer-readable storage medium. The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data mining and analysis method based on a hybrid model in any of the above embodiments. For example, the data analysis system obtains a data set from a database through an SQL query based on a user instruction; determines a target model according to business scenario requirements; calculates the data set through the target model to generate a calculation result; automatically filters the calculation result; invokes a large model to perform data mining and analysis on the calculation result after filtering to generate a data mining and analysis result; and displays the data mining and analysis result at the front end of the data analysis system.

[0102] The above specifically shows and describes the exemplary embodiments of this application. It should be understood that this application is not limited to the detailed structures, setting methods or implementation methods described here; on the contrary, this application is intended to cover various modifications and equivalent settings included within the spirit and scope of the appended claims.

Claims

1. A data mining and analysis system based on a hybrid model, characterized in that It includes: A data module for obtaining a data set from a database through an SQL query based on a user instruction; A front-end module for determining a target model according to the requirements of the business scenario; A model calculation module for calculating the data set through the target model to generate a calculation result; A data processing module for automatically filtering the calculation result; A large model module for invoking a large model to perform data mining analysis on the calculation result after filtering processing to generate a data mining analysis result; A display module for displaying the data mining analysis result at the front end of the data analysis system.

2. A data mining and analysis method based on a hybrid model, characterized in that, It includes: The data analysis system obtains a data set from a database through an SQL query based on a user instruction; Determine a target model according to the requirements of the business scenario; Calculate the data set through the target model to generate a calculation result; Automatically filter the calculation result; Invoke a large model to perform data mining analysis on the calculation result after filtering processing to generate a data mining analysis result; Display the data mining analysis result at the front end of the data analysis system.

3. The method according to claim 2, characterized in that, It also includes: The user applies to access the data analysis system through a client; Perform security verification on the client based on a reverse proxy server; After the security verification is passed, allow the user to access the data analysis system.

4. The method according to claim 2, wherein The data analysis system obtains a data set from a database through an SQL query based on a user instruction, including: The data analysis system provides a front-end access interface through the progressive JavaScript framework Vue3; The user selects the data set to be used in the front-end interface; Execute an SQL query to obtain a data set from the database; Preprocess the data in the data set.

5. The method according to claim 4, characterized in that, Preprocessing the data in the data set includes: Export the data in the data set to the CSV format; Perform data cleaning on the data; Convert the data set into a Pandas data set.

6. The method according to claim 2, wherein Determining a target model according to the requirements of the business scenario includes: Determine the type of algorithm model from the model library according to the requirements of the business scenario; Determine the target model from multiple models based on the type.

7. The method according to claim 2, characterized in that, Calculating the data set through the target model to generate a calculation result includes: Perform feature data screening and feature extraction on the data in the data set; Invoke the target model through a python algorithm engine to calculate the data set to generate a calculation result.

8. The method according to claim 2, wherein Automatically filtering the calculation result includes: Automatically perform result value filtering on the calculation result; and / or Automatically perform valid value extraction on the calculation result; and / or Automatically perform data classification on the calculation result; and / or Automatically perform sorting and assembly on the calculation result; and / or Automatically perform data specification unification on the calculation result.

9. The method according to claim 2, wherein Invoking a large model to perform data mining analysis on the calculation result after filtering processing to generate a data mining analysis result includes: Integrate the calculation result after filtering processing; Invoke the large model and its corresponding knowledge base through the java backend; Perform data mining and analysis on the calculation results after integrating data through a large model and its corresponding knowledge base; Generate an analysis result set, analyze text content, and generate visualization charts.

10. The method according to claim 9, wherein Integrate the calculation results after filtering, including: Perform business scenario analysis on the calculation results after filtering; and / or Perform data visualization classification on the calculation results after filtering; and / or Perform data conversion and collection on the calculation results after filtering.

11. The method according to claim 2, wherein Display the data mining and analysis results on the front end of the data analysis system, including: Integrate the analysis result set, analysis text content, and visualization charts in the data mining and analysis results to generate a data analysis report; Display the data analysis report on the front end.

12. An electronic device, characterized in that, Including: One or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 2 to 11.

13. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 2 to 11.

14. A computer program product, characterized in that, Including computer programs / instructions, and when the computer programs / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 11.

Images