An intelligent analysis method based on unstructured mass data

Abstract

The application discloses an intelligent analysis method based on unstructured mass data, which sets a task target and a database according to the task target, collects files into the database by a data collector, sets a standard template, converts all files into a unified text format according to the standard template by a data conversion module and stores the files into the database, sets a clustering model to perform clustering analysis on the text to obtain a theme type label of the text, adjusts the type generated by the machine clustering by an expert, then sets a classification model to classify the text according to the adjusted type, and performs correlation analysis on the same classification according to time nodes, so that the evolution of a concept along a time line can be effectively found.

Description

Technical Field

[0001] This invention relates to the field of computer science, and more specifically to an intelligent analysis method based on unstructured massive data. Background Technology

[0002] In the era of big data, massive amounts of text data are constantly being generated online, such as emails, web pages, documents, and voice recordings. To utilize this existing data, we need to analyze and process the vast amounts of collected information more comprehensively to extract valuable insights. Currently, the raw data generated is increasingly unstructured. Before using this raw data, it is mainly processed manually through labeling and cleaning. However, with the increasing volume of information, relying solely on manual processing is problematic. Firstly, repetitive work is prone to errors. Secondly, with the increasing volume of data, the efficiency of manual labeling and cleaning in the initial stages is also low, easily leading to errors. Therefore, how to utilize machine algorithms to assist humans in efficiently analyzing and refining this raw data remains a problem that needs to be solved in this field. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes an intelligent analysis method based on unstructured massive data. The specific technical solution is as follows:

[0004] An intelligent analysis method based on unstructured massive data, characterized by:

[0005] Includes the following steps:

[0006] S1: Set task objectives, and establish a database for the task objectives;

[0007] S2: The data collector gathers the files into the database;

[0008] S3: It has a standard template set up. The data conversion module converts all files into a unified text format according to the standard template and stores them in the database.

[0009] S4: The data processing module creates a preliminary term-text matrix for the text in the database, where the rows of the matrix represent text and the columns of the matrix represent terms. The data processing module then sends the term-text matrix to the review end.

[0010] S5: Experts examine the terminology-text matrix through the review process, removing interfering words, reducing the matrix dimension, and obtaining the final terminology-text matrix;

[0011] S6: A clustering model is set up. The data processing module calls the clustering model to perform clustering analysis on the text and obtain preliminary topic tags. The data processing module then sends the topic tags to the review end.

[0012] S7: Experts evaluate the topic tags. After experts manually evaluate and add or delete topic tags, the category tags are saved to the database through the review process.

[0013] S8: The data processing module uses a text set labeled with classification tags as training data to train a classification model;

[0014] S9: The data processing module calls the trained classification model to classify the text, assigning the text to the corresponding labels to obtain a collection of classified files;

[0015] S10: For the same set of files in a category, the data processing module selects file sets from different time points to obtain multiple sets of files;

[0016] S11: The data processing module performs conceptual association analysis on the file content in each set of files;

[0017] S12: The data processing module organizes key concept groups under the same category into reports according to time nodes and sends them to the review end;

[0018] S13: The reviewer will present the report in a visual format, making it easy to observe key concepts and their development trends within the same category.

[0019] 2. The intelligent analysis method based on unstructured massive data is characterized in that: the clustering model adopts the K-means clustering algorithm.

[0020] 3. The intelligent analysis method based on unstructured massive data is characterized in that: the data collector is a web crawler.

[0021] 4. The intelligent analysis method based on unstructured massive data, characterized in that: the collected data includes text files, XML files, emails, web pages, and voice recordings.

[0022] 5. The intelligent analysis method based on unstructured massive data is characterized in that: the speech is converted into a text script by a speech recognition model and input into the database.

[0023] The beneficial effects of this invention are as follows: This invention sets up a standard template that can convert unstructured text into a structured template; it sets up a clustering model to perform cluster analysis on the text to obtain the topic type labels of the text; experts make manual adjustments to the types generated by machine clustering; then the set up classification model classifies the text according to the adjusted types; and under the same category, it performs correlation analysis according to time nodes, effectively discovering the evolution of concepts over time. Attached Figure Description

[0024] Figure 1This is a flowchart of the process of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] like Figure 1 As shown:

[0027] An intelligent analysis method based on unstructured massive data.

[0028] Includes the following steps:

[0029] S1: Set task objectives, and establish a database for the task objectives;

[0030] S2: The data collector collects files into a database. In this embodiment, the collected data includes text files, XML files, emails, web pages, and voice recordings.

[0031] S3: It has a standard template set up. The data conversion module converts all files into a unified text format according to the standard template and stores them in the database.

[0032] S4: The data processing module creates a preliminary term-text matrix for the text in the database, where the rows of the matrix represent text and the columns of the matrix represent terms. The data processing module then sends the term-text matrix to the review end.

[0033] S5: Experts examine the terminology-text matrix through the review process, removing interfering words, reducing the matrix dimension, and obtaining the final terminology-text matrix;

[0034] S6: A clustering model is set up, wherein, in this embodiment, the clustering model adopts the K-means clustering algorithm.

[0035] The data processing module calls the clustering model to perform cluster analysis on the text, obtains preliminary topic tags, and then sends the topic tags to the review end.

[0036] S7: Experts evaluate the topic tags. After experts manually evaluate and add or delete topic tags, the category tags are saved to the database through the review process.

[0037] S8: The data processing module uses a text set labeled with classification tags as training data to train a classification model;

[0038] S9: The data processing module calls the trained classification model to classify the text, assigning the text to the corresponding labels to obtain a collection of classified files;

[0039] S10: For the same set of files in a category, the data processing module selects file sets from different time points to obtain multiple sets of files;

[0040] S11: The data processing module performs conceptual association analysis on the file content in each set of files;

[0041] S12: The data processing module organizes key concept groups under the same category into reports according to time nodes and sends them to the review end;

[0042] S13: The reviewer will present the report in a visual format, making it easy to observe key concepts and their development trends within the same category.

[0043] This embodiment can also analyze speech, converting it into a text script through a speech recognition model and inputting it into a database.

[0044] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0045] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An intelligent analysis method based on unstructured massive data, characterized in that: Includes the following steps: S1: Set task objectives, and establish a database for the task objectives; S2: The data collector gathers the files into the database; S3: It has a standard template set up. The data conversion module converts all files into a unified text format according to the standard template and stores them in the database. S4: The data processing module creates a preliminary term-text matrix for the text in the database, where the rows of the matrix represent text and the columns of the matrix represent terms. The data processing module then sends the term-text matrix to the review end. S5: Experts examine the terminology-text matrix through the review process, removing interfering words, reducing the matrix dimension, and obtaining the final terminology-text matrix; S6: A clustering model is set up. The data processing module calls the clustering model to perform clustering analysis on the text and obtain preliminary topic tags. The data processing module then sends the topic tags to the review end. S7: Experts evaluate the topic tags. After experts manually evaluate and add or delete topic tags, the category tags are saved to the database through the review process. S8: The data processing module uses a text set labeled with classification tags as training data to train a classification model; S9: The data processing module calls the trained classification model to classify the text, assigning the text to the corresponding labels to obtain a collection of classified files; S10: For the same set of files in a category, the data processing module selects file sets from different time points to obtain multiple sets of files; S11: The data processing module performs conceptual association analysis on the file content in each set of files; S12: The data processing module organizes key concept groups under the same category into reports according to time nodes and sends them to the review end; S13: The reviewer will present the report in a visual format, making it easy to observe key concepts and their development trends within the same category.

2. The intelligent analysis method based on unstructured massive data according to claim 1, characterized in that: The clustering model uses the K-means clustering algorithm.

3. The intelligent analysis method based on unstructured massive data according to claim 2, characterized in that: The data collector is a web crawler.

4. The intelligent analysis method based on unstructured massive data according to claim 3, characterized in that: The collected data includes text files, XML files, emails, web pages, and voice recordings.

5. The intelligent analysis method based on unstructured massive data according to claim 4, characterized in that: The speech is converted into a text script by a speech recognition model and input into the database.

Images