Text similarity analysis method and system based on cosine similarity and intersection-union ratio

Abstract

The invention discloses a text similarity analysis method and system based on cosine similarity and intersection-parallel ratio, relates to the technical field of big data analysis, and aims to solve the problems of low efficiency and poor accuracy of a traditional data comparison method. Screening business-related effective keywords to form a keyword set corresponding to each group of data; each group of preprocessed data forms an independent data set and is uniquely associated with a keyword set; on the basis of the key field text and the keyword sets, TF-IDF matrix quantification text features are constructed through word segmentation, and an average vector of each data set is calculated and cached for subsequent semantic similarity comparison; a comprehensive scoring mechanism is constructed by combining cosine similarity and a Jaccard intersection-to-union ratio, similar associated data sets are screened, high-similarity sets are generated and stored in batches, and data support is provided for visual display and deep analysis. According to the invention, intelligent matching and correlation analysis of different data texts can be realized.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, specifically to a text similarity analysis method and system based on cosine similarity and intersection-union ratio. Background Technology

[0002] In traditional data comparison systems, the comparison of raw data typically employs methods such as fuzzy string matching and fixed field comparison, which are insufficient to handle complex and highly unstructured data. For example:

[0003] 1. The text contains a large amount of natural language description, which cannot be directly quantified and compared;

[0004] 2. Data from the same data source naturally exhibits a high degree of similarity;

[0005] 3. Modern data often uses multiple methods to conceal itself, requiring accurate identification of its behavioral patterns.

[0006] 4. Existing technologies lack modeling and similarity assessment mechanisms for the overall semantic level of text, resulting in low recognition accuracy and high manual costs.

[0007] Therefore, how to break through the limitations of traditional data comparison technology and build an intelligent comparison mechanism that can deeply understand semantic connotations, accurately identify similar patterns, and effectively deal with data cover-up behavior has become an urgent technical problem to be solved in the current data processing field. Summary of the Invention

[0008] In traditional data comparison systems, the comparison of raw data usually adopts methods such as fuzzy string matching and fixed field comparison, which are difficult to deal with the problems of complex content and high degree of unstructuredness. Based on this, the present invention provides a text similarity analysis method and system based on cosine similarity and intersection-union ratio.

[0009] Firstly, the present invention provides a text similarity analysis method based on cosine similarity and intersection-union ratio, and the technical solution adopted to solve the above-mentioned technical problem is as follows:

[0010] A text similarity analysis method based on cosine similarity and intersection-union ratio includes the following steps:

[0011] S1. Obtain multiple sets of raw data, extract the key field text of each set of raw data, clean and filter the extracted key field text, and select effective keywords related to the business scenario to form a set of effective keywords corresponding to each set of raw data; wherein, each set of raw data constitutes an independent dataset after preprocessing, and each dataset uniquely corresponds to a set of effective keywords.

[0012] S2. Based on the key field text and effective keyword set corresponding to each dataset, construct a TF-IDF matrix through word segmentation to quantify text features; calculate the average vector of each dataset and cache it to provide a standardized vector representation for subsequent semantic similarity comparison between datasets;

[0013] S3. Based on the average vector and text features of each dataset, a comprehensive scoring mechanism is constructed through multi-dimensional calculation of cosine similarity and Jaccard intersection-union ratio to screen out datasets with similar associations, providing a quantitative basis for accurate comparison analysis.

[0014] S4. Based on the selected datasets with similar relationships, further generate a high text similarity set and store it in batches in the database to provide data support for subsequent visualization and in-depth analysis.

[0015] Optionally, step S1 specifically includes:

[0016] S1.1. Through data access interfaces or database query tools, batch acquire multiple sets of raw data information to be compared, covering different data forms such as text and structured records, record the data source and collection time, ensure the integrity, comprehensiveness and traceability of the data source, and provide a complete data foundation for subsequent processing;

[0017] S1.2 For each set of raw data information, based on specific business needs and data characteristics, accurately locate and extract key field texts that can reflect the core meaning of the data;

[0018] S1.3. For the key field text corresponding to each set of original data information, perform the following operations in sequence to complete data cleaning and effective word selection: Filter out redundant words in the key field text that do not affect semantic expression through a preset stop word list; remove garbled characters, special symbols, repeated characters, and meaningless words that are irrelevant to the current business scenario from the key field text; based on business rules and semantic analysis, select effective keywords that can accurately reflect the core behavior, attributes, or themes of the data, thereby forming a set of effective keywords corresponding to each set of original data.

[0019] Further optionally, step S2 specifically includes:

[0020] S2.1 Based on the effective keyword set obtained through screening, a word segmentation tool is used to segment the key field text corresponding to each dataset, converting the text into a structured sequence of terms; based on the word segmentation results, a TF-IDF matrix is ​​constructed. By quantifying the term frequency (TF) of a term in a single dataset and the inverse document frequency (IDF) in the overall dataset, the unstructured text is transformed into a computable numerical feature matrix, thus completing the mathematical representation of the text.

[0021] S2.2 Calculate the mean of the row vectors of the TF-IDF matrix corresponding to each dataset to obtain the average vector of each dataset;

[0022] S2.3 Store the average vector of each dataset in the cache space.

[0023] Further optionally, step S3 specifically includes:

[0024] S3.1 For text data with an inverse document frequency (IDF) lower than a preset value, calculate the cosine similarity between the vectors of the two datasets to be compared based on the average vector of the corresponding dataset, and quantify the semantic similarity.

[0025] S3.2 For effective keywords with inverse document frequency (IDF) higher than or equal to a preset value, calculate the Jaccard intersection-union ratio between the two sets based on the effective keyword set corresponding to each dataset to quantify the overlap at the keyword level.

[0026] S3.3 Construct a comprehensive scoring mechanism, combine cosine similarity and Jaccard intersection-union ratio with preset weights to calculate the final similarity score, quantify the overall similarity level of the two datasets; based on the comprehensive score, select datasets with similarity greater than a preset threshold to form a candidate dataset with similar associations.

[0027] Further optionally, step S4 specifically includes:

[0028] S4.1. Perform secondary verification on the selected datasets with similar relationships, and select the high text similarity set that meets the business requirements according to the preset similarity threshold, and clearly mark the similarity relationship and core similarity features between datasets in the high text similarity set.

[0029] S4.2. The generated high text similarity set and its associated information are written into the database through the batch insertion interface to establish a structured data storage table. The database storage supports data classification according to business scenarios and time periods, providing a standardized data source that can be directly called for subsequent visualization of comparison results and association rule mining.

[0030] Secondly, the present invention provides a text similarity analysis system based on cosine similarity and intersection-union ratio, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0031] A text similarity analysis system based on cosine similarity and intersection-union ratio, comprising:

[0032] The data acquisition and preprocessing module is used to acquire multiple sets of raw data, extract the key field text of each set of raw data, clean and filter the extracted key field text, and select effective keywords related to the business scenario, thereby forming a set of effective keywords corresponding to each set of raw data. Each set of raw data constitutes an independent dataset after preprocessing, and each dataset uniquely corresponds to a set of effective keywords.

[0033] The matrix construction and mean calculation module is used to construct a TF-IDF matrix to quantify text features based on the key field text and effective keyword set corresponding to each dataset through word segmentation; calculate the mean vector of each dataset and cache it to provide a standardized vector representation for subsequent semantic similarity comparison between datasets;

[0034] The similarity calculation and filtering module is used to construct a comprehensive scoring mechanism based on the average vector and text features of each dataset, through multi-dimensional calculation of cosine similarity and Jaccard intersection-union ratio, to filter out datasets with similar associations and provide quantitative basis for accurate comparison analysis.

[0035] The filtering and batch storage module is used to further generate high text similarity sets based on the filtered datasets with similar relationships, and to store them in batches in the database, providing data support for subsequent visualization and in-depth analysis.

[0036] Optionally, the data acquisition and preprocessing modules involved specifically include:

[0037] The data batch acquisition unit is used to acquire multiple sets of raw data information to be compared in batches through data access interfaces or database query tools. It covers different data forms such as text and structured records, records the data source and collection time, ensures the integrity, comprehensiveness and traceability of the data source, and provides a complete data foundation for subsequent processing.

[0038] The key field extraction unit is used to accurately locate and extract key field text that reflects the core meaning of the data for each set of raw data information, based on specific business needs and data characteristics.

[0039] The text preprocessing unit performs the following operations sequentially on the key field text corresponding to each set of original data information to complete data cleaning and effective word selection: It filters out redundant words in the key field text that do not affect semantic expression using a preset stop word list; it removes garbled characters, special symbols, repeated characters, and meaningless words unrelated to the current business scenario from the key field text; and based on business rules and semantic analysis, it selects effective keywords that accurately reflect the core behavior, attributes, or themes of the data, thereby forming a set of effective keywords corresponding to each set of original data.

[0040] Optionally, the matrix construction and mean calculation modules involved specifically include:

[0041] The word segmentation unit is used to segment the key field text corresponding to each dataset based on the filtered set of effective keywords, and convert the text into a structured sequence of terms.

[0042] The matrix construction unit is used to construct a TF-IDF matrix based on the word segmentation results. By quantifying the term frequency (TF) of a term in a single dataset and the inverse document frequency (IDF) in the overall dataset, it realizes the transformation of unstructured text into a computable numerical feature matrix and completes the mathematical representation of the text.

[0043] The mean calculation unit is used to calculate the mean of the row vectors of the TF-IDF matrix corresponding to each dataset, so as to obtain the mean vector of each dataset.

[0044] The mean storage unit is used to store the mean vector of each dataset into the cache space.

[0045] Optionally, the similarity calculation and filtering modules involved specifically include:

[0046] The setting unit is used to set a preset value for the Inverse Document Frequency (IDF).

[0047] The similarity calculation unit is used to calculate the cosine similarity between the vectors of the two datasets to be compared, based on the average vector of the corresponding dataset, for text data whose inverse document frequency (IDF) is lower than a preset value, thereby quantifying the semantic similarity.

[0048] The intersection-union ratio (IUU) calculation unit is used to calculate the Jaccard IUU between two sets of effective keywords with an inverse document frequency (IDF) higher than or equal to a preset value, based on the effective keyword set corresponding to each dataset, to quantify the overlap at the keyword level.

[0049] The calculation and filtering unit is used to construct a comprehensive scoring mechanism. It calculates the final similarity score by combining cosine similarity and Jaccard intersection-union ratio according to preset weights, which quantifies the overall similarity level of the two datasets. Based on the comprehensive score, the datasets with similarity greater than a preset threshold are selected to form a candidate dataset with similar associations.

[0050] Optionally, the filtering and batch storage modules involved specifically include:

[0051] A similarity setting unit is used to preset a similarity threshold;

[0052] The dataset filtering unit is used to perform secondary verification on the filtered datasets with similar relationships. Based on the preset similarity threshold, it filters out the high text similarity set that meets the business requirements and clearly marks the similarity relationship and core similarity features between datasets within the high text similarity set.

[0053] The batch storage unit is used to write the generated high text similarity set and its associated information into the database through the batch insertion interface, and establish a structured data storage table. The database storage supports data classification according to business scenarios and time periods, providing a standardized data source that can be directly called for subsequent visualization tools to display comparison results and carry out association rule mining.

[0054] The text similarity analysis method and system based on cosine similarity and intersection-union ratio of the present invention have the following advantages compared with the prior art:

[0055] This invention extracts semantic features from raw data text, constructs TF-IDF vectors and calculates the average vector, and combines this with text information with high inverse document frequency (IDF) values ​​to perform multi-dimensional similarity modeling, thereby achieving intelligent matching and association analysis of different data texts. It can be widely applied in various fields and has high precision, high scalability and real-time processing capabilities. Attached Figure Description

[0056] Appendix Figure 1 This is a flowchart of the method according to Embodiment 1 of the present invention;

[0057] Appendix Figure 2 This is a module connection block diagram of Embodiment 2 of the present invention. Detailed Implementation

[0058] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0059] Example 1:

[0060] Combined with appendix Figure 1 This embodiment proposes a text similarity analysis method based on cosine similarity and intersection-union ratio, which includes the following steps:

[0061] S1. Obtain multiple sets of raw data, extract the key field text of each set of raw data, clean and filter the extracted key field text (such as removing redundant characters, stop words, etc.), and filter out effective keywords related to the business scenario, thereby forming a set of effective keywords corresponding to each set of raw data; wherein, each set of raw data constitutes an independent dataset after preprocessing, and each dataset uniquely corresponds to a set of effective keywords.

[0062] This step specifically includes:

[0063] S1.1. Through data access interfaces or database query tools, batch acquire multiple sets of raw data information to be compared, covering different data forms such as text and structured records, record the data source and collection time, ensure the integrity, comprehensiveness and traceability of the data source, and provide a complete data foundation for subsequent processing;

[0064] S1.2 For each set of raw data information, based on specific business needs and data characteristics, accurately locate and extract key field texts that can reflect the core meaning of the data;

[0065] S1.3. For the key field text corresponding to each set of original data information, perform the following operations in sequence to complete data cleaning and effective word selection: Filter out redundant words in the key field text that do not affect semantic expression through a preset stop word list; remove garbled characters, special symbols, repeated characters, and meaningless words that are irrelevant to the current business scenario from the key field text; based on business rules and semantic analysis, select effective keywords that can accurately reflect the core behavior, attributes, or themes of the data, thereby forming a set of effective keywords corresponding to each set of original data.

[0066] S2. Based on the key field text and effective keyword set corresponding to each dataset, construct a TF-IDF matrix through word segmentation to quantify text features; calculate the average vector of each dataset and cache it to provide a standardized vector representation for subsequent semantic similarity comparison between datasets.

[0067] This step specifically includes:

[0068] S2.1 Based on the effective keyword set obtained through screening, a word segmentation tool is used to segment the key field text corresponding to each dataset, converting the text into a structured sequence of terms; based on the word segmentation results, a TF-IDF matrix is ​​constructed. By quantifying the term frequency (TF) of a term in a single dataset and the inverse document frequency (IDF) in the overall dataset, the unstructured text is transformed into a computable numerical feature matrix, thus completing the mathematical representation of the text.

[0069] S2.2 Calculate the mean of the row vectors of the TF-IDF matrix corresponding to each dataset to obtain the average vector of each dataset;

[0070] S2.3 Store the average vector of each dataset in the cache space.

[0071] S3. Based on the average vector and text features of each dataset, a comprehensive scoring mechanism is constructed through multi-dimensional calculation of cosine similarity and Jaccard intersection-union ratio to screen out datasets with similar associations, providing a quantitative basis for accurate comparison analysis.

[0072] This step specifically includes:

[0073] S3.1 For text data with an inverse document frequency (IDF) lower than a preset value, calculate the cosine similarity between the vectors of the two datasets to be compared based on the average vector of the corresponding dataset, and quantify the semantic similarity.

[0074] S3.2 For effective keywords with inverse document frequency (IDF) higher than or equal to a preset value, calculate the Jaccard intersection-union ratio between the two sets based on the effective keyword set corresponding to each dataset to quantify the overlap at the keyword level.

[0075] S3.3 Construct a comprehensive scoring mechanism, combine cosine similarity and Jaccard intersection-union ratio with preset weights to calculate the final similarity score, quantify the overall similarity level of the two datasets; based on the comprehensive score, select datasets with similarity greater than a preset threshold to form a candidate dataset with similar associations.

[0076] S4. Based on the selected datasets with similar relationships, further generate a high text similarity set and store it in batches in the database to provide data support for subsequent visualization and in-depth analysis.

[0077] This step specifically includes:

[0078] S4.1. Perform secondary verification on the selected datasets with similar relationships, and select the high text similarity set that meets the business requirements according to the preset similarity threshold. Clearly mark the similar relationship between datasets in the high text similarity set (such as one-to-one, one-to-many relationship) and core similarity features (such as common keywords, semantically overlapping segments).

[0079] S4.2. The generated high text similarity set and its associated information are written into the database through the batch insertion interface to establish a structured data storage table. The database storage supports data classification according to business scenarios and time periods, providing a standardized data source that can be directly called for subsequent visualization of comparison results and association rule mining.

[0080] It should be noted that, for the calculation steps S2 to S4, parallel computing, asynchronous processing, cache optimization, and dynamic table name management are used to improve the overall efficiency and data isolation capabilities of the system, supporting large-scale data comparison scenarios.

[0081] In step S2, during the construction of the TF-IDF matrix, the ForkJoinPool thread pool is used to split the text segmentation and vector calculation tasks in parallel. The large-scale text data is distributed to multiple sub-threads for parallel calculation according to a preset granularity, which shortens the overall time of TF-IDF vector generation and improves the processing efficiency of the feature quantization stage.

[0082] For the cosine similarity and Jaccard intersection-union comparison calculation task in step S3, asynchronous processing is achieved through a thread pool. The dataset task to be compared (i.e., the task of comparing the vectors or effective keyword sets of the two datasets) is submitted to the thread pool queue. The main thread does not need to wait for the calculation to be completed before continuing to execute the subsequent process. By decoupling the tasks, the concurrent processing capability of the system is improved and the blocking risk during large-scale data comparison is reduced.

[0083] The average vector caching results from step S2 are integrated, and intermediate results frequently called during the similarity calculation process (such as the calculated average vector of the dataset and the IDF value of commonly used terms) and high-frequency comparison results (such as recent high similarity sets) are cached and stored. The number of repeated calculations and database accesses is reduced through memory caching or distributed caching, thereby improving the system response speed.

[0084] For the database storage stage in step S4, it supports dynamically generating or specifying data table names based on business scenarios (such as different business lines and data types), achieving physical isolation storage of different types of comparison results (such as high text similarity sets and their associated information); dynamic table name management facilitates quick location of target datasets and associated information based on conditions, simplifies data query and maintenance processes, and improves the flexibility and security of data management.

[0085] Example 2:

[0086] Combined with appendix Figure 2 This embodiment proposes a text similarity analysis system based on cosine similarity and intersection-union ratio, which includes:

[0087] The data acquisition and preprocessing module is used to acquire multiple sets of raw data, extract the key field text of each set of raw data, clean and filter the extracted key field text (such as removing redundant characters, stop words, etc.), and filter out effective keywords related to the business scenario, thereby forming a set of effective keywords corresponding to each set of raw data; wherein, each set of raw data constitutes an independent dataset after preprocessing, and each dataset uniquely corresponds to a set of effective keywords.

[0088] The matrix construction and mean calculation module is used to construct a TF-IDF matrix to quantify text features based on the key field text and effective keyword set corresponding to each dataset through word segmentation; calculate the mean vector of each dataset and cache it to provide a standardized vector representation for subsequent semantic similarity comparison between datasets;

[0089] The similarity calculation and filtering module is used to construct a comprehensive scoring mechanism based on the average vector and text features of each dataset, through multi-dimensional calculation of cosine similarity and Jaccard intersection-union ratio, to filter out datasets with similar associations and provide quantitative basis for accurate comparison analysis.

[0090] The filtering and batch storage module is used to further generate high text similarity sets based on the filtered datasets with similar relationships, and to store them in batches in the database, providing data support for subsequent visualization and in-depth analysis.

[0091] In this embodiment, the data acquisition and preprocessing module specifically includes:

[0092] The data batch acquisition unit is used to acquire multiple sets of raw data information to be compared in batches through data access interfaces or database query tools. It covers different data forms such as text and structured records, records the data source and collection time, ensures the integrity, comprehensiveness and traceability of the data source, and provides a complete data foundation for subsequent processing.

[0093] The key field extraction unit is used to accurately locate and extract key field text that reflects the core meaning of the data for each set of raw data information, based on specific business needs and data characteristics.

[0094] The text preprocessing unit performs the following operations sequentially on the key field text corresponding to each set of original data information to complete data cleaning and effective word selection: It filters out redundant words in the key field text that do not affect semantic expression using a preset stop word list; it removes garbled characters, special symbols, repeated characters, and meaningless words unrelated to the current business scenario from the key field text; and based on business rules and semantic analysis, it selects effective keywords that accurately reflect the core behavior, attributes, or themes of the data, thereby forming a set of effective keywords corresponding to each set of original data.

[0095] In this embodiment, the matrix construction and mean calculation module specifically includes:

[0096] The word segmentation unit is used to segment the key field text corresponding to each dataset based on the filtered set of effective keywords, and convert the text into a structured sequence of terms.

[0097] The matrix construction unit is used to construct a TF-IDF matrix based on the word segmentation results. By quantifying the term frequency (TF) of a term in a single dataset and the inverse document frequency (IDF) in the overall dataset, it realizes the transformation of unstructured text into a computable numerical feature matrix and completes the mathematical representation of the text.

[0098] The mean calculation unit is used to calculate the mean of the row vectors of the TF-IDF matrix corresponding to each dataset, so as to obtain the mean vector of each dataset.

[0099] The mean storage unit is used to store the mean vector of each dataset into the cache space.

[0100] In this embodiment, the similarity calculation and filtering module specifically includes:

[0101] The setting unit is used to set a preset value for the Inverse Document Frequency (IDF).

[0102] The similarity calculation unit is used to calculate the cosine similarity between the vectors of the two datasets to be compared, based on the average vector of the corresponding dataset, for text data whose inverse document frequency (IDF) is lower than a preset value, thereby quantifying the semantic similarity.

[0103] The intersection-union ratio (IUU) calculation unit is used to calculate the Jaccard IUU between two sets of effective keywords with an inverse document frequency (IDF) higher than or equal to a preset value, based on the effective keyword set corresponding to each dataset, to quantify the overlap at the keyword level.

[0104] The calculation and filtering unit is used to construct a comprehensive scoring mechanism. It calculates the final similarity score by combining cosine similarity and Jaccard intersection-union ratio according to preset weights, which quantifies the overall similarity level of the two datasets. Based on the comprehensive score, the datasets with similarity greater than a preset threshold are selected to form a candidate dataset with similar associations.

[0105] In this embodiment, the filtering and batch storage module specifically includes:

[0106] A similarity setting unit is used to preset a similarity threshold;

[0107] The dataset filtering unit is used to perform secondary verification on the filtered datasets with similar relationships. Based on the preset similarity threshold, it filters out the high text similarity set that meets the business requirements and clearly marks the similarity relationship between datasets in the high text similarity set (such as one-to-one and one-to-many relationships) and core similarity features (such as common keywords and semantically overlapping segments).

[0108] The batch storage unit is used to write the generated high text similarity set and its associated information into the database through the batch insertion interface, and establish a structured data storage table. The database storage supports data classification according to business scenarios and time periods, providing a standardized data source that can be directly called for subsequent visualization tools to display comparison results and carry out association rule mining.

[0109] It should be added that the system in this embodiment also has the following optimization mechanisms:

[0110] The matrix construction and mean calculation module uses ForkJoinPool for parallel vector computation.

[0111] The similarity calculation and filtering module uses a thread pool to perform similarity calculation tasks.

[0112] The filtering and batch storage module uses ConcurrentHashMap to cache the calculated average vector; the filtering and batch storage module supports dynamic table name management to achieve controllable data isolation.

[0113] In summary, the text similarity analysis method and system based on cosine similarity and intersection-union ratio of this invention can achieve intelligent matching and association analysis of different data texts by extracting semantic features of the original data text, constructing TF-IDF vectors and calculating the average vector, and combining text information with high inverse document frequency (IDF) values ​​for multi-dimensional similarity modeling. This solves the problem that traditional data comparison systems using string fuzzy matching and fixed field comparison are difficult to handle when dealing with complex and highly unstructured content.

[0114] The above specific examples illustrate the principles and implementation methods of the present invention in detail. These embodiments are merely for the purpose of helping to understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made to the present invention by those skilled in the art without departing from the principles of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A text similarity analysis method based on cosine similarity and intersection-union ratio, characterized in that, Includes the following steps: S1. Obtain multiple sets of raw data, extract the key field text of each set of raw data, clean and filter the extracted key field text, and select effective keywords related to the business scenario to form a set of effective keywords corresponding to each set of raw data; wherein, each set of raw data constitutes an independent dataset after preprocessing, and each dataset uniquely corresponds to a set of effective keywords. S2. Based on the key field text and effective keyword set corresponding to each dataset, construct a TF-IDF matrix through word segmentation to quantify text features; calculate the average vector of each dataset and cache it to provide a standardized vector representation for subsequent semantic similarity comparison between datasets; S3. Based on the average vector and text features of each dataset, a comprehensive scoring mechanism is constructed through multi-dimensional calculation of cosine similarity and Jaccard intersection-union ratio to screen out datasets with similar associations, providing a quantitative basis for accurate comparison analysis. S4. Based on the selected datasets with similar relationships, further generate a high text similarity set and store it in batches in the database to provide data support for subsequent visualization and in-depth analysis.

2. The text similarity analysis method based on cosine similarity and intersection-union ratio according to claim 1, characterized in that, Step S1 specifically includes: S1.

1. Through data access interfaces or database query tools, batch acquire multiple sets of raw data information to be compared, covering different data forms such as text and structured records, record the data source and collection time, ensure the integrity, comprehensiveness and traceability of the data source, and provide a complete data foundation for subsequent processing; S1.2 For each set of raw data information, based on specific business needs and data characteristics, accurately locate and extract key field texts that can reflect the core meaning of the data; S1.

3. For the key field text corresponding to each set of original data information, perform the following operations in sequence to complete data cleaning and effective word selection: Filter out redundant words in the key field text that do not affect semantic expression through a preset stop word list; remove garbled characters, special symbols, repeated characters, and meaningless words that are irrelevant to the current business scenario from the key field text; based on business rules and semantic analysis, select effective keywords that can accurately reflect the core behavior, attributes, or themes of the data, thereby forming a set of effective keywords corresponding to each set of original data.

3. The text similarity analysis method based on cosine similarity and intersection-union ratio according to claim 2, characterized in that, Step S2 specifically includes: S2.1 Based on the effective keyword set obtained through screening, a word segmentation tool is used to segment the key field text corresponding to each dataset, converting the text into a structured sequence of terms; based on the word segmentation results, a TF-IDF matrix is ​​constructed. By quantifying the term frequency (TF) of a term in a single dataset and the inverse document frequency (IDF) in the overall dataset, the unstructured text is transformed into a computable numerical feature matrix, thus completing the mathematical representation of the text. S2.2 Calculate the mean of the row vectors of the TF-IDF matrix corresponding to each dataset to obtain the average vector of each dataset; S2.3 Store the average vector of each dataset in the cache space.

4. The text similarity analysis method based on cosine similarity and intersection-union ratio according to claim 3, characterized in that, Step S3 specifically includes: S3.1 For text data with an inverse document frequency (IDF) lower than a preset value, calculate the cosine similarity between the vectors of the two datasets to be compared based on the average vector of the corresponding dataset, and quantify the semantic similarity. S3.2 For effective keywords with inverse document frequency (IDF) higher than or equal to a preset value, calculate the Jaccard intersection-union ratio between the two sets based on the effective keyword set corresponding to each dataset to quantify the overlap at the keyword level. S3.3 Construct a comprehensive scoring mechanism, combine cosine similarity and Jaccard intersection-union ratio with preset weights to calculate the final similarity score, quantify the overall similarity level of the two datasets; based on the comprehensive score, select datasets with similarity greater than a preset threshold to form a candidate dataset with similar associations.

5. The text similarity analysis method based on cosine similarity and intersection-union ratio according to claim 4, characterized in that, Step S4 specifically includes: S4.

1. Perform secondary verification on the selected datasets with similar relationships, and select the high text similarity set that meets the business requirements according to the preset similarity threshold, and clearly mark the similarity relationship and core similarity features between datasets in the high text similarity set. S4.

2. The generated high text similarity set and its associated information are written into the database through the batch insertion interface to establish a structured data storage table. The database storage supports data classification according to business scenarios and time periods, providing a standardized data source that can be directly called for subsequent visualization of comparison results and association rule mining.

6. A text similarity analysis system based on cosine similarity and intersection-union ratio, characterized in that, It includes: The data acquisition and preprocessing module is used to acquire multiple sets of raw data, extract the key field text of each set of raw data, clean and filter the extracted key field text, and select effective keywords related to the business scenario, thereby forming a set of effective keywords corresponding to each set of raw data. Each set of raw data constitutes an independent dataset after preprocessing, and each dataset uniquely corresponds to a set of effective keywords. The matrix construction and mean calculation module is used to construct a TF-IDF matrix to quantify text features based on the key field text and effective keyword set corresponding to each dataset through word segmentation; calculate the mean vector of each dataset and cache it to provide a standardized vector representation for subsequent semantic similarity comparison between datasets; The similarity calculation and filtering module is used to construct a comprehensive scoring mechanism based on the average vector and text features of each dataset, through multi-dimensional calculation of cosine similarity and Jaccard intersection-union ratio, to filter out datasets with similar associations and provide quantitative basis for accurate comparison analysis. The filtering and batch storage module is used to further generate high text similarity sets based on the filtered datasets with similar relationships, and to store them in batches in the database, providing data support for subsequent visualization and in-depth analysis.

7. A text similarity analysis system based on cosine similarity and intersection-union ratio according to claim 6, characterized in that, The data acquisition and preprocessing module specifically includes: The data batch acquisition unit is used to acquire multiple sets of raw data information to be compared in batches through data access interfaces or database query tools. It covers different data forms such as text and structured records, records the data source and collection time, ensures the integrity, comprehensiveness and traceability of the data source, and provides a complete data foundation for subsequent processing. The key field extraction unit is used to accurately locate and extract key field text that reflects the core meaning of the data for each set of raw data information, based on specific business needs and data characteristics. The text preprocessing unit performs the following operations sequentially on the key field text corresponding to each set of original data information to complete data cleaning and effective word selection: It filters out redundant words in the key field text that do not affect semantic expression using a preset stop word list; it removes garbled characters, special symbols, repeated characters, and meaningless words unrelated to the current business scenario from the key field text; and based on business rules and semantic analysis, it selects effective keywords that accurately reflect the core behavior, attributes, or themes of the data, thereby forming a set of effective keywords corresponding to each set of original data.

8. A text similarity analysis system based on cosine similarity and intersection-union ratio according to claim 7, characterized in that, The matrix construction and mean calculation module specifically includes: The word segmentation unit is used to segment the key field text corresponding to each dataset based on the filtered set of effective keywords, and convert the text into a structured sequence of terms. The matrix construction unit is used to construct a TF-IDF matrix based on the word segmentation results. By quantifying the term frequency (TF) of a term in a single dataset and the inverse document frequency (IDF) in the overall dataset, it realizes the transformation of unstructured text into a computable numerical feature matrix and completes the mathematical representation of the text. The mean calculation unit is used to calculate the mean of the row vectors of the TF-IDF matrix corresponding to each dataset, so as to obtain the mean vector of each dataset. The mean storage unit is used to store the mean vector of each dataset into the cache space.

9. A text similarity analysis system based on cosine similarity and intersection-union ratio according to claim 8, characterized in that, The similarity calculation and filtering module specifically includes: The setting unit is used to set a preset value for the Inverse Document Frequency (IDF). The similarity calculation unit is used to calculate the cosine similarity between the vectors of the two datasets to be compared, based on the average vector of the corresponding dataset, for text data whose inverse document frequency (IDF) is lower than a preset value, thereby quantifying the semantic similarity. The intersection-union ratio (IUU) calculation unit is used to calculate the Jaccard IUU between two sets of effective keywords with an inverse document frequency (IDF) higher than or equal to a preset value, based on the effective keyword set corresponding to each dataset, to quantify the overlap at the keyword level. The calculation and filtering unit is used to construct a comprehensive scoring mechanism. It calculates the final similarity score by combining cosine similarity and Jaccard intersection-union ratio according to preset weights, which quantifies the overall similarity level of the two datasets. Based on the comprehensive score, the datasets with similarity greater than a preset threshold are selected to form a candidate dataset with similar associations.

10. A text similarity analysis system based on cosine similarity and intersection-union ratio according to claim 9, characterized in that, The filtering and batch storage module specifically includes: A similarity setting unit is used to preset a similarity threshold; The dataset filtering unit is used to perform secondary verification on the filtered datasets with similar relationships. Based on the preset similarity threshold, it filters out the high text similarity set that meets the business requirements and clearly marks the similarity relationship and core similarity features between datasets within the high text similarity set. The batch storage unit is used to write the generated high text similarity set and its associated information into the database through the batch insertion interface, and establish a structured data storage table. The database storage supports data classification according to business scenarios and time periods, providing a standardized data source that can be directly called for subsequent visualization tools to display comparison results and carry out association rule mining.

Images