R language-based similarity and distance calculation system and method

Abstract

The invention discloses a similarity and distance calculation system and method based on an R language, and relates to the technical field of data processing, and the key point of the technical scheme is that the system comprises the following modules: a data reading module, a data standardization module, a similarity calculation module and an error checking module. Through automatic data inspection, standardized processing and efficient similarity calculation, the user can easily perform complex similarity analysis. The method has a wide prospect in the application of big data analysis, machine learning and the like, a more convenient and more accurate similarity calculation scheme can be provided for a user, and the data processing and analysis efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and more specifically, to a system and method for similarity and distance calculation based on R language. Background Art

[0002] In the current data analysis field, the integration of big data and machine learning technologies has driven a significant demand for complex data processing capabilities. Similarity metrics and distance calculations, as the cornerstones of data analysis, are essential for understanding data patterns, trends, and anomalies. These algorithms not only play a core role in data mining and clustering analysis, but also play an important role in the training and optimization of machine learning models. They enable algorithms to identify similar patterns in data, thereby achieving efficient data classification and prediction in areas such as pattern recognition, recommendation systems, and genetic analysis.

[0003] However, although spreadsheet software such as Excel are very convenient for data management and preliminary analysis, they have limitations when handling complex data analysis tasks. Excel files often contain mixed data types, including text, dates, and numbers, which are not friendly for performing advanced similarity measures and distance calculations. In addition, Excel's performance is limited when processing large-scale data sets, which limits its application in big data analysis.

[0004] To overcome these limitations, existing similarity and distance calculation tools usually require users to convert data into pure numerical formats in advance. This preprocessing step is often time-consuming and complex, requiring users to have certain data processing skills. In addition, these tools often lack flexibility when processing high-dimensional data from Excel, especially when multiple similarities or distance metrics (such as Euclidean distance and cosine similarity) need to be calculated simultaneously. Users have to rely on multiple tools or manually write code to achieve this, which further increases the complexity of data analysis.

[0005] To address these challenges, the present invention proposes a system and method for similarity and distance calculation based on R language, which is developed based on R language and is specifically used to efficiently read and process data from Excel files and automatically perform data standardization and similarity / distance calculation. Summary of the invention

[0006] The purpose of the present invention is to provide a system and method for similarity and distance calculation based on R language to solve the deficiencies in data processing and similarity calculation in the prior art.

[0007] The above technical objectives of the present invention are achieved through the following technical solutions: A system for similarity and distance calculation based on R language includes the following modules:

[0008] The data reading module is used to read data from Excel files and convert the data format to ensure that all data is numeric. It also automatically identifies the first row as the column name and the first column as the row name, and processes non-numeric data.

[0009] The data standardization module is used to standardize the data by column so that each column of data satisfies the distribution with a mean of zero and a standard deviation of one after transformation, so as to eliminate the dimensional differences between different columns;

[0010] Similarity calculation module, used to support multiple similarity and distance calculation algorithms, including Euclidean distance and cosine similarity, and provide sorted similarity or distance matrix;

[0011] The error checking module is used to automatically check whether the data format is correct before data processing and to issue prompts for non-numeric data.

[0012] The present invention is further configured as follows: the data reading module includes a built-in function `read_ma trix_from_excel(file_path,sheet)`, which is used to read a matrix from an Excel file and check for non-digital content.

[0013] The present invention is further configured such that: the data standardization module includes a built-in function `standa rdize_matrix(mat)` for standardizing the input matrix by columns.

[0014] The present invention is further configured as follows: the similarity calculation module comprises:

[0015] Built-in function `compute_euclidean_distances(mat)`, used to calculate the Euclidean distance of a normalized matrix;

[0016] Built-in function `cosine_similarity(x,y)`, used to calculate the cosine similarity between two vectors;

[0017] Built-in function `get_sorted_euclidean_distances(dist_matrix)`, which returns the sorted Euclidean distances;

[0018] Built-in function `compute_cosine_similarities(mat)`, used to calculate the cosine similarity of a normalized matrix;

[0019] Built-in function `get_sorted_cosine_similarities(sim_matrix)`, which returns sorted cosine similarities;

[0020] Built-in function `get_top_5_comparisons(sorted_distances, sorted_similari ties)`, which returns the top 5 comparisons of distances and similarities.

[0021] The present invention is further configured as follows: the system further comprises a main function `process_excel_file(file_path,sheet)`, which is used to process the Excel file and return the results of the first five comparisons.

[0022] Another object of the present invention is to provide a method for similarity and distance calculation based on R language, comprising the following steps:

[0023] Step 1: Import Excel file and import data by specifying the file path;

[0024] Step 2: Check the data to ensure that all data are numerical;

[0025] Step 3: Data standardization: standardize the numerical data by column;

[0026] Step 4: Similarity / distance calculation, select different similarity or distance algorithms for calculation;

[0027] Step 5: Output the results, output the top N groups of similarity or distance results after sorting.

[0028] By adopting the above technical solution, the present invention proposes a similarity and distance calculation system and method based on R language. The data reading and processing module of the system of the present invention realizes the automatic data import function. Through the built-in `read_matrix_from_excel(file_path,sheet)` function, the user can directly read data from the Excel file. This module automatically identifies the first line as the column name and the first column as the row name, and checks and processes non-numeric data. This automated process greatly simplifies the user's data preprocessing work, avoids the errors that may be caused by manual data integration, and improves the efficiency and accuracy of data processing.

[0029] The data standardization module ensures that each column of data is standardized before performing similarity or distance calculations, ensuring that the mean is zero and the standard deviation is one. Through the `standardize_matrix(mat)` function, the system automatically completes this preprocessing step, avoiding the problem of data dimension differences affecting the analysis results. This not only improves the reliability of the analysis, but also saves users from the trouble of manually performing tedious data conversion.

[0030] The similarity and distance calculation module is the core function of the system, supporting multiple calculation methods such as Euclidean distance and cosine similarity. Using the functions `compute_euclidean_distances(mat)` and `compute_cosine_similarities(mat)`, the system can quickly and accurately calculate high-dimensional matrices. The calculation results are automatically sorted and returned in a clear table format for further analysis by the user. The optimization of this module ensures the ability to efficiently handle complex data sets, making it suitable for a variety of application scenarios.

[0031] To enhance the user experience, the system also includes a detailed error checking module. Before data processing, the module automatically checks whether the data format is correct. For non-numeric data, the system will issue a prompt and ask the user to correct it to avoid errors during analysis. This function ensures the accuracy of data input and the stable operation of the system, enhancing user confidence.

[0032] The system of the present invention demonstrates extremely high flexibility and extensibility. In addition to Excel files, the system can be extended to support other data input formats, and through the rich ecology of the R language, users can combine other machine learning tools to expand the analysis function. This makes the present invention not only limited to current analysis tasks, but also able to adapt to more complex data analysis needs in the future, becoming a powerful tool that is both practical and adaptable.

[0033] In summary, the present invention has the following beneficial effects:

[0034] 1. The present invention enables users to directly import data from Excel files and automatically complete data format conversion and standardization processing through an automated data reading and standardization module. This not only reduces the time for manual data sorting and preprocessing, but also reduces data errors caused by human operations and improves the efficiency of data processing.

[0035] 2. The present invention supports a variety of similarity and distance calculation algorithms, and standardizes the data before calculation to ensure the fairness and accuracy of the analysis results. In addition, the system has good scalability, allowing users to combine other R language tools for more in-depth analysis to meet the complex needs of different fields. This improves the applicability of the system in a variety of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic diagram of a system for similarity and distance calculation based on R language in Example 1 of the present invention;

[0037] Figure 2 It is a flow chart of a method for similarity and distance calculation based on R language in Example 2 of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example 1: A system for similarity and distance calculation based on R language, such as Figure 1 As shown, it includes the following modules:

[0040] Data reading module: The data reading module implements its key functions through the built-in `read_matrix_from_excel(file_path,sheet)` function. This process first involves reading the entire block of data from the specified Excel file and worksheet and loading it into the system memory.

[0041] The data reading module has intelligent data parsing capabilities, automatically identifying and separating the first line as column names and the first column as row names, ensuring the clarity of the table structure and the operability of the data. In order to ensure the numerical consistency of the data, the module performs data format conversion and strictly checks each data unit to identify and process possible non-numeric data, such as text or special characters, and promptly performs necessary numerical conversion or prompts the user to make manual corrections.

[0042] Data standardization module: The data standardization module implements the standardization of the input matrix through the `standardize_matrix(mat)` function. The main goal of this module is to perform standardization on the data of each column and adjust it to a normal distribution with a mean of zero and a standard deviation of one, so as to eliminate the influence of different dimensions on the similarity calculation. This process includes the following steps:

[0043] First, the module calculates the mean and standard deviation of each column of the input matrix. Then, the mean of the column is subtracted from each data item to obtain the mean-free value. Next, the mean-free value is divided by the standard deviation of the column to complete the normalization process. This normalization method not only unifies the dynamic range of the data, but also maintains the relative position of each data item relative to its column.

[0044] Similarity calculation module: This module supports a variety of similarity and distance calculation algorithms, including Euclidean distance and cosine similarity. Users can choose different calculation methods according to their needs and obtain sorted similarity or distance matrices. The functions of the similarity calculation module can be divided into various calculation and sorting operations, as follows:

[0045] Calculate Euclidean distance (`compute_euclidean_distances(mat)`): This function accepts a standardized matrix as input and calculates the Euclidean distance between the data points in the matrix pair by pair to form a distance matrix. The Euclidean distance is obtained by calculating the straight-line distance between the coordinates of different data points.

[0046] Sorted Euclidean distances (`get_sorted_euclidean_distances(dist_matrix)`): After obtaining the distance matrix, this function sorts all the distances, usually in ascending order, allowing users to easily identify the closest pairs of data points.

[0047] Calculate cosine similarity (`cosine_similarity(x,y)`): This function is used to calculate the cosine similarity between two vectors, which measures the consistency of the vector direction by the product of the dot product and the modulus. This similarity is often used in text or high-dimensional data, emphasizing direction rather than length.

[0048] Compute cosine similarities matrix (`compute_cosine_similarities(mat)`): Similar to the Euclidean distance calculation, this function calculates the cosine similarity for each pair of data points in the normalized matrix to form a similarity matrix.

[0049] Sorted cosine similarities (`get_sorted_cosine_similarities(sim_matrix)`): This function sorts the similarities in the similarity matrix, usually in descending order, so that users can easily find the most similar pairs of data points.

[0050] Get top 5 comparisons (`get_top_5_comparisons(sorted_distances, sorted_similarities)`): This function extracts the top five most representative comparisons from the sorted distance and similarity lists, helping users quickly identify and analyze the most significant relationships.

[0051] Through these functions, this module allows users to freely choose to use different algorithms to determine the relationship between data points according to specific analysis needs, and efficiently obtain the most important comparison results to support further decision-making and analysis.

[0052] Error checking module: The error checking module automatically verifies the data format before data analysis, especially issues warnings for non-numeric data to ensure smooth analysis. The `process_excel_file(file_path,sheet)` function reads data from the specified Excel file and worksheet, performs format verification and standardization, then calculates and sorts the Euclidean distance and cosine similarity, and finally returns the top 5 most important comparison results.

[0053] Example 2: A method for calculating similarity and distance based on R language. The flowchart of the method for calculating similarity and distance based on R language is as follows: Figure 2 As shown, the following steps are included:

[0054] Step 1: Data Import

[0055] The user imports data by specifying the path of the Excel file. This step is implemented by the built-in function `read_matrix_from_excel(file_path,sheet)`, which can automatically identify and read the Excel file in the path specified by the user. The system will read the contents of the Excel file and convert it into a data structure that can be processed by the R language. In this process, the system will automatically identify the first row of the Excel file as the column name and the first column as the row name, so that the data can be easily referenced in subsequent analysis.

[0056] Step 2: Data Check

[0057] After the data is imported, the system will conduct a strict check to ensure that all data is numeric. This step is crucial to ensure the accuracy of subsequent calculations. The system will check the data items one by one, and if any non-numeric data such as text or dates is found, the system will immediately prompt the user to correct it. This prompt mechanism enables users to understand data problems in a timely manner and take appropriate measures, such as data conversion or cleaning, to avoid these errors affecting subsequent calculations.

[0058] Step 3: Data Standardization

[0059] After the data is checked, the system will standardize the numerical data. This step is achieved through the built-in function `standardize_matrix(mat)`, which standardizes the data by column to ensure that the mean of each column is zero and the standard deviation is one. This standardization helps to eliminate the influence of different dimensional data on similarity calculations, so that data of different dimensions can be compared and analyzed under the same dimension.

[0060] Step 4: Similarity / distance calculation

[0061] After the data is standardized, users can choose different similarity or distance algorithms for calculation. The system supports multiple algorithms, such as Euclidean distance and cosine similarity. The user's choice will determine which algorithm the system uses to calculate the similarity or distance between data. After the calculation is completed, the system will return the sorted results in the form of a table. Each record will be displayed in the format of "Group No. 1 Group No. 2 Similarity / Distance", which is clear and easy for users to understand and analyze.

[0062] Step 5: Output the results

[0063] Finally, the system will output the top N similarity or distance results after sorting according to the user's selection. These results will be saved in a highly readable data format, such as CSV or Excel files, to facilitate further analysis and research by users. This step ensures that users can easily obtain and use the analysis results, whether for further data processing or directly for reporting and presentation.

[0064] Through the above five steps, this embodiment provides a complete analysis process from data import to result output. This process not only improves the efficiency and accuracy of data analysis, but also makes the entire analysis process simple and intuitive through user-friendly prompts and output formats. This similarity and distance calculation method based on R language is particularly suitable for users who need to process and analyze large amounts of complex data, such as data scientists, researchers, and analysts.

[0065] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A system for similarity and distance calculation based on R language, characterized by: Includes the following modules: The data reading module is used to read data from Excel files and convert the data format to ensure that all data is numeric. It also automatically identifies the first row as the column name and the first column as the row name, and processes non-numeric data. The data standardization module is used to standardize the data by column so that each column of data satisfies the distribution with a mean of zero and a standard deviation of one after transformation, so as to eliminate the dimensional differences between different columns; Similarity calculation module, used to support multiple similarity and distance calculation algorithms, including Euclidean distance and cosine similarity, and provide sorted similarity or distance matrix; The error checking module is used to automatically check whether the data format is correct before data processing and to issue prompts for non-numeric data.

2. The system according to claim 1, characterized in that: The data reading module includes a built-in function `read_matrix_from_excel(file_path,sheet)`, which is used to read a matrix from an Excel file and check for non-numeric content.

3. The system according to claim 1, characterized in that: The data normalization module includes a built-in function `standardize_matrix(mat)`, which is used to normalize the input matrix by columns.

4. The system according to claim 1, characterized in that: The similarity calculation module comprises: Built-in function `compute_euclidean_distances(mat)`, used to calculate the Euclidean distance of a normalized matrix; Built-in function `cosine_similarity(x,y)`, used to calculate the cosine similarity between two vectors; Built-in function `get_sorted_euclidean_distances(dist_matrix)`, which returns the sorted Euclidean distances; Built-in function `compute_cosine_similarities(mat)`, used to calculate the cosine similarity of a normalized matrix; Built-in function `get_sorted_cosine_similarities(sim_matrix)`, which returns sorted cosine similarities; Built-in function `get_top_5_comparisons(sorted_distances, sorted_similari ties)`, which returns the top 5 comparisons of distances and similarities.

5. The system according to claim 1, characterized in that: The system also includes a main function `process_excel_file(file_path,sheet)` for processing the Excel file and returning the results of the first 5 comparisons.

6. A method for similarity and distance calculation based on R language, characterized by: The following steps are involved: Step 1: Import Excel file and import data by specifying the file path; Step 2: Check the data to ensure that all data are numerical; Step 3: Data standardization: standardize the numerical data by column; Step 4: Similarity / distance calculation, select different similarity or distance algorithms for calculation; Step 5: Output the results, output the top N groups of similarity or distance results after sorting.

Images