Dimension reduction visualization method and device for high-dimensional medical and invasive data and medium

Abstract

The invention relates to the field of data processing and visualization, in particular to a dimensionality reduction visualization method and device for high-dimensional medical data and a medium. The method comprises the following steps: obtaining high-dimensional medical and invasive data, and carrying out data cleaning and standardization processing to obtain a preprocessed data set; extracting features from the preprocessed data set and automatically selecting a dimension reduction algorithm to generate dimension reduction parameters; performing distributed dimension reduction processing based on the dimension reduction parameters to obtain low-dimension data representation; and inputting the low-dimensional data into an intelligent interaction visualization tool, carrying out three-dimensional visualization display, and carrying out dynamic optimization in combination with the interaction behavior of the user. According to the method, the processing efficiency of the high-dimensional data is improved, and deep interaction and analysis of the user on the data are supported.

Description

Technical Field

[0001] The present invention relates to the field of data processing and visualization, and in particular to a dimensionality reduction visualization method, device and medium for high-dimensional scientific and technological innovation data. Background Art

[0002] With the continuous development of science and technology, the amount and dimension of data in the field of scientific research are increasing. Especially in the process of scientific innovation and technological research and development, the high-dimensional characteristics of data have brought huge challenges to data analysis and visualization. Traditional data processing methods and visualization technologies are difficult to deal with this "dimensionality disaster" problem, resulting in low data processing efficiency and poor visualization effects.

[0003] Dimensionality reduction technology is an effective means to deal with high-dimensional data processing problems. It simplifies the representation of data by reducing the data dimension while retaining the main features of the data. However, existing dimensionality reduction algorithms often have limited applicability, high computational complexity, and poor adaptability to different data sets. Especially for large-scale high-dimensional scientific and technological data, how to select a suitable dimensionality reduction algorithm and achieve efficient computational processing remains a technical problem that needs to be solved urgently.

[0004] In addition, traditional visualization tools usually lack intelligent interaction functions when facing reduced-dimensional data, and cannot effectively support users' in-depth exploration and analysis of data features. When users interact with data, it is difficult to dynamically adjust the display content according to their own needs, which limits the data exploration process and makes it impossible to fully tap the potential value of the data. Therefore, developing a visualization system that can automatically reduce dimensionality and support user interaction is of great practical significance for improving the efficiency of high-dimensional data analysis and user experience. Summary of the invention

[0005] The present invention provides a dimensionality reduction visualization method, device and medium for high-dimensional scientific and technological data to solve the problem of how to efficiently reduce the dimensionality of complex high-dimensional scientific and technological data so as to maintain core features during the visualization process and support users to dynamically explore and deeply analyze the data through interactive behaviors.

[0006] In order to solve the above technical problems, the present invention provides a dimensionality reduction visualization method for high-dimensional scientific and technological innovation data, comprising: Acquire high-dimensional scientific and technological innovation data, clean and standardize the data, and obtain a preprocessed data set; Extract and analyze features from the preprocessed data set, automatically select a dimensionality reduction algorithm, and obtain dimensionality reduction parameters; Based on the dimension reduction parameters, the data set is subjected to distributed dimension reduction processing to obtain a low-dimensional data representation after dimension reduction; The low-dimensional data is input into an intelligent interactive visualization tool, the data is visualized in three dimensions, and dynamically optimized in combination with user interaction behaviors to generate interactive visualization display results.

[0007] Furthermore, the method further comprises: Obtain the high-dimensional scientific and technological innovation data, and extract a high-dimensional data set including multi-dimensional features, indicators and experimental data from a specified data source; Performing data cleaning on the data set to remove missing values, outliers and duplicates to obtain a cleaned data set; The cleaned data is standardized to unify the dimensions and scales of different features and generate a standardized data set.

[0008] Furthermore, the method further comprises: Extracting main features of data from the standardized data set; Perform statistical analysis on the extracted features, evaluate the complexity of the data, and determine the need for dimensionality reduction; A dimensionality reduction algorithm is automatically selected based on the analysis results, and dimensionality reduction parameters are generated.

[0009] Furthermore, the dimension reduction algorithm includes an improved PCA algorithm and a t-SNE algorithm, and the dimension reduction parameters include a target dimension number, a number of iterations and a learning rate.

[0010] Furthermore, the method further comprises: Based on the dimensionality reduction parameters, determine a dimensionality reduction strategy and initialize a dimensionality reduction calculation process in the data set; Using distributed computing technology, the data set is subjected to distributed dimensionality reduction processing to map the data into a low-dimensional space; Merge the computation results of distributed computing nodes to obtain a complete low-dimensional data representation.

[0011] Furthermore, the distributed dimensionality reduction processing includes: Execute the dimensionality reduction algorithm on multiple distributed nodes to perform dimensionality reduction calculations on the sub-data sets held by each node; Each node calculates the data projection matrix or low-dimensional embedding matrix according to the target dimension number of the dimensionality reduction algorithm.

[0012] Furthermore, the method further comprises: Inputting the low-dimensional data representation into an intelligent interactive visualization tool to generate an initial three-dimensional data visualization display; The data is visualized in three dimensions, allowing users to explore the data features through rotation, scaling, and selection interactions.

[0013] Furthermore, the system comprises: A data preprocessing module is used to obtain high-dimensional scientific and technological innovation data from multiple data sources, and perform data cleaning and standardization on the data to generate a preprocessed data set; A feature extraction and dimensionality reduction algorithm selection module is used to extract features from the preprocessed data set, and automatically select a dimensionality reduction algorithm according to data complexity to generate dimensionality reduction parameters; A distributed dimensionality reduction calculation module, used to perform distributed dimensionality reduction processing on the preprocessed data set based on the dimensionality reduction parameters to generate a low-dimensional data representation; The intelligent interactive visualization module is used to input the low-dimensional data representation into the intelligent interactive visualization tool, and supports users to explore data features by rotating, zooming, and selecting interactive methods to generate interactive visualization display results.

[0014] Furthermore, the device comprises: A data acquisition unit, used to acquire high-dimensional scientific and technological innovation data from multiple data sources, wherein the data includes experimental data, sensor data, and historical data; A data processing unit, used to perform data cleaning and standardization processing on the acquired high-dimensional data to obtain a preprocessed data set; A feature extraction and dimensionality reduction unit, used to extract effective features from the preprocessed data set, automatically select a dimensionality reduction algorithm, and generate dimensionality reduction parameters; The visualization presentation unit is used to input the low-dimensional data after dimensionality reduction into a three-dimensional visualization engine to generate a visualization presentation of the three-dimensional data, and support users to explore the data in an interactive manner.

[0015] Furthermore, a computer program is stored thereon, and when the program is executed by a processor, any of the above-mentioned methods for dimensionality reduction and visualization of high-dimensional scientific and technological innovation data is executed.

[0016] The key innovative features of the present invention include: (1) Automatically select dimensionality reduction algorithms and combine them with data complexity assessment to achieve a more intelligent and accurate dimensionality reduction process; (2) Distributed dimensionality reduction computing method improves the processing efficiency of large-scale data; (3) Intelligent interactive visualization tools provide optimized functions for user dynamic interaction and data display, enhancing the intuitiveness and convenience of data exploration.

[0017] The following are its main beneficial effects: (1) Efficient data processing: The present invention effectively simplifies complex high-dimensional data through data cleaning, standardization and feature extraction, automatically selects appropriate dimensionality reduction algorithms, and improves the accuracy and speed of dimensionality reduction processing. Especially when processing multi-dimensional scientific and technological data, it can significantly reduce the computational complexity and solve the bottleneck of traditional dimensionality reduction methods in high-dimensional data processing.

[0018] (2) Distributed dimensionality reduction computing: Using a distributed architecture, the data set is distributed to multiple nodes for parallel dimensionality reduction processing, ensuring that large-scale data can be quickly reduced in dimensionality, thereby improving processing efficiency. Compared with the single-node dimensionality reduction method, distributed computing can significantly shorten the computing time and improve the adaptability to large-scale data sets.

[0019] (3) Intelligent interactive visualization: The present invention uses a three-dimensional visualization tool to display the reduced-dimensional data in three dimensions, and combines the user's interactive behavior to dynamically optimize the display effect in real time, supporting users to conduct in-depth exploration and analysis of the data. Compared with traditional static display, the intelligent interactive function of the present invention improves the user experience and enables users to flexibly view and analyze data features. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a method for visualizing dimensionality reduction of high-dimensional scientific and technological innovation data provided in an embodiment of the present application; Figure 2 A system structure diagram of a dimensionality reduction and visualization method for high-dimensional scientific and technological innovation data provided in an embodiment of the present application; Figure 3 A structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first" and "second" in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0022] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0023] Embodiment 1: Reference Figure 1 , is a flow chart of a method for dimensionality reduction and visualization of high-dimensional scientific and technological innovation data provided by an embodiment of the present invention, and the process may at least include steps S100-S400: S100, obtaining high-dimensional scientific and technological innovation data, cleaning and standardizing the data, and obtaining a preprocessed data set; S200, extracting and analyzing features from the preprocessed data set, automatically selecting a suitable dimensionality reduction algorithm, and obtaining dimensionality reduction parameters; S300, performing distributed dimensionality reduction processing on the data set based on the dimensionality reduction parameter to obtain a low-dimensional data representation after dimensionality reduction; S400, inputting the low-dimensional data into an intelligent interactive visualization tool, performing three-dimensional visualization on the data, and dynamically optimizing the data in combination with user interaction behavior to generate an interactive visualization display result.

[0024] Step S100 at least includes S110-S130: S110: Obtain high-dimensional scientific and technological innovation data and extract high-dimensional data sets from specified data sources, including various multidimensional features, indicators, and experimental data.

[0025] In step S110, the system extracts a high-dimensional scientific and technological innovation data set from the specified data source. The data set includes multi-dimensional features, indicators and experimental data, specifically including data tables obtained from the scientific and technological innovation data platform and experimental data collected by sensors. The data set is defined as ,in ={ , ,…, }, each is a sample with multi-dimensional features, and the number of features is m. These features include scientific and technological innovation indicators, project parameters, and research results data. For each sample’s feature set ={ , ,…, }, the data sources and dimensions of each dimension are different, which lays the foundation for subsequent unified processing.

[0026] S120: Clean the data to remove missing values, outliers and duplicates.

[0027] For the dataset Perform integrity check, process features with missing values, and define the processed data set as For each feature dimension The missing part in , then use the mean filling strategy And interpolation method to complete. Specifically, if a dimension If missing, calculate the mean of the dimension in all samples: ,like If missing =

[0028] Furthermore, for each feature Use z-score normalization to detect outliers and calculate the standard deviation of each feature and mean :

[0029] When | When |>3, the data point is determined as an outlier and processed, for example, by replacing it with the mean of the adjacent normal values. The processed data set continues to be .

[0030] For the dataset Duplicates are checked and if a sample is found If it is exactly the same as other samples, it will be deleted. The dataset obtained after removing duplicates is defined as , used for standardization.

[0031] S130: Standardize the cleaned data, unify the dimensions and scales of different features, and generate a standardized data set.

[0032] After the data cleaning is completed, go to step S130 to clean the data set. Perform standardization to obtain a standardized data set .

[0033] Specifically, for each feature , use the Min-Max normalization method to scale it to the interval [0,1], the formula is as follows:

[0034] in and Characteristics The minimum and maximum values ​​in the data set. After this standardization step, the standardized data set obtained is ={ , ,…, }, each Contains normalized features.

[0035] Furthermore, after the standardization process in step S130, the obtained The data set will be further input into the feature extraction module (S200) of the next stage for automatically selecting a suitable dimensionality reduction algorithm.

[0036] Step S200 at least includes steps S210-S230: S210: Extract the main features of the data from the preprocessed data set for analysis in S220.

[0037] In step S210, based on the data set after the standardization process in step S130 ={ , ,…, }, extract the main features of the data set. Specifically, by analyzing the data distribution of each feature dimension and the correlation between features, the core features of the data are extracted.

[0038] For the dataset Each standardized feature dimension in Calculate its statistics, including the mean ,variance , and the skewness and kurtosis of the data distribution.

[0039]

[0040]

[0041] Furthermore, based on all the features The linear correlation between them is used to calculate the correlation coefficient matrix R between the features. The elements in the matrix Representation characteristics and The Pearson correlation coefficient between is defined as follows:

[0042] Through the correlation coefficient matrix R, we can further analyze the linear relationship between features and obtain highly correlated feature pairs for reference in subsequent dimensionality reduction.

[0043] According to the feature distribution and correlation analysis results, the main features of the data set are extracted and a feature set is generated. ={ , … }, used for statistical analysis in the subsequent step S220.

[0044] S220: Perform statistical analysis on the extracted features, and evaluate the complexity of the data and the need for dimensionality reduction based on the properties of the features.

[0045] For each feature ∈ Evaluate linear and nonlinear properties and calculate linearity indicators and nonlinearity index . Specifically, Indicates the degree of linear relationship between a feature and other features, while It measures the nonlinear distribution of the feature. It is defined as follows:

[0046] Furthermore, for the feature set, the complexity of the entire data set is calculated , to determine the difficulty of dimensionality reduction. Complexity Defined as the weighted average of all characteristic nonlinearity indices:

[0047] According to the complexity of the data , classify and evaluate the dimensionality reduction requirements. > When , the data has high complexity and tends to choose nonlinear dimensionality reduction methods; when ≤ When , the linear dimensionality reduction method can meet the requirements, where is a pre-set complexity threshold.

[0048] S230: Based on the analysis results, automatically select a suitable dimensionality reduction algorithm and generate dimensionality reduction parameters.

[0049] After completing the statistical analysis and complexity evaluation of the features, proceed to step S230 to automatically select a suitable dimensionality reduction algorithm based on the analysis results and generate dimensionality reduction parameters to guide the subsequent dimensionality reduction process.

[0050] According to the complexity classification result obtained in step S220, select a suitable data dimensionality reduction algorithm. > When , choose a nonlinear dimensionality reduction method; otherwise, choose a linear dimensionality reduction method, such as the improved PCA algorithm.

[0051] For the selected dimensionality reduction algorithm, further determine the dimensionality reduction parameter set , including the target dimension d and the number of iterations , learning rate α. Specifically, the target dimension number d can be set according to the cumulative contribution rate of the features, so that the contribution rate is greater than 90%.

[0052] The generated dimension reduction parameter set The output is used to guide the subsequent dimensionality reduction process (module S300), which includes the selected dimensionality reduction algorithm type and specific parameter configuration.

[0053] Step S300 at least includes steps S310-S330: S310: Determine a dimensionality reduction strategy using the selected dimensionality reduction parameters, and initialize a dimensionality reduction calculation process in the data set.

[0054] Based on the dimension reduction parameter set , determine the type of dimensionality reduction algorithm to be used. If the improved PCA algorithm is selected, the target dimension d and sample matrix in the parameter set are used ={ , ,…, }Initialize PCA dimension reduction matrix calculation; if the improved t-SNE algorithm is selected, initialize t-SNE similarity measure matrix calculation.

[0055] Furthermore, the initial parameters of the dimensionality reduction calculation are set according to the selected dimensionality reduction algorithm, such as the target dimension d, the number of iterations And learning rate α. For PCA, calculate the feature covariance matrix C:

[0056] in, is the mean vector of the data set. For t-SNE, the similarity probability based on the Euclidean distance is calculated :

[0057] Initialize the distributed computing environment and set the data set Split into multiple sub-datasets ( , ,… ), where p is the number of distributed nodes, ensuring that each node is assigned a dimensionality reduction task in the initial stage.

[0058] S320: Perform distributed dimensionality reduction on the data set based on the improved PCA or t-SNE algorithm to map the data into a low-dimensional space.

[0059] On each distributed node, the sub-datasets held by each node are Perform dimensionality reduction calculations, where (q=1,2,…,p). For nodes using the PCA algorithm, calculate the data projection matrix :

[0060] in, is the eigenvector matrix of the first d principal components obtained by eigenvalue decomposition. For nodes using the t-SNE algorithm, the low-dimensional embedding is updated by gradient descent:

[0061] in, is the cost function of t-SNE, ∇ for its gradient.

[0062] Multiple iterative calculations are performed on each distributed node to make the low-dimensional representation converge to a reasonable range. For t-SNE calculations, the similarity probability obtained by local calculations is used. Map low-dimensional points until the cost function convergence:

[0063] Each distributed node will calculate the low-dimensional representation matrix Keep it locally and wait for merging and processing in the subsequent step S330.

[0064] S330: Execute dimensionality reduction calculation in a distributed environment, merge the calculation results of each distributed node, and obtain a low-dimensional data representation after dimensionality reduction.

[0065] Collect the low-dimensional representation matrix of each sub-dataset from all distributed nodes ( , …, }Furthermore, all sub-matrices are merged by sample index to generate a complete low-dimensional dataset representation .

[0066] For the merged low-dimensional dataset Smoothing is performed to remove inconsistencies that may arise from distributed computing and to ensure that the structure of the overall data in the low-dimensional space remains consistent.

[0067] Output the final low-dimensional data representation , which is used in the subsequent intelligent interactive visualization tool (S400 module) to enable users to conduct in-depth exploration and interactive operations on the data.

[0068] Step S400 at least includes steps S410-S430: S410: Input the low-dimensional data representation after dimensionality reduction into an intelligent interactive visualization tool to generate an initial three-dimensional data visualization display.

[0069] The low-dimensional data is represented as ={ , ,…, } is mapped into a three-dimensional coordinate space, where each data point Represents the three-dimensional coordinate position of a sample, defined as =( , , ).according to The characteristic range of determines the range and distribution of the three-dimensional coordinate axes.

[0070] Furthermore, a 3D rendering engine is used to draw all data points in 3D space. , marked with corresponding colors or shapes in three-dimensional space to distinguish different types of samples or categories. The feature label information is obtained from the feature set in the previous feature extraction stage (S210). , and used to display the corresponding feature names in the visualization.

[0071] Generate initial 3D data graphics , as the basis for subsequent interactive display. The graphic is displayed on the user interface and waits for the user's interactive input.

[0072] S420: Perform three-dimensional visualization on the data, and support the user to explore different features of the data through rotation, zooming, and selection interaction.

[0073] After the initial three-dimensional data display is completed, the process proceeds to step S420 to perform three-dimensional visualization of the data, supporting the user to explore different features of the data through rotation, scaling, and selection interactions.

[0074] Understandably, receiving the user's interactive operation instructions includes rotation, scaling, and selection operations. When the user requests to rotate the three-dimensional view, the system responds based on the angle θ input by the user. For 3D graphics Perform a rotation transformation. The rotation matrix R(θ, ) acts on each data point , calculate the rotated coordinates:

[0075] Among them, R(θ, ) is a three-dimensional rotation matrix constructed based on the rotation angle specified by the user.

[0076] Furthermore, users can adjust the scale of the visualization by zooming in and out. The system scales the coordinates of all data points in the three-dimensional graph according to the zoom factor s entered by the user:

[0077] Among them, s>1 indicates a zoom-in operation, and s<1 indicates a zoom-out operation.

[0078] When the user selects a data point, the system captures the selected point , and display the feature information of the point based on the previous feature extraction result (S210). The feature information of the selected data point is obtained by querying the feature set Display so that users can gain in-depth understanding of the specific characteristics of the data.

[0079] S430: Based on the user's interactive behavior, dynamically optimize the visualization results, adjust the display angle and data details, and generate interactive visualization results.

[0080] Based on the user's interactive behavior records, the system analyzes the user's operation frequency and operation type, and determines the user's focus and interaction preferences during the data exploration process. Define the user behavior matrix B = { ),in Indicates the user's data point The frequency of performing the jth type of operation (such as rotation, scaling, selection).

[0081] According to the user behavior matrix B, the display angle and details of the 3D visualization are dynamically optimized. For example, if the user selects a certain type of data point multiple times, the system automatically zooms in on the display of that area and adds detailed markings for the data points. At the same time, the default display angle of the 3D graphics is adjusted for the directions where users frequently interact, to better suit the user's operating habits.

[0082] Generate the final interactive visualization display results , and presents it in the user interface. The optimized visualization results include adjusted data point positions, automatically annotated feature information, and optimized 3D perspectives, allowing users to conduct deeper data exploration.

[0083] The key innovative features of the present invention include: (1) Automatically select dimensionality reduction algorithms and combine them with data complexity assessment to achieve a more intelligent and accurate dimensionality reduction process; (2) Distributed dimensionality reduction computing method improves the processing efficiency of large-scale data; (3) Intelligent interactive visualization tools provide optimized functions for user dynamic interaction and data display, enhancing the intuitiveness and convenience of data exploration.

[0084] The following are its main beneficial effects: (1) Efficient data processing: The present invention effectively simplifies complex high-dimensional data through data cleaning, standardization and feature extraction, automatically selects appropriate dimensionality reduction algorithms, and improves the accuracy and speed of dimensionality reduction processing. Especially when processing multi-dimensional scientific and technological data, it can significantly reduce the computational complexity and solve the bottleneck of traditional dimensionality reduction methods in high-dimensional data processing.

[0085] (2) Distributed dimensionality reduction computing: Using a distributed architecture, the data set is distributed to multiple nodes for parallel dimensionality reduction processing, ensuring that large-scale data can be quickly reduced in dimensionality, thereby improving processing efficiency. Compared with the single-node dimensionality reduction method, distributed computing can significantly shorten the computing time and improve the adaptability to large-scale data sets.

[0086] (3) Intelligent interactive visualization: The present invention uses a three-dimensional visualization tool to display the reduced-dimensional data in three dimensions, and combines the user's interactive behavior to dynamically optimize the display effect in real time, supporting users to conduct in-depth exploration and analysis of the data. Compared with traditional static display, the intelligent interactive function of the present invention improves the user experience and enables users to flexibly view and analyze data features.

[0087] Embodiment 2: Figure 2 is a structural block diagram of a dimensionality reduction visualization system for high-dimensional scientific and technological innovation data provided by an embodiment of the present invention. Figure 2 As shown, the system may include: The data preprocessing module 11 is used to obtain high-dimensional scientific and technological innovation data from multiple data sources and preprocess it, including data cleaning, missing value filling, and data standardization operations. Specifically, the data preprocessing module 10 first obtains a data set containing multi-dimensional features from multiple sources (such as experimental data, sensor collection data, and historical document data), cleans it, removes outliers and missing values, and then standardizes different features to unify the data scale, providing a basis for subsequent data analysis.

[0088] The feature extraction and dimension reduction algorithm selection module 12 is used to extract effective features from the data set output by the data preprocessing module 10, and automatically select a suitable dimension reduction algorithm according to the nature of the features and the complexity of the data. Specifically, the module extracts features from the preprocessed data set, determines the dimension reduction requirements by analyzing the correlation between features and the complexity of the data, and automatically selects an appropriate dimension reduction algorithm, such as an improved PCA or t-SNE algorithm, to ensure the efficiency and effect of the dimension reduction.

[0089] The distributed dimension reduction calculation module 13 is used to perform dimension reduction calculations and map high-dimensional data into low-dimensional space. Specifically, the module uses distributed computing technology to reduce the dimension of data based on the dimension reduction parameters generated in the feature extraction and dimension reduction algorithm selection module 20 to improve the computational efficiency of large-scale data sets. A distributed architecture is used to distribute data to multiple nodes for parallel dimension reduction calculations, and finally the calculation results of all nodes are merged to obtain a low-dimensional data representation after dimension reduction.

[0090] The intelligent interactive visualization module 14 is used to perform three-dimensional visualization of the reduced-dimensional data and support dynamic interactive operations of users. Specifically, the intelligent interactive visualization module 40 inputs the low-dimensional data output by the distributed dimensionality reduction calculation module 30 into the three-dimensional visualization engine to generate an initial visualization display. Users can rotate, scale, and select data through the interactive interface. The system dynamically adjusts the display effect according to the user's interactive behavior, so that users can explore and understand data features more intuitively.

[0091] The present invention can efficiently reduce the dimensionality of high-dimensional scientific and technological data and visualize it by integrating multifunctional modules of data preprocessing, feature extraction, distributed dimensionality reduction and intelligent interactive visualization, solving the problems of computational complexity and visualization difficulties caused by the dimensionality disaster in traditional methods. The system can significantly improve data processing efficiency and provide users with an intuitive and dynamic visual interactive interface, improving user experience and enabling users to better mine useful information from high-dimensional data.

[0092] The high-dimensional scientific and technological innovation data dimensionality reduction visualization system provided by the present invention can efficiently complete the dimensionality reduction processing and visualization display of high-dimensional data through the integration of data preprocessing module, feature extraction and dimensionality reduction module, distributed dimensionality reduction calculation module and intelligent interactive visualization module. The system realizes efficient processing of large-scale data sets through distributed architecture, and helps users intuitively understand the characteristics and internal relationships of data in an interactive manner through visualization, which significantly improves the user experience.

[0093] Embodiment three: Figure 3 FIG. 2 shows a block diagram of a computer device according to an embodiment of the present application. Figure 3 As shown, the device may include: The data acquisition unit 21 is used to acquire high-dimensional scientific and technological innovation data from multiple data sources, and the data includes experimental data, sensor data and historical data; further, the data acquisition unit 21 is also used to acquire multi-dimensional feature information in the high-dimensional data for subsequent processing.

[0094] The data processing unit 22 is used to clean and standardize the acquired high-dimensional scientific and technological innovation data, including removing missing values, outliers and duplicates to ensure the consistency and accuracy of the data; the data processed by the data processing unit 22 will be used for subsequent feature extraction and dimensionality reduction analysis.

[0095] The feature extraction and dimensionality reduction unit 23 is used to extract effective features from the data output by the data processing unit 22 and automatically select a dimensionality reduction algorithm; specifically, the feature extraction and dimensionality reduction unit 23 selects a suitable dimensionality reduction method based on the correlation and complexity analysis of the features, including improved PCA and t-SNE algorithms, to generate low-dimensional data after dimensionality reduction.

[0096] The visualization presentation unit 24 is used to input the reduced low-dimensional data into the three-dimensional visualization engine to generate a visualization display of the three-dimensional data; further, it supports users to rotate, scale and select the visualization content in an interactive manner to more intuitively explore and understand the characteristics of the high-dimensional data.

[0097] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned dimensionality reduction and visualization method of high-dimensional scientific and technological innovation data can be implemented.

[0098] The computer-readable storage medium may be a non-volatile storage medium, such as a read-only memory (ROM), flash memory or other types of memory, or a volatile memory, such as a random access memory (RAM).

[0099] Furthermore, an embodiment of the present application also provides a chip, which includes a processor and a memory, wherein the processor is used to load and run instructions stored in the memory, so that a computing device equipped with the chip executes the above-mentioned method for dimensionality reduction and visualization of high-dimensional scientific and technological innovation data. The present invention can significantly improve the analysis efficiency of high-dimensional data and the user interaction experience by integrating data processing, dimensionality reduction calculation and visualization display.

[0100] The computer device structure provided by the present invention combines data acquisition, processing, feature extraction and dimensionality reduction, and visual presentation units to achieve comprehensive processing and presentation of high-dimensional data. The device automatically selects a suitable dimensionality reduction algorithm through effective data preprocessing and feature extraction, and combines the dynamic interactive presentation of the visual presentation unit to greatly improve the analysis efficiency of high-dimensional data and the user's interactive experience. By combining the calculation process with user interaction, users can quickly gain insights into the data to support decision-making and scientific research.

[0101] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to effectively replace some of the technical features therein. Any effective structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is also within the scope of patent protection of this application.

Claims

1. A dimensionality reduction visualization method for high-dimensional scientific and technological data, characterized in that: The following steps are involved: Acquire high-dimensional scientific and technological innovation data, clean and standardize the data, and obtain a preprocessed data set; Extract and analyze features from the preprocessed data set, automatically select a dimensionality reduction algorithm, and obtain dimensionality reduction parameters; Based on the dimension reduction parameters, the data set is subjected to distributed dimension reduction processing to obtain a low-dimensional data representation after dimension reduction; The low-dimensional data is input into an intelligent interactive visualization tool, the data is visualized in three dimensions, and dynamically optimized in combination with user interaction behaviors to generate interactive visualization display results.

2. The dimensionality reduction visualization method for high-dimensional scientific and technological innovation data according to claim 1 is characterized in that: The method further comprises: Obtain the high-dimensional scientific and technological innovation data, and extract a high-dimensional data set including multi-dimensional features, indicators and experimental data from a specified data source; Performing data cleaning on the data set to remove missing values, outliers and duplicates to obtain a cleaned data set; The cleaned data is standardized to unify the dimensions and scales of different features and generate a standardized data set.

3. The dimensionality reduction visualization method for high-dimensional scientific and technological innovation data according to claim 2 is characterized in that: The method further comprises: Extracting features of data from the standardized data set; Perform statistical analysis on the extracted features, evaluate the complexity of the data, and determine the need for dimensionality reduction; A dimensionality reduction algorithm is automatically selected based on the analysis results, and dimensionality reduction parameters are generated.

4. The dimensionality reduction visualization method for high-dimensional scientific and technological innovation data according to claim 3 is characterized in that: The dimension reduction algorithms include an improved PCA algorithm and a t-SNE algorithm, and the dimension reduction parameters include a target dimension number, a number of iterations, and a learning rate.

5. The dimensionality reduction visualization method for high-dimensional scientific and technological innovation data according to claim 1, characterized in that: The method further comprises: Based on the dimensionality reduction parameters, determine a dimensionality reduction strategy and initialize a dimensionality reduction calculation process in the data set; Using distributed computing technology, the data set is subjected to distributed dimensionality reduction processing to map the data into a low-dimensional space; Merge the computation results of distributed computing nodes to obtain a complete low-dimensional data representation.

6. The dimensionality reduction visualization method for high-dimensional scientific and technological innovation data according to claim 5, characterized in that: The distributed dimensionality reduction process includes: Execute the dimensionality reduction algorithm on multiple distributed nodes to perform dimensionality reduction calculations on the sub-data sets held by each node; Each node calculates the data projection matrix and low-dimensional embedding matrix according to the target dimension number of the dimensionality reduction algorithm.

7. The dimensionality reduction visualization method of high-dimensional scientific and technological innovation data according to claim 1 is characterized in that: The method further comprises: Inputting the low-dimensional data representation into an intelligent interactive visualization tool to generate an initial three-dimensional data visualization display; The data is visualized in three dimensions, allowing users to explore the data features through rotation, scaling, and selection interactions.

8. A dimensionality reduction visualization system for high-dimensional scientific and technological data, characterized in that: The system comprises: A data preprocessing module is used to obtain high-dimensional scientific and technological innovation data from multiple data sources, and perform data cleaning and standardization on the data to generate a preprocessed data set; A feature extraction and dimensionality reduction algorithm selection module is used to extract features from the preprocessed data set, and automatically select a dimensionality reduction algorithm according to data complexity to generate dimensionality reduction parameters; A distributed dimensionality reduction calculation module, used to perform distributed dimensionality reduction processing on the preprocessed data set based on the dimensionality reduction parameters to generate a low-dimensional data representation; The intelligent interactive visualization module is used to input the low-dimensional data representation into the intelligent interactive visualization tool, and supports users to explore data features by rotating, zooming, and selecting interactive methods to generate interactive visualization display results.

9. A dimensionality reduction visualization device for high-dimensional scientific and technological data, characterized in that: The device comprises: A data acquisition unit, used to acquire high-dimensional scientific and technological innovation data from multiple data sources, wherein the data includes experimental data, sensor data, and historical data; A data processing unit, used to perform data cleaning and standardization processing on the acquired high-dimensional data to obtain a preprocessed data set; A feature extraction and dimensionality reduction unit, used to extract effective features from the preprocessed data set, automatically select a dimensionality reduction algorithm, and generate dimensionality reduction parameters; The visualization presentation unit is used to input the low-dimensional data after dimensionality reduction into a three-dimensional visualization engine to generate a visualization presentation of the three-dimensional data, and support users to explore the data in an interactive manner.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes the dimensionality reduction and visualization method for high-dimensional scientific and technological data according to any one of claims 1 to 9.

Images