Data analysis method and system based on questionnaire survey

Abstract

The invention discloses a data analysis method and system based on questionnaire investigation, and particularly relates to the technical field of data analysis, and the method comprises the following steps: S1, collecting and preprocessing questionnaire data, and obtaining original questionnaire data; s2, questionnaire feature engineering: extracting core features from the standardized questionnaire data set based on a joint screening mechanism of semantic similarity and information gain, and obtaining low-dimensional feature vectors through a dimension reduction algorithm; s3, subject mining and user group division are carried out on the low-dimensional feature vectors; and S4, outputting a result based on the analysis model, and generating a targeted questionnaire analysis report which comprises topic distribution, group difference and potential demand association conclusions. According to the data analysis method and system based on questionnaire investigation, through a multi-strategy fusion preprocessing mechanism, a combined screening feature engineering method, a fusion modeling analysis strategy and a self-adaptive model optimization module, the accuracy and efficiency of questionnaire data analysis are improved.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, and more specifically, to a data analysis method and system based on questionnaire surveys.

[0002] Background Technology Data

[0003] Questionnaire surveys are a common research method in social sciences, market research, and product demand analysis. By designing structured or semi-structured questionnaires, relevant information about the target group is obtained, leading to core conclusions such as group characteristics and preferences. With the development of big data technology, the scale of questionnaire surveys has gradually expanded, and the data dimensions have continuously increased. Traditional questionnaire data analysis methods are no longer sufficient to meet the needs for precise and efficient analysis.

[0004] Existing questionnaire data analysis methods have the following shortcomings: First, the data preprocessing stage is relatively simple. Missing value imputation often uses a single mean, median, or mode method, without considering data type differences and the correlation between fields, resulting in low imputation accuracy. Outlier identification often relies on a single statistical method, which easily misclassifies reasonable extreme values ​​as outliers, affecting data quality. Second, the feature engineering stage lacks an effective mechanism for removing redundant features. For questionnaire items containing a large number of synonyms or highly correlated items, feature dimension redundancy is likely to occur, increasing the computational complexity of subsequent analysis. Traditional dimensionality reduction algorithms, such as Principal Component Analysis (PCA), are difficult to handle the nonlinear features in questionnaire data, leading to data distortion after dimensionality reduction. Third, the analysis and modeling stage often uses a single topic model or clustering model. Topic models, such as the Latent Dirichlet Allocation (LDA) model, do not consider the correlation between topics, resulting in low topic mining accuracy. Clustering models, such as K-means clustering, randomly select initial cluster centers, which can easily get trapped in local optima, affecting the rationality of user group segmentation.

[0005] Therefore, a data analysis method and system based on questionnaire surveys is proposed to address the above problems. Summary of the Invention

[0006] The purpose of this application is to provide a data analysis method and system based on questionnaire surveys.

[0007] This application provides a data analysis method and system based on questionnaire surveys, which adopts the following technical solution:

[0008] On the one hand, a data analysis method based on questionnaire surveys includes the following steps:

[0009] S1: Questionnaire data collection and preprocessing. After obtaining the raw questionnaire data, a standardized questionnaire dataset is obtained through a multi-strategy fusion method for imputing missing values ​​and outlier identification.

[0010] S2: Questionnaire feature engineering, based on a joint screening mechanism of semantic similarity and information gain, extracts core features from a standardized questionnaire dataset and obtains low-dimensional feature vectors through a dimensionality reduction algorithm;

[0011] S3: Construct an analytical model that integrates topic modeling and user profiling to perform topic mining and user group segmentation on low-dimensional feature vectors;

[0012] S4: Based on the output of the analysis model, generate a targeted questionnaire analysis report, including conclusions on topic distribution, group differences, and potential needs.

[0013] By adopting the above technical solutions, through a multi-strategy fusion preprocessing mechanism, a joint screening feature engineering method, a fusion modeling analysis strategy, and an adaptive model optimization module, the accuracy and efficiency of questionnaire data analysis are improved.

[0014] Furthermore, the missing value imputation method of multi-strategy fusion described in S1 is as follows: first, the numerical and categorical missing fields of the questionnaire data are distinguished by the category determination module; for numerical missing fields, an improved weighted nearest neighbor imputation strategy is adopted; and for categorical missing fields, an imputation strategy based on category correlation is adopted.

[0015] By adopting the above technical solutions, data is filled differently according to data type. Numerical data is filled using a weighted nearest neighbor strategy (considering field weights), and categorical data is filled based on class correlation. Compared with traditional mean / mode filling, this method is closer to the inherent relationship of the data, has higher filling accuracy, and reduces the interference of missing data on the analysis results.

[0016] Furthermore, the outlier identification method described in S1 adopts an improved combination of box plots and density clustering. First, suspected outliers are preliminarily screened through box plots, and then the authenticity of outliers is verified through density clustering to exclude reasonable extreme values.

[0017] By adopting the above technical solution, the combination of box plot preliminary screening and DBSCAN clustering verification avoids the problem of a single statistical method misjudging reasonable extreme values ​​as outliers, improves the accuracy of outlier identification, and ensures the integrity and reliability of the dataset.

[0018] Furthermore, the joint screening mechanism based on semantic similarity and information gain described in S2 is as follows: first, redundant features corresponding to synonymous questionnaire items are eliminated by semantic similarity calculation, and then features with a contribution to the analysis target higher than a set threshold are screened by information gain calculation.

[0019] By adopting the above technical solution, redundancy is first removed through semantic similarity, and then core features are selected through information gain. This dual screening not only reduces feature dimensions and computational costs, but also ensures that the retained features contribute highly to the analysis target, thereby improving the efficiency and accuracy of subsequent modeling.

[0020] Furthermore, the dimensionality reduction algorithm described in S2 is an improved kernel principal component analysis algorithm, which improves the dimensionality reduction effect and data fidelity of high-dimensional questionnaire features by adaptively adjusting the kernel function parameters.

[0021] By adopting the above technical solution and adjusting the adaptive kernel function parameters, it is possible to adapt to the feature distribution of different questionnaire data without manual intervention, and better handle nonlinear features. Compared with traditional PCA, the dimensionality-reduced data can better retain the original information and improve the effectiveness of subsequent modeling.

[0022] Furthermore, the analysis model that integrates topic modeling and user profiling described in S3 includes an improved latent Dirichlet assignment topic model and an improved K-means clustering model. The improved latent Dirichlet assignment topic model improves the accuracy of topic mining by introducing topic relevance constraints, and the improved K-means clustering model improves the rationality of group partitioning by initializing cluster centers with density peaks.

[0023] By adopting the above technical solutions, the improved LDA solves the problem of low mining accuracy caused by the traditional LDA ignoring topic relevance; the improved K-means solves the problem of local optima caused by the random selection of initial cluster centers. The two work together to improve the accuracy of topic mining and group segmentation, laying the foundation for accurate user profiling.

[0024] Furthermore, the analysis model outputs described in S4 include topic distribution, group segmentation, and user profile results.

[0025] By adopting the above technical solution, the output results are made clearer and more understandable.

[0026] On the other hand, this application provides a data analysis system based on questionnaire surveys, including a data acquisition module, a data preprocessing module, a feature engineering module, an analysis and modeling module, and a result analysis module, wherein:

[0027] The data acquisition module is used to collect raw questionnaire data and supports the import and integration of questionnaire data from multiple online and offline channels.

[0028] The data preprocessing module is used to impute missing values ​​and identify outliers in the collected raw questionnaire data, and output a standardized questionnaire dataset.

[0029] The feature engineering module is used to perform feature selection and dimensionality reduction on standardized questionnaire datasets, and output low-dimensional feature vectors.

[0030] The analysis and modeling module is used to build analysis models that integrate topic modeling and user profiling to complete topic mining and group segmentation.

[0031] The results output module is used to generate a visual questionnaire analysis report based on the output of the analysis and modeling module.

[0032] By adopting the above technical solution, each module has a clear division of labor and works in concert to achieve fully automated processing from data collection to report output, eliminating the need for manual operation at each step and improving analysis efficiency; at the same time, the modular design facilitates maintenance and expansion.

[0033] Furthermore, it also includes a model optimization module, which is used to adaptively optimize the model parameters in the analysis and modeling module through cross-validation and grid search methods.

[0034] Furthermore, the results output module supports various visualization formats, including bar charts, pie charts, heatmaps, and radar charts, and supports the export and sharing of analysis reports.

[0035] By adopting the above technical solutions, cross-validation and grid search are used to adaptively optimize parameters, replacing manual parameter tuning, reducing the operational threshold, improving the accuracy and stability of model analysis, enhancing the ease of use and reliability of the system, supporting the import of multi-format data and duplicate data detection, improving the flexibility of data collection (adapting to the output formats of different survey channels), and automatically removing duplicate data to avoid redundant data interfering with the analysis results and ensure data quality.

[0036] The technical effects and advantages of this application are as follows:

[0037] Compared with existing technologies, this data analysis method and system based on questionnaire surveys adopts a multi-strategy fusion method for missing value imputation and outlier identification in the data preprocessing stage. Differentiated imputation strategies are used for different types of missing fields. Combined with improved box plots and density clustering to identify outliers, the data preprocessing accuracy is effectively improved and the reliability of subsequent analysis is guaranteed.

[0038] This invention proposes a joint screening mechanism of semantic similarity and information gain in the feature engineering stage, which effectively eliminates redundant features and reduces computational complexity. It adopts an improved kernel principal component analysis algorithm, which better adapts to the nonlinear characteristics of questionnaire data through adaptive kernel function parameter adjustment, thereby improving dimensionality reduction effect and data fidelity.

[0039] This invention constructs an analytical model that integrates topic modeling and user profiling. By introducing topic relevance constraints through an improved LDA topic model, the accuracy of topic mining is enhanced. By using a modified K-means clustering model to initialize cluster centers with density peaks, local optima are avoided, and the rationality of user group segmentation is improved. The user profiles constructed by combining topic distribution and group segmentation results are more accurate.

[0040] The system of this invention has an adaptive model optimization module that automatically optimizes model parameters through cross-validation and grid search without manual intervention, improving the system's ease of use and analytical accuracy. It also supports importing data in multiple formats and outputting reports in various visualization formats, enhancing the system's practicality and scalability. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the method flow of this application;

[0042] Figure 2 This is a schematic diagram of the system flow of this application.

[0043] The attached diagram is labeled as follows: 1. Data acquisition module; 2. Data preprocessing module; 3. Feature engineering module; 4. Analysis and modeling module; 5. Result analysis module; 6. Model optimization module. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] Example 1

[0046] like Figure 1 The data analysis method based on questionnaire surveys shown includes the following steps:

[0047] S1: Questionnaire data collection and preprocessing. After obtaining the raw questionnaire data, the data is first standardized in format, and the data types and field names are unified. Through a multi-strategy fusion method of missing value imputation and outlier identification, a standardized questionnaire dataset is obtained.

[0048] The multi-strategy fusion missing value imputation method is as follows: First, the category determination module distinguishes between numerical and categorical missing fields in the questionnaire data. For numerical missing fields, an improved weighted nearest neighbor imputation strategy is adopted, and for categorical missing fields, an imputation strategy based on category correlation is adopted.

[0049] The missing value imputation method for multi-strategy fusion is as follows:

[0050] S11: The category determination module distinguishes between numerical missing fields (such as age, income, etc.) and categorical missing fields (such as gender, occupation, etc.) in the questionnaire data.

[0051] S12: For numerical missing fields, an improved weighted nearest neighbor imputation strategy is adopted. The weighted Euclidean distance between the missing sample and other non-missing samples is calculated, and the mean of the k nearest samples is selected as the imputation value. The improved weighted Euclidean distance calculation formula is as follows: ,in, Missing sample; These are non-missing samples; For the first The weight of each field is determined based on the information gain of the field. Total number of fields; , Samples , In the The values ​​that can be obtained from each field;

[0052] S13: For categorically missing fields, a category association-based imputation strategy is adopted. The category association between the missing field and other non-missing categorical fields is calculated, and the category with the highest frequency among the fields with the highest association is selected as the imputation value. The formula for calculating the category association is: ,in, For fields With fields Category correlation; For fields With fields The number of samples with the same value; For fields The total number of samples; For fields The total number of samples;

[0053] The outlier identification method described in S1 employs an improved combination of box plots and density clustering. First, box plots are used to initially screen for suspected outliers, and then density clustering is used to verify the authenticity of the outliers and eliminate reasonable extreme values. Specifically:

[0054] S14: Use box plots to initially screen for suspected outliers and calculate the quartiles of numerical fields. (First quartile) (Third quartile) to determine the outlier threshold and (in (Interquartile range), samples exceeding the threshold range are marked as suspected outliers;

[0055] S15: Use the density clustering (DBSCAN) algorithm to cluster field data containing suspected outliers, and set the neighborhood radius. with minimum neighborhood sample number If the number of samples in the cluster containing the suspected outlier is less than If the number of samples in the cluster containing the suspected outlier is greater than or equal to the number of samples in the cluster, then it is determined to be a true outlier and removed; If the value is 0, it is considered a reasonable extreme value and retained.

[0056] S2: Questionnaire feature engineering, based on a joint screening mechanism of semantic similarity and information gain, extracts core features from a standardized questionnaire dataset and obtains low-dimensional feature vectors through a dimensionality reduction algorithm;

[0057] The joint screening mechanism based on semantic similarity and information gain described in S2 specifically involves: first, eliminating redundant features corresponding to synonymous questionnaire items through semantic similarity calculation; and then, selecting features whose contribution to the analysis objective exceeds a set threshold through information gain calculation.

[0058] S21: Perform semantic word segmentation on the questionnaire items, calculate the cosine semantic similarity between different items, and if the similarity is higher than a set threshold (e.g., 0.8), the items are considered synonymous. The feature corresponding to the item with the highest information entropy is retained, while other redundant features are removed. The formula for calculating cosine semantic similarity is: ,in, For the title With the topic semantic similarity; , The titles are as follows , After word segmentation, the first The weight of each word (calculated using TF-IDF). The total number of word segments;

[0059] S22: Calculate the information gain of each feature after filtering, remove features with information gain below a set threshold, and retain the core features; the formula for calculating information gain is: ,in, Features Information gain; For target variable Information entropy; Known features target variable under conditions Conditional information entropy;

[0060] The dimensionality reduction algorithm described in S2 is an improved kernel principal component analysis algorithm. It enhances the dimensionality reduction effect and data fidelity of high-dimensional questionnaire features through adaptive kernel function parameter adjustment; specifically:

[0061] The improved kernel principal component analysis algorithm uses a Gaussian kernel function, whose parameters are... The Gaussian kernel function is adaptively determined through cross-validation and is specifically as follows: ,in, For the sample and The kernel function value; For kernel function parameters, For the sample and Euclidean distance;

[0062] S3: Construct an analytical model that integrates topic modeling and user profiling to perform topic mining and user group segmentation on low-dimensional feature vectors;

[0063] The analytical model integrating topic modeling and user profiling described in S3 includes an improved latent Dirichlet assignment topic model and an improved K-means clustering model. The improved latent Dirichlet assignment topic model improves topic mining accuracy by introducing topic relevance constraints, and its objective function is: ,in, This refers to the number of questionnaire samples. For the number of words, As the main quantity, For the first In the nth sample The number of times each word appears, For the first The sample at the th The probability distribution over each topic For the first The theme in The probability distribution over each word , For hyperparameters, The topic relevance constraint coefficient, Theme With the topic The correlation (calculated using cosine similarity);

[0064] The improved K-means clustering model enhances the rationality of population partitioning by initializing cluster centers using density peaks. Specifically, it calculates the local density and relative distance of each sample, selects samples with high local density and large relative distance as initial cluster centers, and then iteratively updates the cluster centers until they stabilize. The formula for calculating local density is: ,in, For the first Local density of a sample The total number of samples, For the sample With sample European distance, To cut off the distance, For indicator functions, when hour, ,otherwise The formula for calculating relative distance is as follows: ,in, For the first The relative distance of the samples, i.e., all local densities higher than the first sample. The sample set of the nth sample is related to the nth sample. The distance between each sample and its nearest sample;

[0065] S4: Based on the output of the analysis model, generate a targeted questionnaire analysis report, including conclusions on topic distribution, group differences, and potential needs.

[0066] The analysis model described in S4 outputs results including topic distribution, group segmentation, and user profiles.

[0067] Example 2

[0068] Please see Figure 2 A data analysis system based on questionnaire surveys includes a data acquisition module 1, a data preprocessing module 2, a feature engineering module 3, an analysis and modeling module 4, and a results analysis module 5, wherein:

[0069] Data acquisition module 1 is used to collect raw questionnaire data and supports the import and integration of questionnaire data from multiple online and offline channels;

[0070] Data preprocessing module 2 is used to fill in missing values ​​and identify outliers in the collected raw questionnaire data, and output a standardized questionnaire dataset;

[0071] Feature engineering module 3 is used to perform feature filtering and dimensionality reduction on the standardized questionnaire dataset and output a low-dimensional feature vector.

[0072] Analysis and modeling module 4 is used to build an analysis model that integrates topic modeling and user profiling to complete topic mining and group segmentation;

[0073] Result Output Module 5 is used to generate a visualized questionnaire analysis report based on the output of the analysis and modeling module.

[0074] Each module has a clear division of labor and works in synergy to achieve fully automated processing from data collection to report output, eliminating the need for manual operation at each step and improving analysis efficiency; at the same time, the modular design facilitates maintenance and expansion.

[0075] It also includes a model optimization module 6, which is used to adaptively optimize the model parameters in the analysis and modeling module through cross-validation and grid search methods.

[0076] The result output module 5 supports multiple visualization formats, including bar charts, pie charts, heat maps, and radar charts, and also supports the export and sharing of analysis reports.

[0077] By using cross-validation and grid search to adaptively optimize parameters, manual parameter tuning is replaced, reducing the operational threshold while improving the accuracy and stability of model analysis, enhancing the ease of use and reliability of the system. It supports the import of data in multiple formats and the detection of duplicate data, improving the flexibility of data collection (adapting to the output formats of different survey channels), and automatically removing duplicate data to avoid redundant data interfering with the analysis results, thus ensuring data quality.

[0078] Example 3

[0079] This embodiment uses the user needs survey questionnaire data analysis of an e-commerce platform as an example to explain in detail the implementation process of the data analysis method based on questionnaire survey of the present invention.

[0080] Step 1: Questionnaire Data Collection and Preprocessing

[0081] Raw questionnaire data was collected through online questionnaire channels on e-commerce platforms, resulting in 1,000 user questionnaires with 20 fields, including 8 numerical fields (age, income, shopping frequency, etc.) and 12 categorical fields (gender, occupation, shopping preferences, etc.). The raw data was standardized and the fields were uniformly named "User ID, Gender, Age, Occupation, Income, Shopping Frequency, Product Type Preference, etc."

[0082] A multi-strategy fusion method was used to handle missing values: Statistical analysis showed that 5% of the data contained missing values, with 15 missing values ​​in the "Income" field and 20 missing values ​​in the "Product Type Preference" field. For "Income" (numerical), an improved weighted nearest neighbor imputation strategy was adopted, with k=5. The weights were determined based on the information gain of each field, and the weighted Euclidean distance between the missing sample and other non-missing samples was calculated. The mean income of the 5 nearest samples was selected as the imputation value. For "Product Type Preference" (categorical), a category association-based imputation strategy was adopted. The category association between this field and the "Shopping Frequency" field was calculated to be R=0.75 (the highest). The product type preference "Daily Necessities" corresponding to the most frequent "3-5 times per week" in the "Shopping Frequency" field was selected as the imputation value.

[0083] An improved method combining box plots and density clustering was used to identify outliers. Taking the "age" field as an example, Q1=22, Q3=38, IQR=16 were calculated, and the outlier thresholds were 22-1.5×16=-4 (taken as 0) and 38+1.5×16=62. Ten samples with an age greater than 62 were initially selected as suspected outliers. The DBSCAN algorithm was used to cluster the "age" field data, with ε=5 and MinPts=3. The clustering results showed that the number of samples in the clusters containing the 10 suspected outliers was 2 (less than MinPts=3), and they were determined to be true outliers and removed. Finally, a standardized dataset of 990 questionnaires was obtained.

[0084] Step 2: Questionnaire Feature Engineering

[0085] The core features were extracted based on a joint screening mechanism of semantic similarity and information gain: Semantic word segmentation was performed on 20 questionnaire items, and the semantic similarity between "product type preference" and "common shopping categories" was calculated to be 0.85 (higher than the threshold of 0.8), which were determined to be synonymous items. The "product type preference" feature with higher information entropy was retained, while the "common shopping categories" feature was removed. The information gain of the 19 features after screening was calculated, and a threshold of 0.1 was set. The two features "user education level" and "city level of residence" with information gain lower than 0.1 were removed, and 17 core features were retained.

[0086] An improved kernel principal component analysis algorithm was used to reduce the dimensionality of 17 core features: a Gaussian kernel function was used, and the kernel function parameter σ=0.8 was determined through 5-fold cross-validation; kernel mapping and principal component extraction were performed on the core features, and the top 5 principal components with a cumulative variance contribution rate of 90% were selected as low-dimensional feature vectors.

[0087] Step 3: Construct an analytical model that integrates topic modeling and user personas.

[0088] An improved LDA topic model was used for topic mining: the number of topics was set to K=3, the hyperparameters were α=0.1, β=0.01, and the topic relevance constraint coefficient was λ=0.05. The objective function was optimized by Gibbs sampling algorithm and converged after 1000 iterations, resulting in 3 core topics: topic 1 (high-frequency consumption of daily necessities), topic 2 (preference for electronic products), and topic 3 (demand for beauty and skincare products).

[0089] An improved K-means clustering model was used to segment the user groups: the local density and relative distance of 990 samples were calculated, and the three samples with high local density and large relative distance were selected as the initial cluster centers; the number of clusters k=3 was set, and the cluster centers were iteratively updated until they stabilized, resulting in three user groups: Group 1 (young white-collar workers, 22-30 years old, with moderate income), Group 2 (middle-aged housewives, 31-45 years old, with stable income), and Group 3 (elderly consumers, 46-60 years old, with moderate to low income).

[0090] User profiles were constructed by combining the theme distribution and group segmentation results: The theme distribution of group 1 was theme 2 (65%), theme 1 (25%), and theme 3 (10%), and the profile label was "young white-collar workers who prefer electronic products"; The theme distribution of group 2 was theme 1 (70%), theme 3 (20%), and theme 2 (10%), and the profile label was "middle-aged housewives who have a strong need for daily necessities"; The theme distribution of group 3 was theme 1 (80%), theme 2 (5%), and theme 3 (15%), and the profile label was "elderly consumers who frequently consume daily necessities".

[0091] Step 4: Generate an analysis report

[0092] Based on the above analysis results, an e-commerce platform user demand survey and analysis report is generated, including: the percentage distribution of 3 core themes (bar chart), the characteristic differences of 3 user groups (radar chart), the theme preferences of each group (heat map), and potential demand conclusions (such as Group 1 having a higher potential demand for high-end electronic products, and Group 2 having a strong demand for cost-effective daily necessities, etc.); the report supports exporting in PDF format and sharing within the platform.

[0093] Example 4

[0094] This embodiment provides a data analysis system based on a questionnaire survey. The system corresponds to the analysis method in Embodiment 3 and includes a data acquisition module, a data preprocessing module, a feature engineering module, an analysis and modeling module, a result output module, and a model optimization module.

[0095] Data Acquisition Module 1: Collects 1,000 original questionnaires by connecting to the questionnaire system of an e-commerce platform, supporting import in CSV format; detects and removes 5 duplicate data by calculating the sample hash value.

[0096] Data preprocessing module 2: Built-in category determination submodule, which determines that "age" and "income" are numerical fields, and "gender" and "occupation" are categorical fields; realizes multi-strategy fusion for missing value imputation and outlier identification, and outputs a standardized questionnaire dataset of 990 questionnaires.

[0097] Feature Engineering Module 3: It has a built-in semantic similarity calculation submodule (based on TF-IDF and cosine similarity) and information gain calculation submodule to complete feature selection and retain 17 core features; it also completes dimensionality reduction through an improved KPCA algorithm and outputs a 5-dimensional feature vector.

[0098] Analysis and Modeling Module 4: It includes an improved LDA topic model submodule and an improved K-means clustering submodule to complete topic mining and group segmentation, and output topic distribution and user group segmentation results.

[0099] Model optimization module 6: Through 5-fold cross-validation and grid search, the α, β, and λ parameters of the improved LDA model and the k and d_c parameters of the improved K-means clustering model are optimized to determine the optimal parameter combination.

[0100] Output module 5: Generates analysis reports containing various visualization charts, supporting PDF export and sharing within the platform.

[0101] The system in this embodiment operates stably, and its analysis accuracy and efficiency are improved compared to existing systems. It can accurately identify the needs and preferences of e-commerce platform users, providing a reliable basis for product recommendations and marketing strategy formulation for the platform.

Claims

1. A data analysis method based on questionnaire surveys, characterized in that, Includes the following steps: S1: Questionnaire data collection and preprocessing. After obtaining the raw questionnaire data, a standardized questionnaire dataset is obtained through a multi-strategy fusion method for imputing missing values ​​and outlier identification. S2: Questionnaire feature engineering, based on a joint screening mechanism of semantic similarity and information gain, extracts core features from a standardized questionnaire dataset and obtains low-dimensional feature vectors through a dimensionality reduction algorithm; S3: Construct an analytical model that integrates topic modeling and user profiling to perform topic mining and user group segmentation on low-dimensional feature vectors; S4: Based on the output of the analysis model, generate a targeted questionnaire analysis report, including conclusions on topic distribution, group differences, and potential needs.

2. The data analysis method based on questionnaire surveys according to claim 1, characterized in that: The missing value imputation method of multi-strategy fusion described in S1 is as follows: First, the numerical and categorical missing fields of the questionnaire data are distinguished by the category determination module. For numerical missing fields, an improved weighted nearest neighbor imputation strategy is adopted, and for categorical missing fields, an imputation strategy based on category correlation is adopted.

3. The data analysis method based on questionnaire surveys according to claim 1, characterized in that: The outlier identification method described in S1 uses an improved combination of box plots and density clustering. First, box plots are used to preliminarily screen suspected outliers, and then density clustering is used to verify the authenticity of the outliers and exclude reasonable extreme values.

4. The data analysis method based on questionnaire surveys according to claim 3, characterized in that: The joint screening mechanism based on semantic similarity and information gain described in S2 is as follows: First, redundant features corresponding to synonymous questionnaire items are eliminated by semantic similarity calculation, and then features with a contribution to the analysis target higher than a set threshold are selected by information gain calculation.

5. The data analysis method based on questionnaire surveys according to claim 1, characterized in that: The dimensionality reduction algorithm described in S2 is an improved kernel principal component analysis algorithm. By adaptively adjusting the kernel function parameters, it improves the dimensionality reduction effect and data fidelity of high-dimensional questionnaire features.

6. The data analysis method based on questionnaire surveys according to claim 5, characterized in that: The analysis model that integrates topic modeling and user profiling described in S3 includes an improved latent Dirichlet assignment topic model and an improved K-means clustering model. The improved latent Dirichlet assignment topic model improves the accuracy of topic mining by introducing topic relevance constraints, and the improved K-means clustering model improves the rationality of group partitioning by initializing cluster centers with density peaks.

7. The data analysis method based on questionnaire surveys according to claim 6, characterized in that: The analysis model described in S4 outputs results including topic distribution, group segmentation, and user profiles.

8. A data analysis system based on questionnaire surveys, employing the data analysis method based on questionnaire surveys as described in any one of claims 1-7, characterized in that, It includes a data acquisition module (1), a data preprocessing module (2), a feature engineering module (3), an analysis and modeling module (4), and a results analysis module (5), wherein: The data acquisition module (1) is used to collect raw questionnaire data and supports the import and integration of online and offline questionnaire data from multiple channels. The data preprocessing module (2) is used to fill in missing values ​​and identify outliers in the collected raw questionnaire data and output a standardized questionnaire dataset. The feature engineering module (3) is used to perform feature filtering and dimensionality reduction on the standardized questionnaire dataset and output a low-dimensional feature vector. The analysis and modeling module (4) is used to build an analysis model that integrates topic modeling and user profiling to complete topic mining and group segmentation; The results output module (5) is used to generate a visual questionnaire analysis report based on the output results of the analysis and modeling module.

9. A data analysis system based on questionnaire surveys according to claim 8, characterized in that: It also includes a model optimization module (6), which is used to adaptively optimize the model parameters in the analysis and modeling module through cross-validation and grid search methods.

10. A data analysis system based on a questionnaire survey according to claim 8, characterized in that: The result output module (5) supports a variety of visualization formats, including bar charts, pie charts, heat maps and radar charts, and supports the export and sharing of analysis reports.

Images