Method and system for studying and judging population data

Abstract

The invention discloses a population data research and judgment method and system, and the method comprises the steps: collecting multi-source data of a plurality of data formats according to a collection rule, and carrying out the preprocessing of the multi-source data, and obtaining standardized target multi-source data; performing data fusion on the target multi-source data to obtain a unified population data set, and performing population feature learning and extraction on the population data set by using graph embedding and a graph neural network to extract population feature vectors; inputting the population feature vector into an analysis, research and judgment model for analysis and prediction to obtain a prediction result for predicting a population data transformation trend, and displaying the prediction result through an sql statement; multi-source data changes are monitored in real time, and the analysis, research and judgment model and the corresponding prediction result are dynamically updated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, more particularly, to a population data research and judgment method and system. BACKGROUND

[0002] In the prior art, population data research and judgment mainly relies on a single data source (such as population census data) for population data analysis, and uses traditional statistical models for population prediction. These methods have defects such as single data source, limited model capability, lack of multi-source data fusion, insufficient real-time performance, poor visualization and interaction, which result in low analysis accuracy and inaccurate prediction results, and are difficult to support complex decisions.

[0003] Traditional population data research and judgment methods usually only rely on population census data, which is authoritative and comprehensive, but has a long update cycle, generally once every ten years, and is difficult to reflect real-time changes in population. In addition, the accuracy of population census data also has certain limitations, for example, the data at the township (street) level is relatively accurate, but at a more detailed scale (such as community, plot, etc.), it cannot provide detailed population distribution information.

[0004] Data from different sources (such as mobile signaling data, social media data, traffic flow data, etc.) can reflect the distribution, flow and behavior characteristics of the population from multiple angles, but traditional methods often cannot effectively integrate these multi-source data, so as to fully utilize the rich information in the data to improve the analysis accuracy. SUMMARY

[0005] In view of at least one defect or improvement demand of the prior art, the present application provides a population data research and judgment method and system, which solves the problem of low accuracy and poor visualization and interaction of population analysis in the prior art, performs data fusion and feature extraction after multi-source data access, displays the analysis and prediction results through a visualization tool after analysis and prediction, and automatically generates a research and judgment report, thereby improving the prediction accuracy and enhancing the visualization degree and interaction.

[0006] To achieve the above object, according to a first aspect of the present application, a population data research and judgment method is provided, which comprises: collecting multi-source data in multiple data formats according to a collection rule, pre-processing the multi-source data to obtain target multi-source data after standardization; obtaining a unified population data set after data fusion of the target multi-source data, learning and extracting population characteristics of the population data set by using graph embedding and graph neural network, and extracting a population characteristic vector; inputting the population characteristic vector into an analysis and judgment model for analysis and prediction to obtain a prediction result for predicting the population data change trend, and displaying the prediction result through a sql statement; and monitoring the multi-source data changes in real time, dynamically updating the analysis and judgment model and the corresponding prediction result.

[0007] In an example embodiment, the collecting multi-source data in multiple data formats according to a collection rule, and pre-processing the multi-source data to obtain target multi-source data after standardization comprises: data cleaning of the multi-source data in multiple data formats, the multi-source data in multiple data formats including structured data, unstructured data and semi-structured data, the data cleaning including missing value processing, abnormal value processing and duplicate data processing; data alignment of the multi-source data after data cleaning, the data alignment including key value merging, time alignment, space alignment and attribute alignment.

[0008] In an example embodiment, the obtaining a unified population data set after data fusion of the target multi-source data, learning and extracting population characteristics of the population data set by using graph embedding and graph neural network, and extracting a population characteristic vector comprises: uniformly identifying each entity in the multi-source data; creating a node and adding an attribute for each entity; creating an edge and adding an attribute for the relationship between entities; based on the constructed graph structure, using a graph embedding algorithm Node2Vec to convert the created node and edge into a low-dimensional population characteristic vector.

[0009] In an example embodiment, before the inputting the population characteristic vector into an analysis and judgment model for analysis and prediction, the method further comprises: determining the analysis and judgment model according to the nature of the prediction problem, wherein the nature of the prediction problem includes regression problem, classification problem, clustering problem and time series analysis problem; and determining the target parameter combination of the analysis and judgment model by using grid search technology combined with cross-validation method.

[0010] In an example embodiment, the inputting the population feature vector into the analysis and judgment model for analysis and prediction to obtain a prediction result for predicting a population data transformation trend and displaying the prediction result through a SQL statement comprises: setting dynamic data by using various components, and displaying the prediction result through a SQL statement; and binding an API interface, and displaying data information returned by the API interface.

[0011] In an example embodiment, the real-time monitoring of the multi-source data changes and the dynamic updating of the analysis and judgment model and the corresponding prediction result comprises: real-time access to a data stream through a message queue Kafka; and / or, reading incremental update data of a data table through a database log stream; and / or, setting rules and thresholds through a timing task, and periodically detecting multi-source data changes.

[0012] In an example embodiment, the method further comprises: reading and accessing a real-time data stream through a message queue or a database log stream; performing normalization processing on each sample, and calculating a gradient according to a current model parameter and sample data; updating the model parameter using the gradient, and periodically saving the updated model.

[0013] According to a second aspect of the present application, a population data analysis and judgment system is also provided, comprising: a data access module configured to access multi-source data in various data formats collected according to collection rules; a preprocessing module configured to preprocess the multi-source data to obtain standardized target multi-source data; a data fusion module configured to fuse the target multi-source data to obtain a unified population data set, and perform population feature learning and extraction on the population data set by using graph embedding and graph neural network to extract a population feature vector; an analysis module configured to input the population feature vector into an analysis and judgment model for analysis and prediction to obtain a prediction result for predicting a population data transformation trend; a visualization module configured to display the prediction result through a SQL statement; and an optimization module configured to real-time monitor multi-source data changes and dynamically update the analysis and judgment model and the corresponding prediction result.

[0014] In an example embodiment, the system further comprises: an API management module configured to encapsulate data into an API interface.

[0015] According to a third aspect of the present application, an electronic device is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above population data analysis and judgment method through the computer program.

[0016] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0017] (1) The application provides a population data research and judgment method, according to the collection rule, a plurality of data formats of multi-source data are collected, the multi-source data are preprocessed to obtain standardized target multi-source data, the target multi-source data are fused to obtain a unified population data set, population feature learning and extraction are performed on the population data set by using graph embedding and graph neural network, and a population feature vector is extracted; the population feature vector is input into an analysis and judgment model for analysis and prediction, a prediction result for predicting population data change trend is obtained, and the prediction result is displayed through a sql statement; the multi-source data changes are monitored in real time, the analysis and judgment model and the corresponding prediction result are dynamically updated; the multi-source data are processed and fused, the high coverage and data quality of the data are ensured, the key features are extracted from the multi-source data, different machine learning algorithms are selected based on different scenes, and population data analysis and prediction are performed.

[0018] (2) The application supports rich visualization scenarios. By embedding a plurality of different types of component libraries, rich and diverse visualization large-screen display pages can be conveniently and quickly realized through dragging and binding data, and customized research and judgment reports can be generated. The model is updated regularly, and the model is optimized according to user feedback data. The data are encapsulated as API and provided for third-party calling, and perfect API access permission control and access strategy control are supported. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0020] Figure 1 A flowchart of an optional population data research and judgment method provided by the embodiments of the present application is shown in the figure.

[0021] Figure 2 An architecture diagram of an optional population data research and judgment system provided by the embodiments of the present application is shown in the figure.

[0022] Figure 3 A structure diagram of an optional electronic device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0023] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0024] The terms "first", "second", "third" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, and are not used to describe a particular order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0025] According to an aspect of an embodiment of the present application, a population data analysis method is provided. The following will be described in combination with Figure 1 The population data analysis method provided by the embodiment of the present application is described.

[0026] Figure 1 is a flowchart of an optional population data analysis method provided by the embodiment of the present application, as shown in Figure 1 The flowchart of the method can include the following steps:

[0027] S102, collecting multi-source data in multiple data formats according to a collection rule, and pre-processing the multi-source data to obtain standardized target multi-source data;

[0028] S104, obtaining a unified population data set after data fusion of the target multi-source data, performing population feature learning and extraction on the population data set by using graph embedding and graph neural network, and extracting a population feature vector;

[0029] S106, inputting the population feature vector into an analysis and judgment model for analysis and prediction, obtaining a prediction result for predicting population data transformation trend, and displaying the prediction result through a sql statement;

[0030] S108, real-time monitoring of multi-source data changes, dynamic updating of the analysis and judgment model and the corresponding prediction result.

[0031] The population data analysis method provided by the embodiment of the present application is suitable for the scene of population data analysis, and realizes accurate analysis and prediction of population data through multi-source data fusion and machine learning technology.

[0032] In this embodiment, the data sources of the accessed multi-source data include but are not limited to: census data (age, gender, occupation, education level, etc.), social media data (user behavior, interest, geographic location), mobile device data (GPS, cell phone signal, travel trajectory), sensor data (cameras in smart cities, traffic sensors), e-commerce data (consumption habits, purchasing power), public health data (disease distribution, health behavior), geographic data (latitude and longitude, urban / rural division, population density, etc.), economic data (GDP, employment rate, etc.), policy data (birth policy, immigration policy, etc.), and the like.

[0033] Optionally, different types of data can be collected using different collection rules. Flexible collection rules can be developed according to factors such as data importance and update frequency. For example, for census data, a full collection is performed every ten years, and during this period, the change data of key indicators is collected every year; for mobile signaling data, real-time information is collected once an hour; for social media data, the posting and interaction data of popular regions and age groups are filtered according to topic popularity every day.

[0034] After data fusion of the target multi-source data, a unified population data set is obtained, and population feature learning and extraction are performed on the population data set using graph embedding and graph neural networks to extract a population feature vector. The feature extraction method can include:

[0035] Basic feature extraction: directly using original features (such as age, gender, income, etc.), or extracting time features from timestamps (extracting year, month, quarter, etc. from date).

[0036] Statistical feature extraction: performing aggregate statistics on data (such as calculating the average income, highest income, and lowest income of each region), or performing sliding window statistics on time series data (such as calculating the population growth rate over the past 5 years), or extracting distribution features of data (such as calculating the mean, variance, skewness, etc. of age distribution).

[0037] Geographic feature extraction: extracting spatial features from geographic data (such as calculating population density (population / area), distance to the nearest city, etc.), or dividing regional features according to geographic information (such as dividing regions into "east", "west", "center", etc.).

[0038] Text feature extraction: if the data contains text information, natural language processing (NLP) techniques can be used to extract features, such as using TF-IDF or word embedding (Word Embedding) to extract text features.

[0039] Graph structure feature extraction: extracting features of nodes and edges from graph structure, such as centrality, clustering coefficient.

[0040] Further, the population feature vector is input into the analysis and judgment model for analysis and prediction to obtain a prediction result for predicting the population data transformation trend, various components support dynamic data setting, and the analysis and prediction result of the analysis module is displayed through a SQL statement, such as displaying the geographical distribution of the population through a heat map, a point density map, and the like.

[0041] Preferably, the multi-source data changes can be monitored in real time, and the analysis and judgment model and the corresponding prediction result are dynamically updated. The model is version managed, and can be dynamically switched and rolled back. The model retraining is triggered periodically or according to data changes, so as to ensure that the model adapts to the latest data.

[0042] Through the above steps S102 to S108, the multi-source data in multiple data formats is collected according to the collection rule, the multi-source data is preprocessed to obtain standardized target multi-source data, the target multi-source data is fused to obtain a unified population data set, the population feature learning and extraction are performed on the population data set by using graph embedding and graph neural network to extract a population feature vector, the population feature vector is input into the analysis and judgment model for analysis and prediction to obtain a prediction result for predicting the population data transformation trend, and the prediction result is displayed through a SQL statement. The multi-source data changes are monitored in real time, and the analysis and judgment model and the corresponding prediction result are dynamically updated. The problem of low accuracy of population analysis and poor visualization and interactivity in the prior art is solved. Through data fusion and feature extraction after multi-source data access, the analysis and prediction result is displayed through a visualization tool after analysis and prediction, and a judgment report is automatically generated, so that the prediction accuracy is improved, and the visualization degree and interactivity are enhanced.

[0043] In one example embodiment, the multi-source data in multiple data formats is collected according to the collection rule, and the multi-source data is preprocessed to obtain standardized target multi-source data, including:

[0044] S11, the multi-source data in multiple data formats is data cleaned, the multi-source data in multiple data formats includes structured data, unstructured data and semi-structured data, and the data cleaning includes processing missing values, processing abnormal values and processing duplicate data;

[0045] S12, the multi-source data after data cleaning is data aligned, and the data alignment includes key value merging, time alignment, space alignment and attribute alignment.

[0046] In this embodiment, the data access module supports multiple data formats, including structured data (such as tables), unstructured data (such as text, images) and semi-structured data (such as JSON, XML), supports accessing databases, Kafka, API interfaces and uploading files.

[0047] Data cleaning for multi-source data access can include the following steps:

[0048] Handling missing values: If the proportion of missing values is high and has little impact on analysis, directly delete the relevant records or fields; if the missing value is a key field, use a fixed value (such as 0, mean, median, mode) to fill in the missing value; for time series or continuous data, use interpolation method (such as linear interpolation) to interpolate the missing data; or mark the missing value as a special value (such as "Unknown") for subsequent analysis.

[0049] Handling outliers: If outliers are obviously error data, directly delete them; or correct outliers according to context, such as correcting records with age of 200 years to 20 years; if outliers are real data, they can be retained and analyzed separately.

[0050] Handling duplicate data: Use unique identifiers (such as ID) or field combinations to identify duplicate records, such as identifying records with the same name, age, and address, and retaining the first or last record and deleting other duplicate records; if duplicate records contain complementary information, they can be merged.

[0051] Data standardization: Convert data from different data sources to a unified format and unit for subsequent analysis. Unified data format, such as converting date format to "YYYY-MM-DD"; unified data unit, such as converting income unit to "yuan" or "thousand yuan"; unified classification variable coding, such as coding "gender" field to "male" and "female".

[0052] Save the processed data in the data warehouse for subsequent steps.

[0053] Cleaned data is aligned again:

[0054] Merge based on key value: Merge different data sets through common keys (such as region ID, timestamp), such as merging population data and geographic data through "region ID".

[0055] Time alignment: If the data is a time series, unify the timestamps of different data sources to ensure timestamp alignment, such as aligning annual population data with annual economic data.

[0056] Spatial alignment: If the data contains geographic information, unify the geographic position information to the same coordinate system to ensure spatial range consistency, such as aligning city-level population data with city-level environmental data.

[0057] Attribute alignment: Map attribute fields from different data sources, such as aligning "age" and "birth year" to "age".

[0058] Through the embodiment, multi-source data in multiple data formats are accessed and processed, ensuring high coverage and data quality.

[0059] In an example embodiment, the target multi-source data is data fused to obtain a unified population data set, and a graph embedding and a graph neural network are used to learn and extract population features from the population data set to obtain a population feature vector.

[0060] S21, each entity in the multi-source data is uniformly identified;

[0061] S22, a node is created for each entity and an attribute is added;

[0062] S23, an edge is created for the relationship between entities and an attribute is added;

[0063] S24, based on the constructed graph structure, the created nodes and edges are converted into low-dimensional population feature vectors using the graph embedding algorithm Node2Vec.

[0064] In the embodiment, the preprocessed data is data fused, and the multi-source data is integrated into a unified data set and features are extracted. Exemplarily, the multi-source population data is modeled as a graph structure, the relationship between entities (such as individuals, families, and communities) is captured, and the feature representation of nodes and edges is extracted. By defining nodes, edges, and features, and using graph embedding and graph neural network for feature learning, the relationship and pattern in the data are effectively captured, providing support for subsequent analysis and application.

[0065] Specifically, the data is arranged, and the entities such as individuals, families, and communities in different data sources are uniformly identified to ensure consistent data formats (such as timestamps and geographic locations). The nodes are defined, wherein the individual nodes: each individual is a node, and the attributes include age, gender, occupation, etc. The family node: each family is a node, and the attributes include the number of family members, income level, etc. The community node: each community is a node, and the attributes include community population density, infrastructure, etc. The edges are defined, wherein the kinship: the edge between individuals and family members, and the attributes include the relationship type (such as father-son, husband-wife). The neighbor relationship: the edge between individuals and neighbors, and the attributes include the interaction frequency. The colleague relationship: the edge between individuals and colleagues, and the attributes include the working time.

[0066] Further, a graph structure is constructed, and the specific steps include:

[0067] Create nodes: Create nodes for each entity (individual, family, community) and add attributes. For example, individual node Person(id=1, age=25, gender='M'), family node Family(id=101, income=50000), community node Community(id=201, population_density=2000)

[0068] Create edges: Create edges for relationships between entities and add attributes. For example, family relationship: (Person1)-[:FAMILY{relation:'father'}]->(Person2), neighbor relationship: (Person1)-[:NEIGHBOR{interaction:5}]->(Person2)

[0069] Store the constructed graph structure in a graph database or in memory for subsequent querying and analysis.

[0070] Exemplarily, feature extraction is performed, and a graph embedding algorithm Node2Vec is used to convert nodes and edges into low-dimensional vectors, including:

[0071] (1) Random walk: Generate a random walk sequence from each node.

[0072] (2) Train Skip-gram model: Take the walk sequence as input and learn node embeddings.

[0073] (3) Get node vectors: Represent each node as a low-dimensional vector.

[0074] Preferably, graph structure optimization is performed, i.e., dynamically adding or deleting nodes and edges according to new data.

[0075] In an exemplary embodiment, before the population feature vector is input into the analysis and judgment model for analysis and prediction, the method further comprises:

[0076] S31, determining the analysis and judgment model according to the nature of the prediction problem, wherein the nature of the prediction problem includes regression problem, classification problem, clustering problem and time series analysis problem;

[0077] S32, using grid search technology combined with cross-validation method to determine the target parameter combination of the analysis and judgment model.

[0078] In the embodiments of the present application, machine learning algorithms are used to perform deep analysis and prediction on population data, and the analyzed and predicted data results are saved in a data warehouse.

[0079] Optionally, according to the analysis target, the model is adapted, and according to the nature of the problem, a suitable machine learning algorithm can be selected: for regression problems such as predicting income, population growth, linear regression, decision tree regression, random forest regression, and support vector regression (SVR) are supported. For classification problems such as predicting health status, support logistic regression, decision tree classification, random forest classification, and support vector machine (SVM). For clustering problems such as population grouping and regional division, K-means clustering, hierarchical clustering, and DBSCAN are supported. For time series analysis problems such as population growth prediction, LSTM neural networks and ARIMA models are supported.

[0080] Using grid search technology combined with cross-validation method, the key parameters of the selected model are optimized. For example, for the random forest model, the number of trees, maximum depth, minimum leaf node number and other parameters are searched; for neural networks, learning rate, number of hidden layer neurons, activation function and other parameters are optimized. The average score of cross-validation (such as accuracy, mean square error and other evaluation indicators) is used as the measurement standard to find the optimal parameter combination (i.e. target parameter combination) to ensure the best model performance.

[0081] The processed population data is divided into a training set in a ratio of 70%, which is used to learn the patterns and rules in the population data; 15% is used as a validation set to periodically verify the model performance during training to prevent overfitting and adjust the model parameters; the remaining 15% is used as a test set to evaluate the generalization ability of the model and ensure that the model performs well on unseen data.

[0082] The training set data is used to train the selected machine learning model, and the corresponding training algorithm is used according to the model type, and the model loss function (such as mean square error, cross-entropy, etc.) is gradually reduced by continuously iterating according to the set parameters and optimization algorithm until the convergence condition is reached (such as the loss function value is lower than the set threshold or the iteration number reaches the upper limit).

[0083] On the validation set and test set, according to the analysis target, select the appropriate evaluation indicators. For population classification tasks, use accuracy (Accuracy), recall (Recall), F1 value and other indicators to measure the model's ability to distinguish different population groups; in population flow prediction, root mean square error (RMSE), mean absolute error (MAE) are used to evaluate the deviation between the predicted value and the actual value; for population consumption behavior modeling, in addition to using mean square error to evaluate the accuracy of predicted consumption value, ROC curve (used to evaluate the classification model's ability to distinguish consumption preferences) and other indicators can be used to comprehensively evaluate the model performance. By comparing the evaluation indicators of different models on the same data set, the optimal model is determined.

[0084] Using machine learning algorithm to analyze population data can automatically capture the numerous variables and nonlinear relationships between variables in population data, and discover hidden patterns and rules in data to provide objective and data-supported basis for decision-making. This avoids the limitations of relying solely on experience or intuition for decision-making, and improves the scientificity and accuracy of decision-making.

[0085] In an example embodiment, the inputting the population feature vector into the analysis and judgment model for analysis and prediction to obtain a prediction result for predicting population data transformation trend, and displaying the prediction result through a sql statement comprises:

[0086] S41, setting dynamic data by using various components, and displaying the prediction result through a sql statement;

[0087] S42, binding an API interface, and displaying data information returned by the API interface.

[0088] In the embodiment of the application, a visual large screen is supported to be designed by customizing, and population analysis data is displayed through intuitive charts, maps and interactive interfaces. A rich component library is embedded in the system, and the visual large screen can be designed conveniently and quickly by being dragged to a canvas: charts include column chart, line chart, pie chart, scatter chart, bar chart, radar chart, heat map, relationship chart, etc. Maps can select provinces, and support access to Gaode map and offline map. Tables include carousel table, paging table and custom table. Controls include label selector, time selector, drop-down selector, input box, etc.

[0089] Various components support setting dynamic data, and analysis and prediction results of the analysis module are displayed through a sql statement, such as displaying geographical distribution of population through a heat map or a point density map, displaying population change trend (such as birth rate, death rate, migration rate, etc.) through a line chart or a column chart, and displaying population structure information (such as age, gender, education level, etc.) through a pie chart or a stacked column chart. In addition to the sql statement, the dynamic data also supports binding an API interface, and displays related data information returned by the API interface.

[0090] The visual module supports rich interactive functions, and can dynamically filter data through various controls (label selector, time selector, drop-down selector, input box, etc.) through time, region, population attribute and other dimensions; supports data drilling from macro to micro by setting animation and component association, for example, from national to provincial, municipal and county levels; allows adding annotations on a map or a chart to highlight key information; multiple charts or maps can be linked, and operations in one view will be updated synchronously in other views.

[0091] The visualization module also supports automatic generation of visualization research and judgment reports according to the dimensions and indicators selected by the user. The research and judgment report can be selected in PDF or webpage format. At the same time, the built-in hot template is designed, which only needs to be simply data bound to directly perform visual data display.

[0092] In one example embodiment, the real-time monitoring of multi-source data changes dynamically updates the analysis and judgment model and the corresponding prediction results, including:

[0093] S51, accessing data stream in real time through message queue Kafka; and / or,

[0094] S52, reading incremental update data of the data table through database log stream; and / or,

[0095] S53, setting rules and thresholds through a timing task, and periodically detecting multi-source data changes.

[0096] It should be noted that reading incremental update data of the data table through the database log stream includes, for example, starting Binlog in Mysql, starting Logminer in Oracle database, starting CDC (change data capture) in Sql serve, and starting CDC in Oceanbase.

[0097] In one example embodiment, the method further includes:

[0098] S61, accessing real-time data stream through message queue or database log stream;

[0099] S62, performing normalization processing on each sample, and calculating gradient according to current model parameter and sample data;

[0100] S63, updating model parameter using the gradient, and periodically saving the updated model.

[0101] In the embodiment of the application, the online learning algorithm (online gradient descent OGD) updates the model parameter in real time, and the specific steps include data stream access: accessing real-time data stream through message queue or database log stream. Data preprocessing: performing standardization or normalization processing on each sample. Gradient calculation: calculating gradient according to current model parameter and sample data. Parameter update: updating model parameter using the gradient. Model saving: periodically saving the updated model.

[0102] The optimization module supports version management of the model, can dynamically switch and roll back, supports periodic or triggered model retraining according to data changes, and ensures that the model adapts to the latest data.

[0103] The user feedback data, such as the deviation between the predicted result and the actual result, is also supported to be collected, labeled, and formed into new training samples, and the feedback data is used to optimize the model and improve the prediction accuracy. The specific steps are as follows: the feedback data is imported, and the data in formats such as CSV, JSON, and database is supported. The feedback data is displayed on the front-end interface, and the user labels, and multiple labeling types (such as classification, regression, text labeling, and image labeling) are supported. The labeling result is saved to the database. The labeling result is exported and used for model training, model optimization, and prediction accuracy improvement. Based on the latest data and model, the prediction result is generated in real time.

[0104] The following describes the population data research and judgment system provided by the embodiments of the present application. Figure 2 The population data research and judgment system provided by the embodiments of the present application is described.

[0105] Figure 2 An optional architecture schematic diagram of the population data research and judgment system provided by the embodiments of the present application is shown in FIG. 1, which includes the following modules. Figure 2

[0106] The data access module is configured to access multi-source data in multiple data formats collected according to collection rules.

[0107] The preprocessing module is configured to preprocess the multi-source data to obtain standardized target multi-source data.

[0108] The data fusion module is configured to fuse the target multi-source data to obtain a unified population data set, and learn and extract population features from the population data set by using graph embedding and graph neural networks to obtain a population feature vector.

[0109] The analysis module is configured to input the population feature vector into an analysis and judgment model for analysis and prediction to obtain a prediction result for predicting the population data change trend.

[0110] The visualization module is configured to display the prediction result by using a sql statement.

[0111] The optimization module is configured to monitor the multi-source data in real time, and dynamically update the analysis and judgment model and the corresponding prediction result.

[0112] In an exemplary embodiment, the system further includes an API management module configured to encapsulate data into an API interface.

[0113] The population data research and judgment system in the embodiments includes a data access module, a preprocessing module, a data fusion module, an analysis module, a visualization module, an optimization module, and an API management module.

[0114] ​The functions of each module are as follows: the data access module accesses multiple source data and collects population-related data such as census data, public health data, and geographic data. The preprocessing module cleans and preprocesses the accessed data to remove noise and redundant information. The data fusion module performs data fusion and feature extraction, using data fusion techniques to integrate multiple source data into a unified data set; extracts population features, including population size, age structure, gender ratio, migration trends, employment status, etc. The analysis module performs population data analysis and prediction, using machine learning algorithms to analyze population data in depth, identify population change patterns, and predict future population trends, such as population growth trends, aging trends, population size forecasts, and regional population distribution forecasts. The visualization module performs data visualization and report generation, displaying analysis and prediction results through visualization tools (such as heat maps and trend charts), and automatically generating research and analysis reports, including data analysis results, prediction conclusions, and policy recommendations. The optimization module performs dynamic updates and optimization, real-time monitoring of data changes, dynamic updating of analysis models and prediction results, and support for user feedback data to optimize models and improve prediction accuracy.

[0115] The API management module encapsulates the analysis and prediction results in the data warehouse or the accessed multi-source data as API interfaces, providing data integration to other systems. It supports direct selection of data tables or obtaining data sources through custom SQL, and can configure fields as input and output parameters of the API.

[0116] After creating and publishing the API, users can apply for the API, and after the application is approved, an API-TOKEN will be generated for the application record. When calling the API, the third-party user must correctly fill in the value of the API-TOKEN in the request header to successfully call the API, realizing the permission verification of API calling.

[0117] The API management module supports policy management for encapsulated APIs, and through policy management of APIs, it controls the security of API calling:

[0118] 1. Access control policy: You can set black or white list rules. APIs bound to black list rules are not allowed to access APIs from IP addresses / segments set in the rules, and APIs bound to white list rules only allow IP addresses / segments set in the rules to access APIs.

[0119] 2. Flow control policy: You can set the frequency of API calling (times / second). APIs bound to flow control policy rules cannot exceed the value set when calling.

[0120] 3. Row-level permission policy: The range of values of the input parameters allowed to be accessed by the user can be set for the input parameters of the API, realizing row-level permission control of data.

[0121] Through the embodiment, real-time monitoring of data changes in multiple ways, periodic updating of the model, and optimization of the model according to user feedback data are supported. The data is encapsulated as an API and provided to third-party calls, and perfect API access permission control and access policy control are supported, improving real-time data processing and dynamic updating capabilities and enhancing visualization and interactivity.

[0122] According to another aspect of the embodiment of the present application, an electronic device for implementing the above population data research and judgment method is also provided, which can be a server, a terminal, or a combination thereof.

[0123] Figure 3 is a structural schematic diagram of an optional electronic device according to an embodiment of the present application, as shown in Figure 3 includes a processor 302, a communication interface 304, a memory 306, and a communication bus 308, wherein the processor 302, the communication interface 304, and the memory 306 complete communication with each other through the communication bus 308, wherein,

[0124] the memory 306 is configured to store a computer program;

[0125] the processor 302 is configured to execute the computer program stored in the memory 306 to implement the following steps:

[0126] S1, collecting multi-source data in multiple data formats according to a collection rule, and pre-processing the multi-source data to obtain standardized target multi-source data;

[0127] S2, obtaining a unified population data set by data fusion of the target multi-source data, performing population feature learning and extraction on the population data set by using graph embedding and graph neural network, and extracting a population feature vector;

[0128] S3, inputting the population feature vector into an analysis and judgment model for analysis and prediction to obtain a prediction result for predicting a population data transformation trend, and displaying the prediction result through a sql statement;

[0129] S4, real-time monitoring of multi-source data changes, dynamic updating of the analysis and judgment model and the corresponding prediction result.

[0130] Optionally, the communication bus can be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For the convenience of representation, Figure 3 In the figure, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus. The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0131] The memory can include a RAM, and can also include a non-volatile memory, for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0132] The above-mentioned processor can be a general-purpose processor, which can include but is not limited to: a CPU (Central Processing Unit), an NP (Network Processor), and the like; and can also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0133] Optionally, the specific examples in the embodiment can refer to the examples described in the above-mentioned embodiments, and the embodiment will not be described here.

[0134] It should be noted that, for the above-mentioned method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action order described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0135] In the above-mentioned embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can refer to the relevant description of other embodiments.

[0136] Those skilled in the art can understand that all or part of the steps of various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable memory, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0137] The above merely describes exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. Any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will easily derive other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. The present application is intended to cover any variations, uses, or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not described in the present disclosure. The specification and examples are merely considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

[0138] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0139] Those skilled in the art can easily understand that the above only describes preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for analyzing and judging population data, characterized in that: include: Collect multi-source data in various data formats according to the collection rules, and pre-process the multi-source data to obtain standardized target multi-source data; After data fusion of the target multi-source data, a unified population dataset is obtained. Graph embedding and graph neural network are used to learn and extract population features from the population dataset to extract a population feature vector. Input the population feature vector into the analysis and judgment model for analysis and prediction, obtain the prediction result for predicting the population data transformation trend, and display the prediction result through SQL statement; Monitor changes in multi-source data in real time, and dynamically update the analysis and judgment model and the corresponding prediction results.

2. The population data analysis method according to claim 1, characterized in that: The step of collecting multi-source data in various data formats according to the collection rules and pre-processing the multi-source data to obtain standardized target multi-source data includes: Performing data cleaning on the multi-source data in various data formats, wherein the multi-source data in various data formats includes structured data, unstructured data, and semi-structured data, and the data cleaning includes processing missing values, processing outliers, and processing duplicate data; Data alignment is performed on the multi-source data after data cleaning, and the data alignment includes key-value-based merging, time alignment, space alignment, and attribute alignment.

3. The method for analyzing population data according to claim 1, wherein: After data fusion of the target multi-source data, a unified population dataset is obtained. Graph embedding and graph neural network are used to learn and extract population features from the population dataset to extract a population feature vector: Uniformly identifying each entity in the multi-source data; Create a node for each of said entities and add attributes; Create edges and add attributes to relationships between entities; Based on the constructed graph structure, the graph embedding algorithm Node2Vec is used to convert the created nodes and edges into low-dimensional population feature vectors.

4. The method for analyzing population data according to claim 1, wherein: Before inputting the population feature vector into the analysis and judgment model for analysis and prediction, the method further includes: Determining the analysis and judgment model according to the nature of the prediction problem, wherein the nature of the prediction problem includes regression problem, classification problem, clustering problem and time series analysis problem; The grid search technology is used in combination with the cross-validation method to determine the target parameter combination of the analysis and judgment model.

5. The method for analyzing population data according to claim 1, wherein: The population feature vector is input into the analysis and judgment model for analysis and prediction to obtain a prediction result for predicting the population data transformation trend, and the prediction result is displayed through a SQL statement, including: Use various components to set dynamic data and display prediction results through SQL statements; Bind the API interface and display the data information returned by the API interface.

6. The method for analyzing population data according to claim 1, wherein: The real-time monitoring of multi-source data changes and the dynamic updating of the analysis and judgment model and the corresponding prediction results include: Real-time access to data streams via Kafka message queue; and / or, Read incremental update data from a data table via database log streaming; and / or, Set rules and thresholds through scheduled tasks to regularly detect changes in multi-source data.

7. The method for analyzing population data according to claim 6, wherein: The method further comprises: Access real-time data streams through message queues or database log streaming reading; Normalize each sample and calculate the gradient based on the current model parameters and sample data; Use the gradient to update the model parameters and save the updated model periodically.

8. A population data analysis system, characterized in that: include: Data access module, used to access multi-source data in various data formats according to collection rules; A preprocessing module, configured to preprocess the multi-source data to obtain standardized target multi-source data; A data fusion module is used to fuse the target multi-source data to obtain a unified population data set, and use graph embedding and graph neural network to learn and extract population features from the population data set to extract population feature vectors; An analysis module is used to input the population feature vector into the analysis and judgment model for analysis and prediction, and obtain a prediction result for predicting the transformation trend of population data; A visualization module is used to display the prediction results through SQL statements; The optimization module is used to monitor the changes in multi-source data in real time and dynamically update the analysis and judgment model and the corresponding prediction results.

9. The population data analysis system according to claim 8, characterized in that: The system further comprises: The API management module is used to encapsulate data into API interfaces.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images