Centric survey data distributed processing method and platform combined with Internet of Things
By combining the distributed processing method of the Internet of Things, the census data is structured preprocessing, edge processing and visualized, which solves the problems of low efficiency of census data processing and inaccurate feature representation, and realizes efficient and accurate data processing and feature quantification.
Patent Information
- Application Number
- CN202411927583.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-25
- Publication Date
- 2025-04-18
AI Technical Summary
In the census scenario, the data is of a wide variety and uneven distribution, resulting in the inefficiency of traditional centralized data processing methods, inaccurate feature representation, and inability to meet the real-time requirements.
Using a distributed processing method combined with the Internet of Things, data is preprocessed by introducing a predetermined structured strategy, data is extracted at a specific level for edge processing, and combined with visual processing and sampling and aggregation technology, characteristic values of quantitatively characterized data sets are generated.
It improves the processing efficiency of census data and the accuracy of feature quantification, meets the real-time requirements, and enhances the utilization value of data.
Smart Images
Figure CN119848137A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a census data distributed processing method and platform combined with the Internet of Things. Background Art
[0002] With the rapid development of Internet of Things technology, the collection and processing of large-scale data has become one of the key research directions. In the census scenario, the types and quantities of data are growing exponentially, which poses a huge challenge to traditional data processing methods. On the one hand, there is a large uneven distribution of census data, and the differences in data between different collection terminals and regions lead to a decrease in processing efficiency; on the other hand, traditional data processing methods usually rely on centralized computing models. The centralized uploading and processing of data will not only increase the pressure on network transmission, but may also cause delays and data loss problems, and cannot meet real-time requirements. In addition, census data contains multiple types such as text, pictures, audio and video. There is a lack of effective strategies for the processing and fusion of different data types, and data features are difficult to accurately extract, further reducing the value of data utilization. Summary of the invention
[0003] The present application provides a census data distributed processing method and platform combined with the Internet of Things, which is used to solve the technical problems of low efficiency and inaccurate feature representation of existing census data processing when the census data are of various types and unevenly distributed.
[0004] In view of the above problems, the present application provides a distributed processing method and platform for census data combined with the Internet of Things.
[0005] In a first aspect of the present application, a distributed processing method for census data in combination with the Internet of Things is provided, the method comprising: introducing a predetermined structured strategy to perform data preprocessing on a census data set to obtain a structured census data set; performing visualization processing on the structured census data set to obtain a census data visual graph; extracting data of a predetermined level threshold in the census data visual graph to form an edge census data set; performing distributed edge processing on the edge census data set by edge processors arranged at the predetermined level threshold to obtain an edge processing result; rendering the edge processing result to the census data visual graph to obtain a target visual graph; performing sampling and aggregation processing on the target visual graph to obtain a target data set, and activating a central processing unit to analyze the target data set to obtain a first eigenvalue; using the first eigenvalue as a census feature coefficient, wherein the census feature coefficient is used to quantitatively characterize the census data set.
[0006] A second aspect of the present application provides a census data distributed processing platform combined with the Internet of Things, the platform comprising: a data preprocessing module: introducing a predetermined structured strategy to perform data preprocessing on a census data set to obtain a structured census data set; a visualization processing module: performing visualization processing on the structured census data set to obtain a census data visual map; an edge census data set composition module: extracting data of a predetermined level threshold in the census data visual map to form an edge census data set; a distributed edge processing module: performing distributed edge processing on the edge census data set by edge processors arranged at the predetermined level threshold to obtain an edge processing result; an edge processing result rendering module: rendering the edge processing result to the census data visual map to obtain a target visual map; a data set analysis module: performing sampling and aggregation processing on the target visual map to obtain a target data set, and activating a central processing unit to analyze the target data set to obtain a first eigenvalue; a census data set quantification module: using the first eigenvalue as a census feature coefficient, wherein the census feature coefficient is used to quantitatively characterize the census data set.
[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:
[0008] A predetermined structured strategy is introduced to preprocess the census data set to obtain a structured census data set; the structured census data set is visualized to obtain a census data visual map; data of a predetermined level threshold in the census data visual map are extracted to form an edge census data set; the edge census data set is distributedly processed by edge processors arranged at the predetermined level threshold to obtain edge processing results; the edge processing results are rendered to the census data visual map to obtain a target visual map; the target visual map is sampled and aggregated to obtain a target data set, and a central processor is activated to analyze the target data set to obtain a first eigenvalue; the first eigenvalue is used as a census feature coefficient, wherein the census feature coefficient is used to quantitatively characterize the census data set. The technical effect of improving the processing efficiency of census data and the accuracy of feature quantification by introducing a predetermined structured strategy and distributed edge processing is achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0010] Figure 1A schematic flow chart of a distributed processing method for census data in combination with the Internet of Things provided in an embodiment of the present application.
[0011] Figure 2 A schematic diagram of the structure of a census data distributed processing platform combined with the Internet of Things provided in an embodiment of the present application.
[0012] Explanation of the accompanying drawings: data preprocessing module 1, visualization processing module 2, edge census data set composition module 3, distributed edge processing module 4, edge processing result rendering module 5, data set analysis module 6, census data set quantification module 7. DETAILED DESCRIPTION
[0013] The present application provides a distributed processing method and platform for census data combined with the Internet of Things, which is used to solve the technical problems of low efficiency and inaccurate feature representation of existing census data processing when the census data are of various types and unevenly distributed.
[0014] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0015] Embodiment 1, as Figure 1 As shown, the present application provides a distributed processing method for census data combined with the Internet of Things, the method comprising:
[0016] A predetermined structured strategy is introduced to preprocess the census dataset to obtain a structured census dataset.
[0017] In one embodiment, a predetermined structured strategy is introduced to pre-process the census data set, which actually means that when the census data is initially sorted and processed, a set of pre-set standards and methods are used to ensure that the data can be organized and stored in a specific structure and format. This census data set includes population information (such as name, relationship, ID card, etc.), geographic information (such as residential address, type of residence, etc.), economic information (such as income, occupation, etc.), social behavior information (such as travel mode, daily activities, etc.), health and medical information (such as physical health status, medical insurance status, etc.), on-site records, etc. These data are stored in the census data set in the form of pictures, audio, video, etc. This predetermined structured strategy includes clustering plans, text recognition plans, audio recognition plans and video recognition plans. Through the coordinated processing of the census data set by these four plans, the census data set can be converted into a structured census data set, which is convenient for subsequent data analysis, mining or modeling.
[0018] In a possible implementation, a predetermined structured strategy is introduced to preprocess the census data set to obtain a structured census data set, including:
[0019] Extracting a clustering plan from the predetermined structured strategy; performing clustering processing on the census data set according to the clustering plan to obtain a census data clustering result; and constructing the structured census data set based on the preprocessed census data clustering result.
[0020] Optionally, in the preprocessing stage, a clustering plan is first extracted from a predetermined structured strategy. The clustering plan is a set of data type classification rules for guiding clustering processing of the census data set. For example, for image type data, the classification rule is to determine whether the data suffix is ".jpg, .png", etc., for audio type data, the classification rule is to determine whether the data suffix is ".mp3, .wav", etc., for video type data, the classification rule is to determine whether the data suffix is "mp4, .avi", etc.; after extracting the clustering plan, the census data set is traversed, and in each traversal, the suffix of the currently traversed data is compared with the image type classification rules, audio type classification rules, and video type classification rules to determine which data type it belongs to, and the data is added By adding each of the above information to the corresponding cluster and repeating this process, the census data set can be divided into picture clustering clusters, audio clustering clusters and video clustering clusters. The census data clustering results are obtained by aggregating the picture clustering clusters, audio clustering clusters and video clustering clusters. Subsequently, the text recognition plan, audio recognition plan and video recognition plan are extracted from the predetermined structured strategy. The picture clustering clusters in the census data clustering results are processed using the text recognition plan. The audio clustering clusters in the census data clustering results are processed using the audio recognition plan. The video clustering clusters in the census data clustering results are processed using the video recognition plan. Finally, all the processed picture clustering clusters, audio clustering clusters and video clustering clusters are organized together to form a structured census data set to provide support for subsequent data analysis, feature extraction and visualization processing.
[0021] In a possible implementation, forming the structured census data set based on the preprocessed census data clustering result includes:
[0022] Perform text recognition processing on the image clusters in the census data clustering results to obtain a first data set; perform audio recognition processing on the audio clusters in the census data clustering results to obtain a second data set; perform video recognition processing on the video clusters in the census data clustering results to obtain a third data set; the first data set, the second data set and the third data set constitute the structured census data set.
[0023] Optionally, after extracting the text recognition plan, audio recognition plan and video recognition plan, the text recognition plan is used to perform text recognition processing on the image clustering cluster. In this process, the resolution of each image in the image clustering cluster is checked, and the images with resolution lower than the resolution tolerance lower limit are eliminated. Then, the pre-processing module of the optical character recognition (OCR) technology (such as OpenCV or Tesseract) is used to detect the text area in the image. Subsequently, the area containing the text is determined by edge detection, color segmentation and other technologies, and the OCR tool (such as Tesseract, Google VisionAPI) is used to extract characters from the detected text area to identify the text content in the image. The identified text information is then organized into a first data set, which includes the path of the image file, the text content and its location information; the audio recognition plan is used to perform audio recognition processing on the audio clustering cluster. In this process, each audio in the audio clustering cluster is converted into a standard format (such as .wav) and subjected to noise reduction processing to filter out background noise and ensure voice clarity. Then, the speech recognition technology (such as Google Speech-to-Text API) is used to transcribe the noise-reduced audio sentence by sentence and extract text information from the speech. After that, natural language processing (NLP) is performed on the transcribed text information to extract key words. The audio file path, recognized text content, key words and other information are then organized into a second data set. The video clusters are subjected to video recognition processing using a video recognition plan. In this process, key frames (such as 1 frame per second) are extracted according to the set frame rate, thereby decomposing each video in the video cluster into several frames of images. The above-mentioned text recognition plan is used to perform text recognition processing on each frame of the image to obtain the video text information. The above-mentioned audio recognition plan is used to perform audio recognition processing on the audio part of the video to obtain the video speech. audio / text information, and then organize the video text information and video voice / text information into a third data set; after obtaining the first data set, the second data set and the third data set, organize the first data set, the second data set and the third data set according to a unified field specification, for example, each record contains the data source (picture / audio / video), identification content, file path, etc., and then associate the picture, audio and video records through metadata (such as timestamp, geographic location, household head information) to ensure that data from the same census object can be integrated and analyzed; finally, integrate the associated data records to form a structured census data set, which can comprehensively and clearly represent the multi-dimensional information of the census object and provide support for further data processing and analysis.
[0024] The structured census data set is visualized to obtain a census data visualization graph.
[0025] In one embodiment, after obtaining a structured census data set, the structured census data set is visualized, that is, the structured census data set is divided into multiple data sets according to the levels corresponding to different data, each data set corresponds to a level, and each level corresponds to a vertex. The data sets at adjacent levels are then subjected to correlation analysis, and the correlation analysis result is used as the length between two vertices. Based on the determined vertices and the lengths between the vertices, a census data visualization graph is constructed, and the structured census data set is presented in an intuitive graphical form, so that the characteristics and distribution patterns of the data are clearer, thereby improving the readability and analysis efficiency of the data.
[0026] In a possible implementation, it includes:
[0027] Obtain a predetermined census level; construct a structure vertex set based on the predetermined census level; extract a first vertex and a second vertex from the structure vertex set, wherein the first vertex and the second vertex are at adjacent census levels; sequentially match a fourth data set corresponding to the first vertex and a fifth data set corresponding to the second vertex in the structured census data set; perform a many-to-many correlation analysis on the fourth data set and the fifth data set to obtain a correlation analysis result; obtain a correlation index according to the correlation analysis result, and match an associated unit length corresponding to the correlation index; obtain a visual graph of the census data based on the first vertex, the second vertex and the associated unit length.
[0028] Optionally, when visualizing a structured census dataset, a predetermined census level is first obtained. This predetermined census level is determined according to the census task requirements and is divided into the first level, the second level, the third level, the fourth level, and the fifth level. The first level corresponds to the national scope, and the data categories that may be included at this level include the national total population, population growth rate, gross domestic product (GDP), etc. The second level corresponds to the provincial scope, and the data categories that may be included at this level include the provincial total population, urban-rural population ratio, provincial GDP, etc. The third level corresponds to the city scope, and the data categories that may be included at this level include the city total population, floating population size, and average household population. , urban GDP, etc. The fourth level corresponds to the county level, which may include the total population of the county, the population distribution of each village and town, the county GDP, etc. The fifth level corresponds to the community scope, which may include the total number of households in the community, the number of permanent residents and floating population, the proportion of registered population and non-registered population, the per capita income of the community, etc. These levels represent different levels and scopes of data, which is convenient for step-by-step analysis and processing; then, the census levels are organized into a structural vertex set according to the hierarchical relationship, and each vertex in the vertex set corresponds to a census level. For example, the first level corresponds to the first vertex V1, the second level corresponds to the second vertex V2, The third level corresponds to the third vertex V3, the fourth level corresponds to the fourth vertex V4, and the fifth level corresponds to the fifth vertex V5. After that, two adjacent vertices are extracted from the structure vertex set in sequence as the first vertex and the second vertex, for example, V1 corresponding to the first level and V2 corresponding to the second level, or V3 corresponding to the third level and V4 corresponding to the fourth level. For the two extracted vertices, the fourth data set corresponding to the first vertex and the fifth data set corresponding to the second vertex are extracted from the structured census data set. This fourth data set contains the specific data of the data category corresponding to the first vertex, such as the national total population, population growth rate, and domestic The fifth data set contains the specific data of the data category corresponding to the second vertex, such as the provincial total population, urban-rural population ratio, provincial GDP and other data types in the second level; after obtaining the fourth and fifth data sets, check the fields of the fourth and fifth data sets to ensure that the data dimensions are consistent. For example, the county population and community population need to be expressed in the same unit and time period, and then correspond the data in the fourth data set with the data in the fifth data set. In the correspondence process, many-to-many relationships need to be considered. For example, a county may contain multiple communities, or a community may span multiple counties. Ensure that all matching relationships are covered;After the correspondence is completed, the Pearson correlation coefficient is calculated for each pair of fields (such as county-level population and community-level population), the correlation r between the two is quantified, and the field pairs with statistical significance are screened out. The screening process is based on preset screening criteria, which include extremely significant correlation criteria, significant correlation criteria, and low correlation criteria. The correlation coefficient corresponding to the extremely significant correlation criteria is greater than the first correlation coefficient lower limit (such as 0.8), the correlation coefficient corresponding to the significant correlation criteria is greater than the second correlation coefficient lower limit (such as 0.5), and the correlation coefficient corresponding to the low correlation criteria is less than or equal to the second correlation coefficient lower limit; based on this preset screening criteria, extremely significant correlated field pairs, significantly correlated field pairs, and low correlated field pairs can be screened out from the fourth and fifth data sets after the correspondence, and the correlation analysis results can be obtained by sorting out these field pairs; then, the corresponding Comprehensively analyze the extremely significantly correlated field pairs, significantly correlated field pairs, and the total number of field pairs in the correlation analysis results, calculate the correlation index of the correlation analysis results, and match the preset correlation unit length according to the correlation index. For example, a higher correlation index corresponds to a shorter unit length, indicating that the relationship between the two vertices is closer; finally, visually connect the first vertex (such as the county level) and the second vertex (such as the community) through the corresponding correlation unit length, repeat the above process, perform the same operation on other vertices, and construct a census data visualization diagram. In the census data visualization diagram, each vertex is displayed as a node, and the correlation unit length is presented as an edge. Edges of different unit lengths are distinguished by color, thickness or length, which intuitively displays the correlation between data at different levels and provides an effective support tool for in-depth analysis of census data. ;
[0029] In a possible implementation, obtaining a correlation index according to the correlation analysis result and matching an association unit length corresponding to the correlation index include:
[0030] Extract extremely significant correlation indicators from the correlation analysis results, and obtain extremely significant numbers by counting; extract significant correlation indicators from the correlation analysis results, and obtain significant numbers by counting; weight the extremely significant numbers and the significant numbers to obtain a comprehensive significant number, and take the ratio of the comprehensive significant number to the total number in the correlation analysis results, which is recorded as the correlation index.
[0031] Optionally, the correlation analysis results are traversed, and the number of extremely significant correlation indicators (i.e., the aforementioned extremely significant correlated field pairs) is counted to obtain the extremely significant number, and the number of significant correlation indicators (i.e., the aforementioned significantly correlated field pairs) is counted to obtain the significant number; then, the extremely significant number and the significant number are weighted and summed according to preset weights (determined according to task requirements and expert decisions) to obtain a comprehensive significant number, and then the ratio of this comprehensive significant number to the total number of all field pairs in the correlation analysis results is calculated to obtain a correlation index, which represents the proportion of the comprehensive significant number in the total correlation indicators, and is used to quantify the overall correlation strength, providing a precise quantitative indicator for subsequent analysis, which can guide the matching or visualization of the correlation unit length.
[0032] Data within a predetermined level threshold in the census data visibility graph is extracted to form a marginal census data set.
[0033] In one embodiment, according to the requirements of the census task, a predetermined level threshold for filtering data is set. The predetermined level threshold can be determined based on the geographical level (such as province, city, county) or the upper and lower limits of data indicators (such as population density, economic indicators, etc.). For example, only data between cities and counties are selected, and only data in areas with a population density of more than 1,000 people / square kilometer are selected; then, the data of the nodes in the census data visualization graph are traversed to determine whether the predetermined level threshold conditions are met. If so, these data are extracted to form an edge census data set. The data set contains data points that meet the threshold conditions and related information, such as node names, data values, etc., which provides accurate data input for subsequent distributed edge processing.
[0034] The edge census data set is subjected to distributed edge processing by edge processors arranged at the predetermined level threshold to obtain edge processing results; the edge processing results are rendered to the census data visual graph to obtain a target visual graph.
[0035] In one embodiment, after obtaining the edge census data set, each deployed edge processor obtains the corresponding edge census data from the acquired edge census data set according to a predetermined level threshold, for example, edge processor A processes city-level data, and edge processor B processes county-level data; in each edge processor, the acquired edge census data will be subjected to distributed data processing, including removal of abnormal data, data deduplication, data standardization, etc. For removal of abnormal data, obviously unreasonable data will be removed based on the recognition rules of data anomalies, for example, the population density value exceeds a reasonable range (greater than 10,000 people / square kilometer or 1 person / square kilometer), and the value of income or other indicators obviously deviates from the mean (the 3 times standard deviation rule can be used); for data deduplication, the primary key (such as geocoding) or composite key (such as community name + timestamp) will be used. ) Identify duplicate entries and check whether there are duplicate records in the edge census data. For example, two communities may have reported the same set of data repeatedly due to the time of collection. If duplicate records are found, the data with the latest time or reliable source will be retained and the duplicate records will be deleted; for data standardization, the maximum and minimum method will be used to scale the continuous data to the specified range (such as [0,1] or [-1,1]); after the distributed data processing is completed, each edge processor will output the processed edge census data, and the edge processing results will be constructed by integrating the outputs of these edge processors; after obtaining the edge processing results, the edge census data in the edge processing results will be updated to the census data visual map, and the data of the corresponding vertices will be updated according to the cleaned data, for example, the population density is adjusted to the average value after removing outliers. Through the above process, the quality of the edge processed data is significantly improved, providing a reliable basis for subsequent analysis.
[0036] The target visible graph is sampled and aggregated to obtain a target data set, and a central processing unit is activated to analyze the target data set to obtain a first eigenvalue; the first eigenvalue is used as a census characteristic coefficient, wherein the census characteristic coefficient is used to quantitatively characterize the census data set.
[0037] In one embodiment, when sampling and aggregating the target visual graph, it is first necessary to extract the lowest level and the second lowest level data from the target visual graph. The data at these two levels are the core of constructing the target data set. The lowest level data usually represents the most detailed census unit (such as the community level), including detailed information such as the population, income, and area of each community, while the second lowest level data represents the summary unit of the upper layer (such as the county level), including statistical data such as the total population, population density, and average income within the county. Through such hierarchical data, it can be ensured that the sampling aggregation process has both precision and a global perspective. By aggregating the lowest level and the second lowest level data, a target data set is constructed. This target data set contains both the detailed features of the lowest level data and the comprehensive features of the second lowest level data. A global perspective with strong representativeness and analytical value. The fields of the target data set may include key information such as the lowest level average population density, the second lowest level total income, and the concentrated value of the lowest level income distribution. After the target data set is completed, it will be input into the central processing unit for analysis. The central processing unit aims at fast analysis and quickly processes the fields in the target data set through the internal census prediction model to generate the first eigenvalue. This first eigenvalue is a rough quantitative result of the census data, which can timely reflect the key characteristics of the regional census, such as the ratio of total population density to average income. Finally, this first eigenvalue is defined as the census characteristic coefficient, which is used to quantitatively characterize the overall characteristics of the census data set, providing a fast and intuitive analysis basis, and at the same time providing a reference basis for subsequent detailed analysis and decision-making.
[0038] In a possible implementation, sampling and aggregation processing is performed on the target visibility graph to obtain a target data set, and a central processing unit is activated to analyze the target data set to obtain a first eigenvalue, including:
[0039] Obtain the lowest level data set corresponding to the lowest census level in the target visible graph; perform multi-layer sampling on the lowest level data set to obtain a lowest level sampling set; perform average aggregation processing on the data in the lowest level sampling set to obtain a lowest level aggregation result; obtain the second lowest level data set corresponding to the second lowest census level in the target visible graph; perform multi-layer sampling on the second lowest level data set to obtain a second lowest level sampling set, and adjust the second lowest level sampling set in combination with the lowest level aggregation result to obtain a target second lowest level data set; perform concentrated value aggregation processing on the target second lowest level data set to obtain a second lowest level aggregation result; and form the target data set based on the lowest level aggregation result and the second lowest level aggregation result.
[0040] Optionally, when constructing the target data set, first extract the data corresponding to the lowest census level (such as the community level) from the target visual graph, including detailed information such as population, income, and area. These data constitute the lowest level data set, and then perform multi-layer sampling on the lowest level data set. By setting stratified sampling rules to extract samples, ensure that the sampling covers the key features of the community. For example, according to the population density distribution of the community, the community is divided into three levels: high density, medium density, and low density. Then, samples are randomly selected from each level to generate the lowest level sampling set. After the lowest level sampling set is generated, the data therein is averaged and aggregated. By calculating each field, By calculating the average value, the overall characteristics of the community level can be obtained. For example, the average population density or average income of all sampled communities can be calculated to form the lowest level aggregation results, which can reflect the basic characteristics of the community as a whole. Subsequently, the second lowest level data set corresponding to the second lowest census level (such as county level) is obtained from the target visual map. The second lowest level data set contains statistical information related to the county level, such as the total population, total area, total income, etc. of the county. After obtaining the second lowest level data set, multi-layer sampling is also required for it. By setting stratification rules (such as stratification by population or area), samples are extracted to generate the second lowest level sampling set. After the second lowest level sampling set is generated, The second-lowest-level sampling set should be adjusted in combination with the lowest-level aggregation results. The adjustment process includes correcting the second-lowest-level data according to the aggregation value of the lowest-level data. For example, if the deviation between the total population density of a county and the average value of the lowest-level community data exceeds the deviation tolerance threshold, the total population density of the county is adjusted to the average value of the lowest-level community data to reflect the actual distribution of the community data. Through this correction, the consistency and accuracy between the second-lowest-level sampling set and the lowest-level data can be ensured, thereby generating the target second-lowest-level data set. After the target second-lowest-level data set is generated, it is further subjected to centralized value aggregation processing. By calculating the median or mode, the impact of extreme values can be reduced to ensure the stability and representativeness of the second-lowest-level data. For example, the median or mode of county-level income is calculated to more accurately reflect the regional economic characteristics. The result of this processing will form the second-lowest-level aggregation result, representing the key features at the county level. Finally, based on the lowest-level aggregation results and the second-lowest-level aggregation results, the two are integrated to form the target data set. The target data set contains the lowest-level detailed features and the second-lowest-level global features, such as the average population density at the community level and the median income at the county level. This target data set provides a reliable foundation for subsequent data analysis and feature extraction.
[0041] In a possible implementation, sampling and aggregation processing is performed on the target visibility graph to obtain a target data set, and a central processing unit is activated to analyze the target data set to obtain a first eigenvalue, including:
[0042] The target data set is used as input information of a census prediction model, and output information is obtained through the census prediction model, wherein the output information includes the first eigenvalue; wherein the census prediction model refers to an intelligent prediction model obtained by supervised training of historical census data sets based on the principle of neural network.
[0043] Optionally, after the target data set is constructed, the central processing unit will be activated and the target data set will be input into the central processing unit. After receiving the target data set, the central processing unit will pass the target data set to the internal census prediction model. The census prediction model triggers the forward propagation calculation according to the input data. The hidden layer of the census prediction model processes the input features through weights and activation functions, extracts implicit relationships and patterns, and then calculates the final first eigenvalue according to the output of the hidden layer. The census prediction model is constructed based on a neural network and a historical census data set. Specifically, a census prediction model is first constructed according to the principle of a neural network. The structure of the prediction model includes an input layer, a hidden layer, and an output layer. Random numbers are then used to initialize the census prediction model, including the weights and biases of the census prediction model. Subsequently, the historical census data set is input into the census prediction model. This historical census data set includes past census data and its corresponding known eigenvalues. After the model receives the historical census data set, it calculates the predicted eigenvalues through forward propagation, and calculates the error between the predicted eigenvalues and the known eigenvalues through the mean square error. Based on the calculated error, the model parameters are adjusted through back propagation until the model converges, that is, the maximum number of iterations is reached, or the error tends to stabilize.
[0044] In a possible implementation, the method further includes:
[0045] Performing matrix-vector processing on the target visible graph to obtain a target matrix vector; decomposing the target matrix vector to obtain a second eigenvalue, and using the second eigenvalue as the census characteristic coefficient, wherein the census characteristic coefficient is used to quantitatively characterize the census data set.
[0046] Optionally, in order to obtain more accurate eigenvalues, the non-numerical data in the vertices and edges in the target visual graph are first digitized, and the node data forms a multidimensional eigenvector according to the census attributes (such as population, income, and area). The weight of the edge is represented by the associated unit length. For example, the eigenvector of a community can be represented as [population, income, area]. After organizing the values of these nodes and edges into an adjacency matrix and a eigenmatrix, the two matrices are added to generate a target matrix vector; subsequently, the target matrix vector is decomposed by dimensionality reduction techniques (such as principal component analysis PCA or singular value decomposition SVD) to extract the main feature information of the data. In the decomposition process, taking principal component analysis PCA as an example, a covariance matrix is constructed based on the target matrix vector. The covariance matrix is a symmetric matrix used to measure the linear correlation between variables. Its diagonal elements represent the variance of each variable, and the non-diagonal elements represent the covariance between variables. Subsequently, the covariance matrix is subjected to eigenvalue decomposition, and the eigenvalues and eigenvectors of the covariance matrix are obtained through mathematical operations. The eigenvalue represents the variance of the original data in the direction of the corresponding eigenvector, and the eigenvector represents the direction of the new coordinate axis after dimensionality reduction. In order to extract the main feature information, the eigenvectors are sorted according to the size of the eigenvalue, and the principal component with the largest eigenvalue is selected as the new feature after dimensionality reduction. These principal components are the result of linear combination and can retain most of the variance information in the data. Information, the ultimate goal of dimensionality reduction is to simplify the data by reducing the data dimension while retaining as much important information as possible. After that, the target matrix vector is projected onto the principal component direction and the projected scalar value is calculated. This projection is to quantify the size of the component of the target matrix vector in the most important feature direction. The result is the second eigenvalue of the target matrix vector. This second eigenvalue focuses on the global data patterns and complex correlations in the target visual graph, and can quantify and characterize the census data set with high accuracy. Finally, the second eigenvalue is defined as the census characteristic coefficient, which is used to comprehensively quantify and characterize the census data set. This characteristic coefficient not only has high accuracy, but also reflects the global characteristics and multidimensional relationships of the data, further improving the refined characterization of the census data.
[0047] Embodiment 2 is based on the same inventive concept as the distributed processing method of census data combined with the Internet of Things in the previous embodiment. Figure 2As shown, the present application provides a census data distributed processing platform combined with the Internet of Things, and the platform and method embodiments in the embodiments of the present application are based on the same inventive concept. Wherein, the platform includes: a data preprocessing module 1: introducing a predetermined structured strategy to perform data preprocessing on the census data set to obtain a structured census data set; a visualization processing module 2: performing visualization processing on the structured census data set to obtain a census data visual map; an edge census data set composition module 3: extracting data of a predetermined level threshold in the census data visual map to form an edge census data set; a distributed edge processing module 4: performing distributed edge processing on the edge census data set through edge processors arranged at the predetermined level threshold to obtain edge processing results; an edge processing result rendering module 5: rendering the edge processing results to the census data visual map to obtain a target visual map; a data set analysis module 6: performing sampling and aggregation processing on the target visual map to obtain a target data set, and activating a central processor to analyze the target data set to obtain a first eigenvalue; a census data set quantification module 7: using the first eigenvalue as a census feature coefficient, wherein the census feature coefficient is used to quantitatively characterize the census data set.
[0048] Furthermore, the data preprocessing module 1 also includes:
[0049] Extracting a clustering plan from the predetermined structured strategy; performing clustering processing on the census data set according to the clustering plan to obtain a census data clustering result; and constructing the structured census data set based on the preprocessed census data clustering result.
[0050] Furthermore, the data preprocessing module 1 also includes:
[0051] Perform text recognition processing on the image clusters in the census data clustering results to obtain a first data set; perform audio recognition processing on the audio clusters in the census data clustering results to obtain a second data set; perform video recognition processing on the video clusters in the census data clustering results to obtain a third data set; the first data set, the second data set and the third data set constitute the structured census data set.
[0052] Furthermore, the visualization processing module 2 also includes:
[0053] Obtain a predetermined census level; construct a structure vertex set based on the predetermined census level; extract a first vertex and a second vertex from the structure vertex set, wherein the first vertex and the second vertex are at adjacent census levels; sequentially match a fourth data set corresponding to the first vertex and a fifth data set corresponding to the second vertex in the structured census data set; perform a many-to-many correlation analysis on the fourth data set and the fifth data set to obtain a correlation analysis result; obtain a correlation index according to the correlation analysis result, and match an associated unit length corresponding to the correlation index; obtain a visual graph of the census data based on the first vertex, the second vertex and the associated unit length.
[0054] Furthermore, the visualization processing module 2 also includes:
[0055] Extract extremely significant correlation indicators from the correlation analysis results, and obtain extremely significant numbers by counting; extract significant correlation indicators from the correlation analysis results, and obtain significant numbers by counting; weight the extremely significant numbers and the significant numbers to obtain a comprehensive significant number, and take the ratio of the comprehensive significant number to the total number in the correlation analysis results, which is recorded as the correlation index.
[0056] Furthermore, the data set analysis module 6 also includes:
[0057] Obtain the lowest level data set corresponding to the lowest census level in the target visible graph; perform multi-layer sampling on the lowest level data set to obtain a lowest level sampling set; perform average aggregation processing on the data in the lowest level sampling set to obtain a lowest level aggregation result; obtain the second lowest level data set corresponding to the second lowest census level in the target visible graph; perform multi-layer sampling on the second lowest level data set to obtain a second lowest level sampling set, and adjust the second lowest level sampling set in combination with the lowest level aggregation result to obtain a target second lowest level data set; perform concentrated value aggregation processing on the target second lowest level data set to obtain a second lowest level aggregation result; and form the target data set based on the lowest level aggregation result and the second lowest level aggregation result.
[0058] Furthermore, the data set analysis module 6 also includes:
[0059] The target data set is used as input information of a census prediction model, and output information is obtained through the census prediction model, wherein the output information includes the first eigenvalue; wherein the census prediction model refers to an intelligent prediction model obtained by supervised training of historical census data sets based on the principle of neural network.
[0060] Furthermore, the data set analysis module 6 also includes:
[0061] Performing matrix-vector processing on the target visible graph to obtain a target matrix vector; decomposing the target matrix vector to obtain a second eigenvalue, and using the second eigenvalue as the census characteristic coefficient, wherein the census characteristic coefficient is used to quantitatively characterize the census data set.
[0062] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0063] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
[0064] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.
Claims
1. A distributed processing method for census data combined with the Internet of Things, characterized in that: include: A predetermined structured strategy is introduced to pre-process the census data set to obtain a structured census data set; Performing visualization processing on the structured census data set to obtain a census data visualization graph; Extracting data within a predetermined level threshold in the census data visibility graph to form a marginal census data set; Performing distributed edge processing on the edge survey data set by edge processors arranged at the predetermined level threshold to obtain edge processing results; Rendering the edge processing result to the census data visual graph to obtain a target visual graph; Performing sampling and aggregation processing on the target visible graph to obtain a target data set, and activating a central processing unit to analyze the target data set to obtain a first eigenvalue; The first eigenvalue is used as a census characteristic coefficient, wherein the census characteristic coefficient is used to quantitatively characterize the census data set.
2. The distributed processing method for census data combined with the Internet of Things as claimed in claim 1, characterized in that: A predetermined structured strategy is introduced to preprocess the census data set to obtain a structured census data set, including: Extracting a clustering plan from the predetermined structured strategy; Performing clustering processing on the census data set according to the clustering plan to obtain a census data clustering result; The structured census data set is constructed based on the clustering results of the preprocessed census data.
3. The distributed processing method of census data combined with the Internet of Things as claimed in claim 2 is characterized in that: The structured census data set is formed based on the clustering results of the preprocessed census data, including: Performing text recognition processing on the image clusters in the census data clustering results to obtain a first data set; Performing audio recognition processing on the audio clusters in the census data clustering results to obtain a second data set; Performing video recognition processing on the video clustering clusters in the census data clustering results to obtain a third data set; The first data set, the second data set and the third data set constitute the structured census data set.
4. The method for distributed processing of census data in combination with the Internet of Things as claimed in claim 1, characterized in that: include: Get the scheduled census level; assembling a set of structure vertices based on the predetermined census level; Extracting a first vertex and a second vertex from the structure vertex set, wherein the first vertex and the second vertex are at adjacent census levels; sequentially matching a fourth data set corresponding to the first vertex and a fifth data set corresponding to the second vertex in the structured census data set; Performing a many-to-many correlation analysis on the fourth data set and the fifth data set to obtain a correlation analysis result; Obtaining a correlation index according to the correlation analysis result, and matching the correlation unit length corresponding to the correlation index; Based on the first vertex, the second vertex and the associated unit length, a visual graph of the census data is obtained.
5. The distributed processing method of census data combined with the Internet of Things as claimed in claim 4 is characterized in that: Obtaining a correlation index according to the correlation analysis result, and matching the correlation unit length corresponding to the correlation index, including: Extracting extremely significant correlation indicators from the correlation analysis results and obtaining extremely significant numbers by counting; Extracting significant correlation indicators from the correlation analysis results and obtaining significant numbers by counting; The extremely significant number and the significant number are weighted to obtain a comprehensive significant number, and the ratio of the comprehensive significant number to the total number in the correlation analysis result is taken and recorded as the correlation index.
6. The distributed processing method for census data combined with the Internet of Things as claimed in claim 1, characterized in that: The target visible graph is sampled and aggregated to obtain a target data set, and a central processing unit is activated to analyze the target data set to obtain a coarse feature value, including: Obtain the lowest level data set corresponding to the lowest census level in the target visual graph; Performing multi-layer sampling on the lowest level data set to obtain a lowest level sampling set; Performing average aggregation processing on the data in the lowest level sampling set to obtain a lowest level aggregation result; Obtain a second lowest level data set corresponding to a second lowest census level in the target visual graph; Perform multi-layer sampling on the second-lowest-level data set to obtain a second-lowest-level sampling set, and adjust the second-lowest-level sampling set in combination with the lowest-level aggregation result to obtain a target second-lowest-level data set; Performing centralized value aggregation processing on the target second-lowest level data set to obtain a second-lowest level aggregation result; The target data set is constructed based on the lowest level aggregation result and the second lowest level aggregation result.
7. The distributed processing method for census data combined with the Internet of Things as claimed in claim 6, characterized in that: The target visible graph is sampled and aggregated to obtain a target data set, and a central processing unit is activated to analyze the target data set to obtain a first eigenvalue, including: Using the target data set as input information of a census prediction model, and obtaining output information through the census prediction model, wherein the output information includes the first eigenvalue; The census prediction model refers to an intelligent prediction model obtained by supervised training of historical census data sets based on the principle of neural networks.
8. The method for distributed processing of census data in combination with the Internet of Things as claimed in claim 1, characterized in that: Also includes: Performing matrix-vector processing on the target visibility graph to obtain a target matrix vector; The target matrix vector is decomposed to obtain a second eigenvalue, and the second eigenvalue is used as the census characteristic coefficient, wherein the census characteristic coefficient is used to quantitatively characterize the census data set.
9. The census data distributed processing platform combined with the Internet of Things is characterized by: The platform is used to execute the distributed processing method of census data combined with the Internet of Things as described in any one of claims 1 to 8, including: Data preprocessing module: introduces a predetermined structured strategy to preprocess the census data set to obtain a structured census data set; Visualization processing module: performs visualization processing on the structured census data set to obtain a visual graph of the census data; A marginal census data set composition module: extracting data within a predetermined level threshold in the census data visibility diagram to compose a marginal census data set; Distributed edge processing module: performing distributed edge processing on the edge census data set by edge processors deployed at the predetermined level threshold to obtain edge processing results; An edge processing result rendering module is configured to render the edge processing result to the census data visual graph to obtain a target visual graph; A data set analysis module: performs sampling and aggregation processing on the target visible graph to obtain a target data set, and activates a central processing unit to analyze the target data set to obtain a first eigenvalue; A census data set quantification module: uses the first eigenvalue as a census characteristic coefficient, wherein the census characteristic coefficient is used to quantitatively characterize the census data set.