Data processing analysis system and method for urban safety development index evaluation

Abstract

The invention discloses a data processing analysis system and method for urban safety development index evaluation, and the system comprises an obtaining module which is used for obtaining the collection data of the urban safety development index evaluation; the first determination module is used for performing data structure identification processing on the acquired data, determining structured data and unstructured data, performing data conversion on the unstructured data, and obtaining format data based on a conversion result and the structured data; the desensitization module is used for desensitizing the format data to obtain desensitized data; and the processing module is used for processing and analyzing the desensitization data to obtain a processing and analyzing result. According to the method, desensitization processing is accurately carried out, data leakage is avoided, meanwhile, the data size is reduced, the analysis rate is improved, and a data processing analysis result is comprehensively and accurately obtained in the data analysis processing process.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing and analysis system and method for evaluating a city safety development index. Background Art

[0002] At present, most domestic and foreign development evaluations use the development index method, which is a data standard for measuring the development level of a certain field. Therefore, the urban safety development index evaluation is often used to determine the regional development and evaluation system. However, in the process of urban safety development index evaluation, it is necessary to collect big data and perform data processing and analysis on the big data. Based on the data processing and analysis results, the urban safety development is evaluated. There are the following technical problems in data processing and analysis: 1. Sensitive data is involved in big data, but it is impossible to accurately perform desensitization processing, resulting in data leakage and large data volume problems; 2. In the process of data analysis and processing, it is relatively biased and the data processing and analysis results cannot be obtained comprehensively and accurately. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems in the above-mentioned technologies to a certain extent. To this end, the purpose of the present invention is to propose a data processing and analysis system and method for urban safety development index evaluation, accurately perform desensitization processing, avoid data leakage, reduce the amount of data, and facilitate improving the analysis rate. In the process of data analysis and processing, comprehensive and accurate data processing and analysis results are obtained.

[0004] To achieve the above purpose, the embodiment of the present invention proposes a data processing and analysis system for urban safety development index evaluation, including:

[0005] The acquisition module is used to obtain the collected data for the urban safety development index assessment;

[0006] The first determination module is used to perform data structure recognition processing on the collected data, determine structured data and unstructured data, perform data conversion on the unstructured data, and obtain format data based on the conversion result and the structured data;

[0007] The desensitization module is used to perform desensitization processing on the format data to obtain desensitized data;

[0008] The processing module is used to process and analyze the desensitized data to obtain processing and analysis results.

[0009] According to some embodiments of the present invention, the acquisition module includes:

[0010] The first acquisition module is used to obtain economic development big data related to the city's economic indicator factors;

[0011] The second acquisition module is used to obtain big data on social progress related to the city’s science and technology and education level factors;

[0012] The third acquisition module is used to obtain accident risk big data related to accident factors in the city;

[0013] The second determination module is used to determine the collected data based on the big data of economic development, the big data of social progress and the big data of accident risks.

[0014] According to some embodiments of the present invention, the first determining module includes:

[0015] An identification and processing module is used to check the format and content of the collected data, perform data structure identification processing on the collected data based on a script or an identification tool, and determine structured data and unstructured data; wherein the identification tool includes a regular expression and a machine learning model; the structured data is stored in a relational database and has fixed fields and formats; the unstructured data has fixed fields and formats;

[0016] The extraction module is used to extract features from unstructured data, obtain feature vectors, and convert them into a target format; and obtain format data based on the data converted into the target format and the structured data.

[0017] According to some embodiments of the present invention, the extraction module includes:

[0018] The first extraction module is used to perform word segmentation and part-of-speech tagging on the text data included in the unstructured data based on natural language processing technology, and extract text topics using topic modeling technology;

[0019] A second extraction module is used to perform image recognition and classification on image data included in the unstructured data using a deep learning model, and extract visual features of the image, including color features and texture features;

[0020] The third extraction module is used to perform speech recognition on the audio data included in the unstructured data, and extract the frequency spectrum characteristics and volume characteristics of the audio.

[0021] According to some embodiments of the present invention, the desensitization module includes:

[0022] Identification module for:

[0023] Match the formatted data with the preset data identification strategy to obtain a matching score;

[0024] Identify the format data based on the identification strategy corresponding to the highest matching score, and extract attribute information of the format data; the attribute information includes length, character type, symbol, and format feature;

[0025] Distribution module for:

[0026] Cluster analysis is performed based on the attribute information of the format data to obtain several sensitive cluster sets;

[0027] Set desensitization configuration information and register multiple data desensitizers to form a desensitizer chain; each data desensitizer processes sensitive data with different characteristics;

[0028] Transmitting several sensitive cluster sets from the first node of the desensitizer chain and distributing them to the nodes of the corresponding data desensitizers;

[0029] The detection module is used to detect an interruption in the data desensitization process and restart the thread or service performing the data desensitization process; after restarting the thread or service performing the data desensitization process, determine the recorded location and perform the desensitization process again during the restart to obtain the desensitized data.

[0030] According to some embodiments of the present invention, the processing module includes:

[0031] Generate modules for:

[0032] Traverse the desensitized data, construct all the feature vectors in the desensitized data, and form a feature vector matrix;

[0033] Using a ball tree construction algorithm, each eigenvector in the eigenvector matrix is ​​recursively grouped into a nested ball tree, and a ball tree index space in which the eigenvectors are nested is obtained;

[0034] Based on the ball tree index space, density-based clustering is performed on all data points to obtain multiple clusters; a unique identification code is assigned to each cluster to generate a feature node;

[0035] Establishing a module for establishing a node index structure according to the correlation between each cluster;

[0036] A first analysis module, used to obtain a first processing and analysis result by processing and analyzing the node index structure;

[0037] The second analysis module is used to run the Spark script to perform processing and analysis to obtain a second processing and analysis result, and to obtain a processing and analysis result based on the first processing and analysis result and the second processing and analysis result.

[0038] According to some embodiments of the present invention, the first analysis module includes:

[0039] A third determination module is used to determine the characteristic value of each node according to the node index structure;

[0040]

[0041] Among them, S(x) is the eigenvalue of the x-th node; L is the number of elements of the x-th node; α i is the importance weight of the i-th element; Y(x) i is the eigenvalue of the i-th element of the x-th node; μ is the mean of the eigenvalues ​​of the elements of the x-th node; γ is the sensitivity coefficient for adjusting the eigenvalue, and its value range is (0,1);

[0042] The fusion module is used to perform feature fusion based on the eigenvalue of each node in a weighted manner to obtain a data fusion result as a processing and analysis result.

[0043] According to some embodiments of the present invention, the second analysis module includes:

[0044] Configuration generation module, used to select Spark data analysis related components based on desensitized data, complete Spark script configuration according to data analysis related components, and generate Spark script;

[0045] The execution management module is used to create the corresponding execution environment and submit tasks according to the tasks that the Spark script needs to execute, and to view the number of executions, running status, and processed data of the tasks;

[0046] The result analysis module is used to analyze the Spark script execution result to obtain the second processing analysis result.

[0047] According to some embodiments of the present invention, it also includes: a visualization processing module, which is used to visualize the processing and analysis results, obtain visualization information and display it.

[0048] In one embodiment, an analysis method of a data processing and analysis system based on urban safety development index evaluation includes:

[0049] Obtain the collected data for the urban safety development index assessment;

[0050] Perform data structure recognition processing on the collected data, determine structured data and unstructured data, perform data conversion on the unstructured data, and obtain format data based on the conversion results and structured data;

[0051] Desensitizing the format data to obtain desensitized data;

[0052] The desensitized data is processed and analyzed to obtain the processing and analysis results.

[0053] The present invention proposes a data processing and analysis system and method for urban safety development index evaluation, which can accurately perform desensitization processing to avoid data leakage, reduce the amount of data, and facilitate improving the analysis rate. In the process of data analysis and processing, comprehensive and accurate data processing and analysis results are obtained.

[0054] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0055] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0057] Figure 1 is a block diagram of a data processing and analysis system for urban safety development index evaluation according to an embodiment of the present invention;

[0058] Figure 2 is a block diagram of an acquisition module according to an embodiment of the present invention;

[0059] Figure 3 It is a flow chart of a data processing and analysis method for urban safety development index evaluation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0061] like Figure 1 As shown, the embodiment of the present invention proposes a data processing and analysis system for evaluating the urban safety development index, including:

[0062] The acquisition module is used to obtain the collected data for the urban safety development index assessment;

[0063] The first determination module is used to perform data structure recognition processing on the collected data, determine structured data and unstructured data, perform data conversion on the unstructured data, and obtain format data based on the conversion result and the structured data;

[0064] The desensitization module is used to perform desensitization processing on the format data to obtain desensitized data;

[0065] The processing module is used to process and analyze the desensitized data to obtain processing and analysis results.

[0066] The working principle of the above technical solution: The acquisition module acquires data in the following ways: acquiring data from multiple data sources based on a data interface and crawling data based on web crawler technology. Acquire the collected data of the urban safety development index assessment, and convert the collected data into format data for data processing. Desensitize the format data to obtain desensitized data, and process and analyze the desensitized data to obtain processing and analysis results.

[0067] The beneficial effects of the above technical solution are: accurate desensitization processing to avoid data leakage, while reducing the amount of data to facilitate improving the analysis rate, and in the process of data analysis and processing, comprehensive and accurate data processing and analysis results are obtained.

[0068] like Figure 2 As shown, according to some embodiments of the present invention, the acquisition module includes:

[0069] The first acquisition module is used to obtain economic development big data related to the city's economic indicator factors;

[0070] The second acquisition module is used to obtain big data on social progress related to the city’s science and technology and education level factors;

[0071] The third acquisition module is used to obtain accident risk big data related to accident factors in the city;

[0072] The second determination module is used to determine the collected data based on the big data of economic development, the big data of social progress and the big data of accident risks.

[0073] The working principle and beneficial effects of the above technical solution: Economic development big data include: GDP per capita, GDP growth rate, the proportion of tertiary industry output value in GDP, the proportion of high-tech product output value in total industrial output value and urbanization rate; Social progress big data include: Research and experimental development expenditure as a proportion of GDP, the number of three national patent authorizations per 100,000 people, average years of education, average life expectancy, work injury insurance coverage rate and urban registered unemployment rate; Accident risk big data include: Accident mortality rate per 100 million yuan of GDP, accident mortality rate per 100,000 employees in industry, mining, commerce and trade, occupational disease incidence rate per 100,000 employees in industry, mining, commerce and trade, road traffic mortality rate per 10,000 vehicles and fire mortality rate per 100,000 people. Based on the first acquisition module, the second acquisition module and the third acquisition module, it is convenient to accurately determine the collected data.

[0074] According to some embodiments of the present invention, the first determining module includes:

[0075] An identification and processing module is used to check the format and content of the collected data, perform data structure identification processing on the collected data based on a script or an identification tool, and determine structured data and unstructured data; wherein the identification tool includes a regular expression and a machine learning model; the structured data is stored in a relational database and has fixed fields and formats; the unstructured data has fixed fields and formats;

[0076] The extraction module is used to extract features from unstructured data, obtain feature vectors, and convert them into a target format; and obtain format data based on the data converted into the target format and the structured data.

[0077] The working principle and beneficial effects of the above technical solution are as follows: Check the format (such as file type, encoding method, etc.) and content (such as whether it contains specific fields, whether the field value is legal, etc.) of the data. Use regular expressions to preliminarily parse the data and identify structured data with fixed fields and formats. For data that cannot be accurately identified by regular expressions, use machine learning models for further analysis to improve the accuracy and flexibility of identification. The identified structured data is stored in a relational database to ensure the integrity and consistency of the data. For unstructured data, it is marked and passed to the extraction module for processing. Use natural language processing (NLP), image recognition and other technologies to extract features from unstructured data to obtain key feature vectors that can reflect the content of the data. Convert the extracted feature vectors to the target format for integration with structured data. The target format can be a standardized data structure (such as JSON, XML, etc.) or a table structure in a relational database. Combine the converted unstructured data with the existing structured data to generate format data in a unified format. Distinguish between structured data and unstructured data, and perform feature extraction and format conversion on the unstructured data to obtain format data in a unified format.

[0078] According to some embodiments of the present invention, the extraction module includes:

[0079] The first extraction module is used to perform word segmentation and part-of-speech tagging on the text data included in the unstructured data based on natural language processing technology, and extract text topics using topic modeling technology;

[0080] A second extraction module is used to perform image recognition and classification on image data included in the unstructured data using a deep learning model, and extract visual features of the image, including color features and texture features;

[0081] The third extraction module is used to perform speech recognition on the audio data included in the unstructured data, and extract the frequency spectrum characteristics and volume characteristics of the audio.

[0082] The working principle and beneficial effects of the above technical solution: Receive text data in unstructured data. Use NLP tools (such as ji eba word segmenter, NLTK, etc.) to segment the text, that is, divide the continuous text into independent words. Perform part-of-speech tagging on the word segmentation results, and assign a part-of-speech tag (such as noun, verb, adjective, etc.) to each word for subsequent analysis. Use topic modeling technology (such as LDA, PLSA, etc.) to extract topics from the text that is segmented and annotated with part-of-speech. Topic modeling technology can identify the potential topic structure in the text and classify similar text content under the same topic. The extracted topics can be used for subsequent data analysis, text classification or information retrieval tasks. Receive image data in unstructured data. Use a pre-trained deep learning model (such as convolutional neural network CNN) to identify and classify images. The deep learning model can automatically learn the feature representation in the image and output the category label to which the image belongs. Color features: Calculate statistics such as the mean and standard deviation of each color channel in the image to reflect the color distribution of the image. Texture features: Use texture analysis methods (such as gray level co-occurrence matrix GLCM) to extract the texture information of the image to reflect the texture pattern and structure of the image. Receive audio data from unstructured data. Use speech recognition technology (such as ASR system) to convert audio into text or extract speech features from audio. Speech recognition technology can recognize the speech content or speech features in audio, providing a basis for subsequent audio analysis. Perform spectral analysis on the recognized audio data to extract the spectral features of the audio. Spectral features reflect the distribution and changes of audio signals in the frequency domain and can be used for tasks such as audio classification and recognition. At the same time, extract the volume features of the audio, such as volume size, volume change, etc., to reflect the volume information of the audio. Based on the extraction module, it is convenient to accurately extract feature vectors in unstructured data and convert them into the target format.

[0083] According to some embodiments of the present invention, the desensitization module includes:

[0084] Identification module for:

[0085] Match the formatted data with the preset data identification strategy to obtain a matching score;

[0086] Identify the format data based on the identification strategy corresponding to the highest matching score, and extract attribute information of the format data; the attribute information includes length, character type, symbol, and format feature;

[0087] Distribution module for:

[0088] Cluster analysis is performed based on the attribute information of the format data to obtain several sensitive cluster sets;

[0089] Set desensitization configuration information and register multiple data desensitizers to form a desensitizer chain; each data desensitizer processes sensitive data with different characteristics;

[0090] Transmitting several sensitive cluster sets from the first node of the desensitizer chain and distributing them to the nodes of the corresponding data desensitizers;

[0091] The detection module is used to detect an interruption in the data desensitization process and restart the thread or service performing the data desensitization process; after restarting the thread or service performing the data desensitization process, determine the recorded location and perform the desensitization process again during the restart to obtain the desensitized data.

[0092] The working principle and beneficial effects of the above technical solution: the preset data identification strategy includes regular expressions, dictionaries, and keyword identification rules for specific types of privacy data, and the identification rules are combined in multiple modes to form multiple data identification strategies; the format data is identified based on the identification strategy corresponding to the highest matching score, and the attribute information of the format data is extracted; the attribute information includes length, character type, symbol, and format characteristics; the main task of the allocation module is to perform cluster analysis based on the attribute information of the format data, and assign the corresponding data desensitizer for processing. Cluster analysis is performed on the attribute information of the format data using clustering algorithms (such as K-means, DBSCAN, etc.). According to the clustering results, the format data is divided into several sensitive cluster sets. Set the desensitization configuration information, register multiple data desensitizers, and each desensitizer is responsible for processing sensitive data with different characteristics. According to the characteristics of the sensitive cluster set, configure the corresponding desensitization strategy to form a desensitizer chain. Transmit the sensitive cluster set from the first node of the desensitizer chain. According to the characteristics of the sensitive data, assign it to the corresponding data desensitizer node for processing. During the data desensitization process, real-time monitoring is performed to see if there are any interruptions (such as system crashes, network failures, etc.). Once an interruption is detected, the restart mechanism is immediately started to re-desensitize the thread or service. After the restart, the recorded location (such as log files, database records, etc.) is determined. Starting from the recorded location, desensitization is re-performed to ensure the integrity and accuracy of the data. The desensitization module can efficiently process sensitive information in formatted data to ensure the privacy and security of the data. At the same time, the reliability and stability of data desensitization are improved through interruption detection and breakpoint resumption mechanisms.

[0093] According to some embodiments of the present invention, the processing module includes:

[0094] Generate modules for:

[0095] Traverse the desensitized data, construct all the feature vectors in the desensitized data, and form a feature vector matrix;

[0096] Using a ball tree construction algorithm, each eigenvector in the eigenvector matrix is ​​recursively grouped into a nested ball tree, and a ball tree index space in which the eigenvectors are nested is obtained;

[0097] Based on the ball tree index space, density-based clustering is performed on all data points to obtain multiple clusters; a unique identification code is assigned to each cluster to generate a feature node;

[0098] Establishing a module for establishing a node index structure according to the correlation between each cluster;

[0099] A first analysis module, used to obtain a first processing and analysis result by processing and analyzing the node index structure;

[0100] The second analysis module is used to run the Spark script to perform processing and analysis to obtain a second processing and analysis result, and to obtain a processing and analysis result based on the first processing and analysis result and the second processing and analysis result.

[0101] The working principle of the above technical solution: In this embodiment, a ball tree construction algorithm is used to recursively group the individual feature vectors in the feature vector matrix into a nested ball tree to obtain a ball tree index space nested with feature vectors, including: input feature vector matrix X, where each row is a feature vector. Set the depth of the tree or the recursive termination condition (for example, the number of feature vectors in the node is less than a certain threshold). Calculate the centroid (center point) and radius of all feature vectors so that all feature vectors are within this ball. Create a root node containing this ball (center of mass and radius) and all feature vectors. Recursively construct child nodes: For the feature vector in the current node, if the termination condition is met (for example, the number of feature vectors is less than the threshold), stop the recursion. Otherwise, select a segmentation method (for example, based on the farthest point pair or randomly select a segmentation point) to divide the feature vector into two parts. Recursively construct child nodes for each part, each child node is also a ball containing its corresponding feature vector. Finally, return the root node, which contains the entire nested ball tree structure.

[0102] In this embodiment, data points are extracted from the ball tree index space. This can be achieved by traversing the nodes of the ball tree, or directly recursively accessing the data points in all leaf nodes starting from the root node. Before executing DBSCAN, two key parameters need to be determined: ε (the size of the neighborhood) and MinPts (the minimum number of neighborhood points required to become a core object). For each data point, all points in its ε neighborhood are found, and the number of points in its neighborhood is calculated. According to the number of points in the neighborhood, the data points are classified as core points, boundary points or noise points. Through density accessibility, the core points and the points that are density-reachable from the core points are divided into the same cluster cluster. After the clustering is completed, a unique ID or label is assigned to each cluster cluster. Based on the ID and data points of the cluster cluster, a feature node is constructed. The feature node can be a data structure (such as a class instance or a dictionary) containing the ID of the cluster cluster, the centroid (which can be obtained by calculating the mean value of all points in the cluster cluster), the radius (which can be defined according to the shape or size of the cluster cluster), the number of data points included, and other attributes.

[0103] In this embodiment, a module is established, a comprehensive class is created, and necessary properties and methods are defined in the comprehensive class. A list containing clusters is determined (each cluster can be an object or a data structure, containing the ID and data points of the cluster); a matrix or dictionary containing the association information between clusters is determined to represent the relative relationship between clusters. A tree structure is determined based on the matrix or dictionary of the association information between clusters. Using the tree structure, a tree can be constructed based on the hierarchy of the association information (such as the threshold of distance or similarity). Methods are defined in the comprehensive class to access and query the index structure and establish a node index structure.

[0104] In this embodiment, Spark can process large-scale data sets in a distributed manner, and implement data analysis from the level. Data analysis is performed from the node based on the first analysis module.

[0105] The beneficial effects of the above technical solution are as follows: for desensitized data, feature vectors are extracted and converted into ball tree index space, density-based clustering is performed on all data points to obtain multiple clusters; a unique identification code is assigned to each cluster to generate feature nodes; a module is established to establish a node index structure according to the correlation between each cluster; based on the first analysis module, analysis and processing of nodes are implemented, and based on the second analysis module, Spark scripts are run for processing and analysis, to implement analysis and processing of levels, and to improve the comprehensiveness and accuracy of data analysis and processing.

[0106] According to some embodiments of the present invention, the first analysis module includes:

[0107] A third determination module is used to determine the characteristic value of each node according to the node index structure;

[0108]

[0109] Among them, S(x) is the eigenvalue of the x-th node; L is the number of elements of the x-th node; α i is the importance weight of the i-th element; Y(x) i is the eigenvalue of the i-th element of the x-th node; μ is the mean of the eigenvalues ​​of the elements of the x-th node; γ is the sensitivity coefficient for adjusting the eigenvalue, and its value range is (0,1);

[0110] The fusion module is used to perform feature fusion based on the eigenvalue of each node in a weighted manner to obtain a data fusion result as a processing and analysis result.

[0111] The working principle and beneficial effects of the above technical solution: It involves the quantitative processing of the internal elements (or features) of the node, and the calculation of the overall eigenvalue of the node by a specific formula. For each node, the number of elements inside it is determined. For each element, a weight is assigned according to its importance. This weight can be determined based on factors such as the type, position, and frequency of occurrence of the element. The eigenvalue of each element is calculated, which is the direct value of the element, the value after conversion or normalization, etc. The mean of the eigenvalues ​​of all elements inside the node is calculated. Based on the sensitivity coefficient of the adjusted eigenvalue, it is used to adjust the sensitivity of the eigenvalue to the difference between the element and the mean. For each node, a fusion weight is assigned according to its eigenvalue and other factors (such as the importance of the node, the position relationship, etc.). In one embodiment, the greater the importance, the closer the position, and the greater the fusion weight. The eigenvalues ​​of each node are fused using a weighted method to obtain a data fusion result. The first analysis module can efficiently process and analyze the data in the node index structure, extract useful feature information, and generate the final processing and analysis results.

[0112] According to some embodiments of the present invention, the second analysis module includes:

[0113] Configuration generation module, used to select Spark data analysis related components based on desensitized data, complete Spark script configuration according to data analysis related components, and generate Spark script;

[0114] The execution management module is used to create the corresponding execution environment and submit tasks according to the tasks that the Spark script needs to execute, and to view the number of executions, running status, and processed data of the tasks;

[0115] The result analysis module is used to analyze the Spark script execution result to obtain the second processing analysis result.

[0116] Working principle and beneficial effects of the above technical solution: Data analysis related components include global parameter components, logical operation components, input and output components, SQL processing components, MapReduce components, calculation components, clustering components and regression components. According to factors such as the type, scale and analysis requirements of the desensitized data, select appropriate components from the many data analysis components provided by Spark. According to the selected components, configure the corresponding parameters, dependency libraries, input data paths, output data paths, etc. Write a Spark script, which will call the selected components and perform the corresponding data analysis tasks according to the configuration. Save the configured script to the file system to generate an executable Spark script. According to the requirements of the Spark script, create or configure the execution environment of the Spark cluster, including resource allocation (such as CPU, memory), node configuration, etc. Use the command line tool or API provided by Spark to submit the generated Spark script to the Spark cluster for execution. Monitor the execution process of the task, including the number of executions of the task (such as the number of retries), the running status (such as running, success, failure, etc.), the amount of data processed, etc. Provide a user-friendly interface or log so that users can view the execution status of the task at any time. Extract the execution results from the output of the Spark task. By configuring the collaborative work of the generation module, the execution management module, and the result analysis module, the second analysis module can efficiently use the Spark framework to perform large-scale data analysis tasks on the desensitized data, generate useful analysis results, and achieve hierarchical data analysis.

[0117] According to some embodiments of the present invention, it also includes: a visualization processing module, which is used to visualize the processing and analysis results, obtain visualization information and display it.

[0118] like Figure 3 As shown, an analysis method of a data processing and analysis system based on urban safety development index evaluation includes steps S1-S4:

[0119] S1. Obtain the collected data for urban safety development index assessment;

[0120] S2. Perform data structure recognition processing on the collected data to determine structured data and unstructured data, perform data conversion on the unstructured data, and obtain format data based on the conversion result and the structured data;

[0121] S3, desensitizing the format data to obtain desensitized data;

[0122] S4. Process and analyze the desensitized data to obtain processing and analysis results.

[0123] The beneficial effects of the above technical solution are: accurate desensitization processing to avoid data leakage, while reducing the amount of data to facilitate improving the analysis rate, and in the process of data analysis and processing, comprehensive and accurate data processing and analysis results are obtained.

[0124] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data processing and analysis system for urban safety development index evaluation, characterized in that: include: The acquisition module is used to obtain the collected data for the urban safety development index assessment; The first determination module is used to perform data structure recognition processing on the collected data, determine structured data and unstructured data, perform data conversion on the unstructured data, and obtain format data based on the conversion result and the structured data; The desensitization module is used to perform desensitization processing on the format data to obtain desensitized data; The processing module is used to process and analyze the desensitized data to obtain processing and analysis results.

2. The data processing and analysis system for evaluating the urban safety development index according to claim 1, characterized in that: The acquisition module comprises: The first acquisition module is used to obtain economic development big data related to the city's economic indicator factors; The second acquisition module is used to obtain big data on social progress related to the city’s science and technology and education level factors; The third acquisition module is used to obtain accident risk big data related to accident factors in the city; The second determination module is used to determine the collected data based on the big data of economic development, the big data of social progress and the big data of accident risks.

3. The data processing and analysis system for evaluating the urban safety development index according to claim 1, characterized in that: The first determining module includes: An identification and processing module is used to check the format and content of the collected data, perform data structure identification processing on the collected data based on a script or an identification tool, and determine structured data and unstructured data; wherein the identification tool includes a regular expression and a machine learning model; the structured data is stored in a relational database and has fixed fields and formats; the unstructured data has fixed fields and formats; The extraction module is used to extract features from unstructured data, obtain feature vectors, and convert them into a target format; and obtain format data based on the data converted into the target format and the structured data.

4. The data processing and analysis system for evaluating the urban safety development index according to claim 3, wherein the extraction module comprises: The first extraction module is used to perform word segmentation and part-of-speech tagging on the text data included in the unstructured data based on natural language processing technology, and extract text topics using topic modeling technology; A second extraction module is used to perform image recognition and classification on image data included in the unstructured data using a deep learning model, and extract visual features of the image, including color features and texture features; The third extraction module is used to perform speech recognition on the audio data included in the unstructured data, and extract the frequency spectrum characteristics and volume characteristics of the audio.

5. The data processing and analysis system for urban safety development index evaluation according to claim 1, wherein the desensitization module comprises: Identification module for: Match the formatted data with the preset data identification strategy to obtain a matching score; Identify the format data based on the identification strategy corresponding to the highest matching score, and extract attribute information of the format data; the attribute information includes length, character type, symbol, and format feature; Distribution module for: Cluster analysis is performed based on the attribute information of the format data to obtain several sensitive cluster sets; Set desensitization configuration information, register multiple data desensitizers, and form a desensitizer chain; Each data desensitizer processes sensitive data with different characteristics; Transmitting several sensitive cluster sets from the first node of the desensitizer chain and distributing them to the nodes of the corresponding data desensitizers; A detection module is used to detect an interruption in the data desensitization process and restart the thread or service performing the data desensitization process; After restarting the thread or service that performs data desensitization processing, determine the location that has been recorded, and perform desensitization processing again when restarting to obtain desensitized data.

6. The data processing and analysis system for evaluating the urban safety development index according to claim 1, wherein the processing module comprises: Generate modules for: Traverse the desensitized data, construct all the feature vectors in the desensitized data, and form a feature vector matrix; Using a ball tree construction algorithm, each eigenvector in the eigenvector matrix is ​​recursively grouped into a nested ball tree, and a ball tree index space in which the eigenvectors are nested is obtained; Based on the ball tree index space, density-based clustering is performed on all data points to obtain multiple clusters; a unique identification code is assigned to each cluster to generate a feature node; Establishing a module for establishing a node index structure according to the correlation between each cluster; A first analysis module, used to obtain a first processing and analysis result by processing and analyzing the node index structure; The second analysis module is used to run the Spark script to perform processing and analysis to obtain a second processing and analysis result, and to obtain a processing and analysis result according to the first processing and analysis result and the second processing and analysis result.

7. The data processing and analysis system for evaluating the urban safety development index according to claim 6, characterized in that: The first analysis module comprises: A third determination module is used to determine the characteristic value of each node according to the node index structure; Among them, S(x) is the eigenvalue of the x-th node; L is the number of elements of the x-th node; α i is the importance weight of the i-th element; Y(x) i is the eigenvalue of the i-th element of the x-th node; μ is the mean of the eigenvalues ​​of the elements of the x-th node; γ is the sensitivity coefficient for adjusting the eigenvalue, and its value range is (0,1); The fusion module is used to perform feature fusion based on the eigenvalue of each node in a weighted manner to obtain a data fusion result as a processing and analysis result.

8. The data processing and analysis system for evaluating the urban safety development index according to claim 6, characterized in that: The second analysis module comprises: Configuration generation module, used to select Spark data analysis related components based on desensitized data, complete Spark script configuration according to data analysis related components, and generate Spark script; The execution management module is used to create the corresponding execution environment and submit tasks according to the tasks that the Spark script needs to execute, and to view the number of executions, running status, and processed data of the tasks; The result analysis module is used to analyze the Spark script execution result to obtain the second processing analysis result.

9. The data processing and analysis system for evaluating the urban safety development index according to claim 1, characterized in that: Also includes: The visualization processing module is used to visualize the processing and analysis results, obtain visualization information and display it.

10. The analysis method of the data processing and analysis system for urban safety development index evaluation according to any one of claims 1 to 9, characterized in that: include: Obtain the collected data for the urban safety development index assessment; Perform data structure recognition processing on the collected data, determine structured data and unstructured data, perform data conversion on the unstructured data, and obtain format data based on the conversion results and structured data; Desensitizing the format data to obtain desensitized data; The desensitized data is processed and analyzed to obtain the processing and analysis results.

Images