An intelligent processing and analysis system for public data based on artificial intelligence

Abstract

The present invention discloses an artificial intelligence-based intelligent processing and analysis system for public data, which belongs to the technical field of public data processing and includes a data main end and an application end; the data main end includes a demand analysis module, a data acquisition module, and a subject database; the demand analysis module is used to determine the scene range and the public data range corresponding to the potential scene according to the management needs of the subject party, and set the data processing method according to the public data range; the data acquisition module is used to perform real-time data acquisition according to the public data range and the data processing method, obtain public data, and perform virtualization processing on the public data to obtain virtualized data; the subject database is used to store and manage the virtualized data; the application end includes a permission module, an application database and an application analysis module; the permission module is used to verify the permission of the logged-in user; the application database is used to store the public data of the application user; the demand analysis module is used to analyze the public data and obtain analysis results.

Description

Technical Field

[0001] The present invention belongs to the technical field of public data processing, and specifically is a public data intelligent processing and analysis system based on artificial intelligence. Background Art

[0002] In today's era of rapid information development, the processing and application of public data has become a crucial force driving government decision-making, social governance, and economic development. However, for a long time, each department has operated independently, establishing its own public data processing systems. While this approach has met the data processing needs of each department to a certain extent, its drawbacks have gradually become apparent over time. First, the data processing systems of various departments lack unified standards and specifications. Due to differences in data processing needs, technologies, and resources, the systems they establish often use different data structures, algorithms, and interfaces. This disparity not only leads to data incompatibility but also increases the difficulty of data sharing and exchange. When cross-departmental data integration and analysis are required, data format conversion and cleansing often consume a significant amount of time and resources, seriously affecting the efficiency and accuracy of data processing. This lack of unified standards not only hinders the circulation and application of data but also limits the maximization of data value. Second, the existence of multiple data processing systems can lead to resource waste. Each system requires independent hardware and software support, which not only increases costs but also consumes a large amount of computing and storage resources. Furthermore, due to the varying data processing needs of various departments, many systems are often idle or semi-idle, resulting in a significant waste of resources. This waste of resources not only increases the government's financial burden, but also limits the investment and development of other departments in data processing.

[0003] Based on this, the present invention provides a public data intelligent processing and analysis system based on artificial intelligence. Summary of the Invention

[0004] In order to solve the problems existing in the above solutions, the present invention provides a public data intelligent processing and analysis system based on artificial intelligence.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] An artificial intelligence-based public data intelligent processing and analysis system, including a data terminal and an application terminal;

[0007] The data terminal includes a demand analysis module, a data acquisition module, and a main database;

[0008] The demand analysis module is used to determine the scenario range according to the management needs of the subject party. The scenario range is composed of several potential scenarios, set the public data range corresponding to the potential scenarios, and set the data processing method corresponding to the potential scenarios according to the public data range.

[0009] Furthermore, the method of setting the public data range of potential scenarios includes:

[0010] Reference data ranges of several potential scenarios are obtained, and the reference data ranges corresponding to the potential scenarios are merged to form a common data range of the potential scenarios.

[0011] Furthermore, before merging the reference data ranges of the potential scenarios, the reference data ranges are corrected, and then merged after the reference data ranges are corrected.

[0012] Furthermore, the public data range of the potential scenario is adjusted in real time, and the adjustment method includes: obtaining the range adjustment requirement of the potential scenario in real time, and adjusting the public data range according to the range adjustment requirement.

[0013] The data acquisition module is used to perform real-time data acquisition based on the public data range and data processing method of the potential scene, obtain the public data of the potential scene, virtualize the public data, and obtain virtualized data corresponding to the potential scene; and tag the virtualized data with the corresponding potential scene label and send it to the main database.

[0014] Furthermore, the method for virtualizing public data is as follows:

[0015] Step SA1: The platform presets a blur processing method and blur processing requirements for the potential scene, and marks the blur processing method preset by the platform as a reference processing method;

[0016] Step SA2: performing blurring processing on the public data of the potential scene using the reference processing method to obtain blurring data;

[0017] Step SA3: obtaining a candidate blurring method in real time according to the blurring processing requirement, comparing the candidate blurring method with a reference processing method, and determining whether the blurring effect of the candidate blurring method is better than that of the reference processing method;

[0018] When it is determined that the blurring effect of the candidate blurring method is better than that of the reference processing method, the candidate blurring method replaces the reference processing method in step SA2 to form a new reference processing method;

[0019] When it is determined that the blurring effect of the selected blurring method is not better than that of the reference processing method, no corresponding processing is performed;

[0020] Step SA4: Loop step SA3.

[0021] Furthermore, the method of comparing the selected blurring method with the reference processing method includes:

[0022] Set optimization indicators and their corresponding weight coefficients. The optimization indicators include blur speed, matching speed, and memory usage.

[0023] Extracting features of the selected blurring method using the optimization index item to obtain a selected blurring speed, a selected matching speed, and a selected occupied memory corresponding to the selected blurring method; extracting features of the benchmark processing method using the optimization index item to obtain a benchmark blurring speed, a benchmark matching speed, and a benchmark occupied memory corresponding to the benchmark processing method;

[0024] According to the formula Calculate the optimal value of the selected blurring method;

[0025] Where: PK is the optimized value; β1, β2, and β3 are the weight coefficients of the corresponding optimization index items; HV1, PV1, and ZC1 are the candidate blur speed, candidate matching speed, and candidate occupied memory respectively; HV2, PV2, and ZC2 are the benchmark blur speed, benchmark matching speed, and benchmark occupied memory respectively;

[0026] Determine whether the blurring effect of the candidate blurring method is better than the baseline processing method based on the optimization value.

[0027] The main body database is used to store and manage the received virtualized data.

[0028] The application end includes a permission module, an application database and an application analysis module;

[0029] The permission module is used to verify the permissions of the logged-in user. When the permission verification is successful, the logged-in user is marked as an application user and the usage permission is opened; when the permission verification fails, the use of the application end is prohibited.

[0030] The application database is used to connect to the main database and store public data corresponding to application users. It includes a matching unit and a storage unit. The matching unit is used to obtain public data and store the obtained public data in the storage unit; the storage unit is used to store public data.

[0031] Furthermore, the method for the matching unit to obtain public data includes:

[0032] Acquire application user information of an application user, determine a user scenario corresponding to the application user based on the application user information, match corresponding virtualized data from the subject database based on the user scenario, and acquire public data corresponding to the application user based on the virtualized data.

[0033] The application analysis module is used to analyze public data, obtain corresponding analysis results, and display the obtained analysis results to application users.

[0034] Furthermore, methods for analyzing public data include:

[0035] The platform presets several alternative requirements and sets corresponding data analysis methods for the alternative requirements; sets a requirement column based on the alternative requirements and the data analysis methods, the requirement column includes function keys corresponding to the alternative requirements and function keys corresponding to additional requirements, the additional requirements being data analysis requirements corresponding to non-alternative requirements;

[0036] When the user has an alternative requirement, the user clicks the function key corresponding to the alternative requirement, and the public data in the application database is analyzed according to the data analysis method corresponding to the alternative requirement to obtain the analysis result;

[0037] When the user has additional requirements, he clicks the function key corresponding to the additional requirements, enters the corresponding data analysis requirements, analyzes the public data in the application database according to the data analysis requirements, and obtains the corresponding analysis results.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] By building a unified public data integration and processing platform, the present invention realizes standardized data processing. The platform adopts a unified data structure, algorithm and interface to ensure the compatibility and sharing of data among various application users. This greatly simplifies the process of data format conversion and cleaning, and improves the efficiency and accuracy of data processing. At the same time, unified standards also help promote the widespread application of data and the mining of its value; by integrating the public data processing systems of various application users, the present invention realizes the sharing of hardware and software, avoiding duplication of construction and waste of resources. At the same time, the platform adopts advanced technical means, such as linking, to represent the remaining public data after processing of each scene to reduce overall memory usage. This approach of optimizing resource allocation not only reduces costs, but also improves resource utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0042] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, a public data intelligent processing and analysis system based on artificial intelligence includes a data terminal and an application terminal;

[0044] The data terminal is set up according to the requirements of the unified manager and maintained and updated by the platform. For example, if a certain region's competent department applies to establish a data terminal, the subordinate departments of the competent department can subsequently use the user terminal as the user subject. The data terminal can also be established by a combination of multiple departments. It is suitable for a variety of situations. The unified management value is marked as the subject. The data terminal includes a demand analysis module, a data collection module, and a subject database.

[0045] The demand analysis module is used to determine the scenario scope according to the management needs of the subject party. The scenario scope is composed of various potential scenarios, and the public data scope corresponding to each potential scenario is set. The data processing method corresponding to the corresponding potential scenario is set according to the public data scope.

[0046] Among them, the management needs of the subject party refer to the public data of which fields and scenarios the subject party wants to process and analyze, such as directly marking the corresponding scenarios of each department, such as high school education, junior high school education, elementary school education, etc.; because the scope of the scenario can be directly determined according to management needs, it is generally reported directly to the platform party by the subject party.

[0047] In one embodiment, the public data range of a potential scenario is set by determining what public data the potential scenario may require based on existing methods, thereby forming the public data range.

[0048] In one embodiment, a method for setting a common data range for potential scenarios includes:

[0049] Obtain the reference data ranges of the potential scenarios, that is, the existing public data ranges required for the potential scenarios, marked as reference data ranges. For example, for high school education, its parameter data range is the public data range corresponding to the public data processing and analysis system used by various high schools, as the reference data range;

[0050] Merge each reference data range to form the public data range of the potential scenario; ensure the comprehensiveness of data obtained by users of subsequent corresponding scenarios.

[0051] In one embodiment, due to regional differences in actual situations, some reference data ranges will not be needed in the potential scenarios corresponding to subsequent subjects, resulting in continuous processing and analysis of public data that is not needed subsequently, causing waste of resources. Therefore, in this embodiment, before merging the reference data ranges, the reference data ranges of the potential scenarios are corrected to remove the reference data ranges that will not be applied.

[0052] In one embodiment, the reference data range may be corrected based on an existing range correction method.

[0053] In one embodiment, the method for correcting the reference data range includes:

[0054] Obtaining information about the subject, such as the region, management system, cases, and other relevant data that will influence the determination of the data scope for subsequent potential scenarios;

[0055] A range correction model is established based on a CNN network or a DNN network, and a corresponding training set is manually established for training. The training set includes input data and output data. The input data is the signature scenario, the subject party information and the reference data range. The output data is the corrected reference data range. The corrected reference data range is obtained by analyzing the range correction model after successful training.

[0056] In one embodiment, as it is used in an actual context, the corresponding user may have adjustment requirements for the predetermined public data range, such as increasing the range or decreasing the range, so the public data range needs to be adjusted, and the range adjustment requirements corresponding to the potential scenario are obtained in real time, and the public data range of the potential scenario is adjusted according to the range adjustment requirements.

[0057] In one embodiment, a data processing method corresponding to a corresponding potential scenario is set according to the scope of public data, and is set by the platform according to an existing method. The data processing method is to obtain public data within the scope of public data from various channels in real time, and then perform pre-processing, screening and other preset measures to process the data.

[0058] The data acquisition module is used to collect data in real time based on the public data range and data processing method of each potential scene, obtain the public data corresponding to each potential scene, perform virtualization processing on each public data, obtain virtualized data corresponding to each potential scene, and tag the virtualized data with the corresponding potential scene and send it to the main database.

[0059] Among them, virtualizing public data means not directly collecting its public data by extracting its collection links, etc., because the memory usage of virtualized data such as links is much smaller than that of public data. Therefore, virtualizing public data by presetting corresponding virtualization methods can achieve subsequent rapid collection of corresponding public data based on the virtualized data; this can be set by the platform.

[0060] In one embodiment, the method for virtualizing public data is:

[0061] Step SA1: The platform may preset a blur processing method and blur processing requirements for each potential scene, and mark the blur processing method as a baseline processing method;

[0062] Step SA2: blurring the public data of the potential scene using a baseline processing method;

[0063] Step SA3: According to the blur processing requirement, existing blur processing methods that can achieve the blur processing requirement are obtained in real time, marked as candidate blur processing methods, and the candidate blur processing methods are compared with the reference processing method to determine whether the blur effect of the candidate blur processing method is better than that of the reference processing method;

[0064] When it is determined that the blurring effect of the candidate blurring method is better than that of the reference processing method, the candidate blurring method replaces the reference processing method to form a new reference processing method, and the new reference processing method is updated and replaced in the supplementary SA2;

[0065] When it is judged that the blurring effect of the selected blurring method is not better than that of the reference processing method, no corresponding processing is performed;

[0066] Step SA4: Loop step SA3.

[0067] In one embodiment, the candidate blurring method is compared with the reference processing method, and the comparison can be based on an existing blurring effect evaluation method.

[0068] In one embodiment, the method of comparing the selected blurring method with the reference processing method includes:

[0069] Setting optimization index items and weight coefficients corresponding to the optimization index items, including blurring speed, matching speed and memory usage; blurring speed refers to the speed of converting preset public data into corresponding blurring data, matching speed refers to the speed of subsequently matching blurring data based on preset public data to corresponding public data; memory usage refers to the memory usage of blurring data formed according to the blurring processing method; that is, it is determined using standard public data; the weight coefficient is set by the platform according to the needs of the subject party, or by the platform according to management needs;

[0070] By optimizing the index items, feature extraction is performed on the selected blurring method to obtain the blurring speed, matching speed, and occupied memory corresponding to the selected blurring method, and in order to distinguish them, they are marked as the selected blurring speed, the selected matching speed, and the selected occupied memory respectively; by optimizing the index items, feature extraction is performed on the benchmark processing method to obtain the blurring speed, matching speed, and occupied memory corresponding to the benchmark processing method, and they are marked as the benchmark blurring speed, the benchmark matching speed, and the benchmark occupied memory;

[0071] According to the formula Calculate the optimal value of the selected blurring method;

[0072] Where: PK is the optimized value; β1, β2, and β3 are the weight coefficients of the optimization index items corresponding to blur speed, matching speed, and memory usage respectively; HV1, PV1, and ZC1 are the candidate blur speed, candidate matching speed, and candidate memory usage respectively; HV2, PV2, and ZC2 are the benchmark blur speed, benchmark matching speed, and benchmark memory usage respectively;

[0073] Determine whether the blurring effect of the candidate blurring method is better than the baseline processing method based on the optimization value.

[0074] The main database is used to store and manage the received virtualized data. Specifically, the storage management is performed according to existing storage management methods, such as classification management and expiration deletion. The main database can use a cloud database.

[0075] The application end is used by corresponding users authorized by the subject party, and users with permissions are marked as application users; it includes a permission module, an application database, and an application analysis module;

[0076] The permission module is used to verify the permissions of the logged-in user. When the permission verification is successful, the logged-in user is marked as an application user and the usage permission is opened; when the permission verification fails, the use of the application end is prohibited.

[0077] Among them, the authority verification of the logged-in user can be based on existing verification methods, such as activation code, subject-assisted verification and other methods.

[0078] The application database is used to connect to the main database and store public data corresponding to application users. It includes a matching unit and a storage unit. The matching unit is used to obtain public data and store the obtained public data in the storage unit; the storage unit is used to store public data. The application database can be stored in the cloud.

[0079] The method used by the matching unit to obtain public data includes:

[0080] Identify application user information, mainly focusing on the relevant data of which scenario it belongs to, determine the corresponding scenario based on the application user information, mark it as the user scenario, match the corresponding virtualized data from the main database based on the user scenario, and obtain the corresponding public data based on the virtualized data.

[0081] The application analysis module is used to analyze public data, obtain corresponding analysis results, and display the obtained analysis results to application users.

[0082] In one embodiment, the method of analyzing public data is: the platform performs analysis based on existing data analysis methods according to data analysis needs, such as establishing an intelligent analysis model based on neural networks, and analyzing the data analysis needs of application users through the successfully trained intelligent analysis model to obtain corresponding analysis results.

[0083] In one embodiment, a method for analyzing public data includes:

[0084] The platform presets various data analysis requirements corresponding to different potential scenarios, marks them as alternative requirements, and sets corresponding data analysis methods for each alternative requirement; sets a requirement column based on each alternative requirement, which includes function keys corresponding to each alternative requirement and function keys corresponding to additional requirements. Additional requirements refer to data analysis requirements among non-alternative requirements, and are associated with corresponding data analysis methods. Clicking the function key corresponding to an alternative requirement allows analysis to be performed directly based on the corresponding data analysis method;

[0085] When the user has an alternative requirement, he clicks the function key corresponding to the alternative requirement, analyzes the public data in the application database according to the data analysis method corresponding to the alternative requirement, and obtains the corresponding analysis result;

[0086] When the user has additional requirements, he clicks the function key corresponding to the additional requirements, enters the corresponding data analysis requirements, analyzes the public data in the application database according to the data analysis requirements, and obtains the corresponding analysis results.

[0087] In one embodiment, the determination of alternative requirements can be based on existing methods, such as settings by the platform, or statistics on various existing data analysis requirements, and selecting data analysis requirements with a corresponding proportion as alternative requirements. The necessity of each data analysis requirement can also be analyzed, and the alternative requirements can be determined based on the necessity analysis.

[0088] In one embodiment, the data analysis method can be determined based on the existing data analysis method because the alternative requirement is an existing data analysis requirement, or it can be set by the platform itself.

[0089] In one embodiment, the public data in the application database is analyzed according to the data analysis requirement, and the analysis can be performed based on the method in the previous embodiment, that is, based on the existing method.

[0090] In one embodiment, this embodiment differs from the previous embodiment in that no additional requirements are included.

[0091] The above formulas are all calculated by removing dimensions and taking their numerical values. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and preset thresholds in the formula are set by technicians in this field according to actual conditions or obtained by simulating a large amount of data.

[0092] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An artificial intelligence-based public data intelligent processing and analysis system, characterized in that: Including data terminal and application terminal; The data terminal includes a demand analysis module, a data acquisition module, and a main database; The demand analysis module is used to determine the scenario range according to the management requirements of the subject party, the scenario range is composed of several potential scenarios, set the public data range corresponding to the potential scenario, and set the data processing method corresponding to the potential scenario according to the public data range; The data acquisition module is used to perform real-time data acquisition based on the public data range and data processing method of the potential scene, obtain the public data of the potential scene, perform virtualization processing on the public data, and obtain virtualized data corresponding to the potential scene; and tag the virtualized data with the corresponding potential scene tag and send it to the main database; The main body database is used to store and manage the received virtualized data; The application end includes a permission module, an application database and an application analysis module; The permission module is used to verify the permissions of the logged-in user. When the permission verification is successful, the logged-in user is marked as an application user and the usage permission is activated; When permission verification fails, the application is prohibited from being used; The application database is used to connect to the main database and store public data corresponding to application users, including a matching unit and a storage unit. The matching unit is used to obtain public data from the main database and store the obtained public data in the storage unit; the storage unit is used to store public data; The application analysis module is used to analyze public data, obtain corresponding analysis results, and display the obtained analysis results to application users; The method for virtualizing public data is: Step SA1: The platform presets a blur processing method and blur processing requirements for the potential scene, and marks the blur processing method preset by the platform as a reference processing method; Step SA2: performing blurring processing on the public data of the potential scene using the reference processing method to obtain blurring data; Step SA3: obtaining a candidate blurring method in real time according to the blurring processing requirement, comparing the candidate blurring method with a reference processing method, and determining whether the blurring effect of the candidate blurring method is better than that of the reference processing method; When it is determined that the blurring effect of the candidate blurring method is better than that of the reference processing method, the candidate blurring method replaces the reference processing method in step SA2 to form a new reference processing method; When it is determined that the blurring effect of the selected blurring method is not better than that of the reference processing method, no corresponding processing is performed; Step SA4: loop step SA3; Methods for comparing candidate blurring methods with baseline processing methods include: Set optimization indicators and their corresponding weight coefficients. The optimization indicators include blur speed, matching speed, and memory usage. Extracting features of the selected blurring method using the optimization index item to obtain a selected blurring speed, a selected matching speed, and a selected occupied memory corresponding to the selected blurring method; extracting features of the benchmark processing method using the optimization index item to obtain a benchmark blurring speed, a benchmark matching speed, and a benchmark occupied memory corresponding to the benchmark processing method; According to the formula Calculate the optimal value of the selected blurring method; Where: PK is the optimized value; β1, β2, and β3 are the weight coefficients of the corresponding optimization index items; HV1, PV1, and ZC1 are the candidate blur speed, candidate matching speed, and candidate occupied memory respectively; HV2, PV2, and ZC2 are the benchmark blur speed, benchmark matching speed, and benchmark occupied memory respectively; Determine whether the blurring effect of the candidate blurring method is better than the baseline processing method based on the optimization value.

2. The public data intelligent processing and analysis system based on artificial intelligence according to claim 1 is characterized in that: Methods for setting the common data range for potential scenarios include: Reference data ranges of several potential scenarios are obtained, and the reference data ranges corresponding to the potential scenarios are merged to form a common data range of the potential scenarios.

3. The public data intelligent processing and analysis system based on artificial intelligence according to claim 2 is characterized in that: Before merging the reference data ranges of the potential scenarios, the reference data ranges are corrected, and then merged after the reference data ranges are corrected.

4. The public data intelligent processing and analysis system based on artificial intelligence according to claim 2 or 3, characterized in that: The public data range of the potential scenario is adjusted in real time, and the adjustment method includes: obtaining the range adjustment requirement of the potential scenario in real time, and adjusting the public data range according to the range adjustment requirement.

5. The public data intelligent processing and analysis system based on artificial intelligence according to claim 1 is characterized in that: The method used by the matching unit to obtain public data includes: Acquire application user information of an application user, determine a user scenario corresponding to the application user based on the application user information, match corresponding virtualized data from the subject database based on the user scenario, and acquire public data corresponding to the application user based on the virtualized data.

6. The public data intelligent processing and analysis system based on artificial intelligence according to claim 1 is characterized in that: Methods for analyzing public data include: The platform presets several alternative requirements and sets corresponding data analysis methods for the alternative requirements; sets a requirement column based on the alternative requirements and the data analysis methods, the requirement column includes function keys corresponding to the alternative requirements and function keys corresponding to additional requirements, the additional requirements being data analysis requirements corresponding to non-alternative requirements; When the user has an alternative requirement, the user clicks the function key corresponding to the alternative requirement, and the public data in the application database is analyzed according to the data analysis method corresponding to the alternative requirement to obtain the analysis result; When the user has additional requirements, he clicks the function key corresponding to the additional requirements, enters the corresponding data analysis requirements, analyzes the public data in the application database according to the data analysis requirements, and obtains the corresponding analysis results.

Images