Public data intelligent processing and analysis system based on artificial intelligence

Abstract

The invention discloses an intelligent public data processing and analyzing system based on artificial intelligence, and belongs to the technical field of public data processing. The data total end comprises a demand analysis module, a data acquisition module and a main database; the demand analysis module is used for determining a scene range and a public data range corresponding to the potential scene according to the management demand of the main party, and setting a data processing mode according to the public data range; the data acquisition module is used for performing real-time data acquisition according to the public data range and the data processing mode to obtain public data, and performing blurring processing on the public data to obtain blurred data; the main body database is used for storing and managing the blurred data; the application end comprises a permission module, an application database and an application analysis module; the permission module is used for performing permission verification on the login user; the application database is used for storing public data of application users; and the demand analysis module is used for analyzing the public data to obtain an analysis result.

Description

Technical Field

[0001] The present invention belongs to the technical field of public data processing, and specifically is a public data intelligent processing and analysis system based on artificial intelligence. Background Art

[0002] In the current era of rapid development of informatization, the processing and application of public data has become an important force in promoting government decision-making, social governance, economic development and other aspects. However, for a long time, each department has been acting independently and setting up its own public data processing system. Although this practice has met the data processing needs of each department to a certain extent, its disadvantages have gradually emerged over time. First, the data processing systems of various departments lack unified standards and specifications. Due to the differences in the needs, technologies and resources of various departments in data processing, the established systems often use different data structures, algorithms and interfaces. This difference not only leads to incompatibility between data, but also increases the difficulty of data sharing and exchange. When cross-departmental data integration and analysis is required, it often takes a lot of time and resources to convert and clean data formats, which seriously affects the efficiency and accuracy of data processing. This lack of unified standards not only hinders the circulation and application of data, but also limits the maximization of data value. Secondly, the existence of multiple data processing systems is prone to waste of resources. Each system requires independent hardware and software support, which not only increases costs, but also occupies a lot of computing and storage resources. At the same time, due to the differences in data processing needs between departments, many systems are often idle or semi-idle, resulting in serious waste of resources. This waste of resources not only increases the government's financial burden, but also limits the investment and development of other departments in data processing.

[0003] Based on this, the present invention provides a public data intelligent processing and analysis system based on artificial intelligence. Summary of the invention

[0004] In order to solve the problems existing in the above solutions, the present invention provides a public data intelligent processing and analysis system based on artificial intelligence.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] An artificial intelligence-based public data intelligent processing and analysis system, including a data main end and an application end;

[0007] The data terminal includes a demand analysis module, a data collection module, and a subject database;

[0008] The demand analysis module is used to determine the scenario scope according to the management needs of the subject party, the scenario scope is composed of several potential scenarios, set the public data scope corresponding to the potential scenario, and set the data processing method corresponding to the potential scenario according to the public data scope.

[0009] Further, the method of setting the public data range of the potential scene includes:

[0010] Reference data ranges of several potential scenarios are obtained, and the reference data ranges corresponding to the potential scenarios are merged to form a common data range of the potential scenarios.

[0011] Furthermore, before the reference data ranges of the potential scenarios are merged, the reference data ranges are corrected, and then merged after the reference data ranges are corrected.

[0012] Furthermore, the public data range of the potential scenario is adjusted in real time, and the adjustment method includes: acquiring the range adjustment requirement of the potential scenario in real time, and adjusting the public data range according to the range adjustment requirement.

[0013] The data acquisition module is used to perform real-time data acquisition according to the public data range and data processing method of the potential scene, obtain the public data of the potential scene, virtualize the public data, and obtain the virtualized data corresponding to the potential scene; and tag the virtualized data with the corresponding potential scene label and send it to the main database.

[0014] Furthermore, the method for virtualizing public data is as follows:

[0015] Step SA1: The platform party presets the blur processing method and blur processing requirements of the potential scene, and marks the blur processing method preset by the platform party as a reference processing method;

[0016] Step SA2: performing blur processing on the public data of the potential scene by using the reference processing method to obtain blur data;

[0017] Step SA3: acquiring a candidate blurring method in real time according to the blurring processing requirement, comparing the candidate blurring method with a reference processing method, and determining whether the blurring effect of the candidate blurring method is better than that of the reference processing method;

[0018] When it is determined that the blurring effect of the candidate blurring method is better than that of the reference processing method, the candidate blurring method replaces the reference processing method in step SA2 to form a new reference processing method;

[0019] When it is determined that the blurring effect of the selected blurring method is not better than the reference processing method, no corresponding processing is performed;

[0020] Step SA4: Loop step SA3.

[0021] Furthermore, the method of comparing the selected blurring method with the reference processing method includes:

[0022] Set optimization index items and their corresponding weight coefficients. The optimization index items include blur speed, matching speed and memory usage.

[0023] Extracting features of the selected blurring method by using the optimization index item, obtaining a selected blurring speed, a selected matching speed, and a selected occupied memory corresponding to the selected blurring method; extracting features of the benchmark processing method by using the optimization index item, obtaining a benchmark blurring speed, a benchmark matching speed, and a benchmark occupied memory corresponding to the benchmark processing method;

[0024] According to the formula Calculate the optimal value of the blurring method to be selected;

[0025] Where: PK is the optimization value; β1, β2, β3 are the weight coefficients of the corresponding optimization index items; HV1, PV1, ZC1 are the selected blur speed, the selected matching speed and the selected occupied memory respectively; HV2, PV2, ZC2 are the benchmark blur speed, the benchmark matching speed and the benchmark occupied memory respectively;

[0026] It is determined according to the optimization value that the blurring effect of the candidate blurring method is better than that of the reference processing method.

[0027] The main body database is used to store and manage the received virtualized data.

[0028] The application end includes a permission module, an application database and an application analysis module;

[0029] The permission module is used to perform permission verification on the logged-in user. When the permission verification succeeds, the logged-in user is marked as an application user and the usage permission is enabled; when the permission verification fails, the use of the application end is prohibited.

[0030] The application database is used to connect to the main database and store public data corresponding to application users, including a matching unit and a storage unit. The matching unit is used to obtain public data and store the obtained public data in the storage unit; the storage unit is used to store public data.

[0031] Furthermore, the method for the matching unit to obtain the public data includes:

[0032] Acquire application user information of an application user, determine a user scenario corresponding to the application user according to the application user information, match corresponding virtualized data from the subject database according to the user scenario, and acquire public data corresponding to the application user according to the virtualized data.

[0033] The demand analysis module is used to analyze public data, obtain corresponding analysis results, and display the obtained analysis results to application users.

[0034] Furthermore, the method of analyzing public data includes:

[0035] The platform party presets several alternative requirements and sets corresponding data analysis methods for the alternative requirements; sets a requirement column according to the alternative requirements and the data analysis methods, wherein the requirement column includes function keys corresponding to the alternative requirements and function keys corresponding to additional requirements, wherein the additional requirements are data analysis requirements corresponding to non-alternative requirements;

[0036] When the user has an alternative requirement, the user clicks the function key corresponding to the alternative requirement, and the public data in the application database is analyzed according to the data analysis method corresponding to the alternative requirement to obtain the analysis result;

[0037] When the user has additional requirements, the user clicks the function key corresponding to the additional requirements, enters the corresponding data analysis requirements, analyzes the public data in the application database according to the data analysis requirements, and obtains the corresponding analysis results.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] By constructing a unified public data integration and processing platform, the present invention realizes standardized data processing. The platform adopts a unified data structure, algorithm and interface to ensure the compatibility and sharing of data among various application users. This greatly simplifies the process of data format conversion and cleaning, and improves the efficiency and accuracy of data processing. At the same time, unified standards also help promote the widespread application of data and the mining of its value; by integrating the public data processing systems of various application users, the present invention realizes the sharing of hardware and software, avoiding duplication of construction and waste of resources. At the same time, the platform adopts advanced technical means, such as linking, to represent the remaining public data after processing of each scene to reduce overall memory usage. This approach of optimizing resource allocation not only reduces costs, but also improves the efficiency of resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, a public data intelligent processing and analysis system based on artificial intelligence includes a data terminal and an application terminal;

[0044] The data terminal is set up according to the requirements of the unified manager and maintained and updated by the platform. For example, if the competent department of a certain region applies to establish a data terminal, the subordinate departments of the competent department can subsequently use the user terminal as the user subject. The data terminal can also be established by a combination of multiple departments. It is suitable for a variety of situations. The unified management value is marked as the subject. The data terminal includes a demand analysis module, a data collection module, and a subject database.

[0045] The demand analysis module is used to determine the scenario scope according to the management needs of the subject party. The scenario scope is composed of various potential scenarios, and the public data scope corresponding to each potential scenario is set. The data processing method corresponding to the corresponding potential scenario is set according to the public data scope.

[0046] Among them, the management needs of the subject party refer to the public data of which fields and scenarios the subject party wants to process and analyze, such as directly marking the corresponding scenarios of each department, such as high school education, junior high school education, elementary education, etc.; because the scope of the scenario can be directly determined according to management needs, it is generally reported directly to the platform by the subject party.

[0047] In one embodiment, the public data range of a potential scenario is set by determining what public data the potential scenario will need based on existing methods, thereby forming the public data range.

[0048] In one embodiment, a method for setting a common data range for a potential scenario includes:

[0049] Obtain each reference data range of the potential scenario, that is, the existing various required public data ranges of the potential scenario, marked as reference data ranges, such as high school education, its parameter data range is the public data range corresponding to the public data processing and analysis system used by various high schools, as the reference data range;

[0050] Merge each reference data range to form a public data range for the potential scenario, ensuring the comprehensiveness of data acquired by users of subsequent corresponding scenarios.

[0051] In one embodiment, due to regional differences in actual situations, some reference data ranges will not be needed in the potential scenarios corresponding to subsequent subjects, resulting in continuous processing and analysis of public data that is not needed subsequently, causing waste of resources. Therefore, in this embodiment, before merging the reference data ranges, the reference data ranges of the potential scenarios are corrected to remove the reference data ranges that will not be applied.

[0052] In one embodiment, the reference data range may be corrected based on an existing range correction method.

[0053] In one embodiment, the method for correcting the reference data range includes:

[0054] Obtain information about the subject, such as the region, management system, cases, and other relevant data that will affect the determination of the data scope for subsequent potential scenarios;

[0055] A range correction model is established based on a CNN network or a DNN network, and a corresponding training set is manually established for training. The training set includes input data and output data. The input data is the signature scenario, the subject party information and the reference data range. The output data is the corrected reference data range. The corrected reference data range is obtained by analyzing the range correction model after successful training.

[0056] In one embodiment, as it is used in an actual context, the corresponding user may have adjustment requirements for the predetermined public data range, such as increasing the range or decreasing the range, so the public data range needs to be adjusted, and the range adjustment requirements corresponding to the potential scene are obtained in real time, and the public data range of the potential scene is adjusted according to the range adjustment requirements.

[0057] In one embodiment, a data processing method corresponding to a corresponding potential scenario is set according to the public data range, and is set by the platform according to an existing method. The data processing method is to obtain public data within the public data range from various channels in real time, and then perform pre-processing, screening and other preset measures to process the data.

[0058] The data acquisition module is used to perform real-time data acquisition according to the public data range and data processing method of each potential scene, obtain the public data corresponding to each potential scene, perform virtualization processing on each public data, obtain the virtualized data corresponding to each potential scene, and tag the virtualized data with the corresponding potential scene label and send it to the main database.

[0059] Among them, virtualizing public data means not directly collecting the public data by extracting its collection link, etc., because the memory occupancy of virtualized data such as links is much smaller than the memory occupancy of public data. Therefore, the public data is virtualized by presetting the corresponding virtualization method, which can realize the subsequent rapid collection of the corresponding public data based on the virtualized data; it can be set by the platform.

[0060] In one embodiment, the method for virtualizing public data is:

[0061] Step SA1: The platform may preset a blur processing method and blur processing requirements for each potential scene, and mark the blur processing method as a reference processing method;

[0062] Step SA2: blurring the public data of the potential scene by using a reference processing method;

[0063] Step SA3: according to the blur processing requirement, an existing blur processing method that can achieve the blur requirement is obtained in real time, and marked as a to-be-selected blur processing method, and the to-be-selected blur processing method is compared with the reference processing method to determine whether the blur effect of the to-be-selected blur processing method is better than that of the reference processing method;

[0064] When it is determined that the blurring effect of the candidate blurring method is better than the reference processing method, the candidate blurring method replaces the reference processing method to form a new reference processing method, and the new reference processing method is updated and replaced in the supplementary SA2;

[0065] When it is judged that the blurring effect of the selected blurring method is not better than that of the reference processing method, no corresponding processing is performed;

[0066] Step SA4: Loop step SA3.

[0067] In one embodiment, the candidate blurring method is compared with the reference processing method, and the comparison can be based on an existing blurring effect evaluation method.

[0068] In one embodiment, the method of comparing the candidate blurring method with the reference processing method includes:

[0069] Setting optimization index items and weight coefficients corresponding to the optimization index items, the optimization index items include blurring speed, matching speed and occupied memory; blurring speed refers to the speed of converting preset public data into corresponding blurred data, matching speed refers to the speed of subsequently matching blurred data according to preset public data to corresponding public data; occupied memory refers to the occupied memory of blurred data formed according to the blurring processing method; that is, it is determined by standard public data; the weight coefficient is set by the platform according to the needs of the subject party, or by the platform according to management needs;

[0070] By optimizing the index items, feature extraction is performed on the selected blurring method to obtain blurring speed, matching speed and occupied memory corresponding to the selected blurring method, and in order to distinguish them, they are marked as the selected blurring speed, the selected matching speed and the selected occupied memory respectively; by optimizing the index items, feature extraction is performed on the benchmark processing method to obtain blurring speed, matching speed and occupied memory corresponding to the benchmark processing method, and they are marked as benchmark blurring speed, benchmark matching speed and benchmark occupied memory;

[0071] According to the formula Calculate the optimal value of the blurring method to be selected;

[0072] Where: PK is the optimization value; β1, β2, β3 are the weight coefficients of the optimization index items corresponding to blur speed, matching speed and memory occupation respectively; HV1, PV1, ZC1 are the selected blur speed, the selected matching speed and the selected memory occupation respectively; HV2, PV2, ZC2 are the benchmark blur speed, the benchmark matching speed and the benchmark memory occupation respectively;

[0073] It is determined according to the optimization value that the blurring effect of the candidate blurring method is better than that of the reference processing method.

[0074] The main body database is used to store and manage the received virtualized data, specifically according to the existing storage management method, such as classification management, expiration deletion, etc. The main body database can use a cloud database.

[0075] The application end is used by corresponding users authorized by the subject party, and the authorized users are marked as application users; it includes a permission module, an application database, and a demand analysis module;

[0076] The permission module is used to perform permission verification on the logged-in user. When the permission verification succeeds, the logged-in user is marked as an application user and the usage permission is enabled; when the permission verification fails, the use of the application end is prohibited.

[0077] Among them, the authority verification of the logged-in user can be based on existing verification methods, such as activation code, subject-assisted verification and other methods.

[0078] The application database is used to connect to the main database and store the public data corresponding to the application users, including a matching unit and a storage unit. The matching unit is used to obtain the public data and store the obtained public data in the storage unit; the storage unit is used to store the public data. The application database can be stored in the cloud.

[0079] The method used by the matching unit to obtain public data includes:

[0080] Identify application user information, mainly for the relevant data of the scene it belongs to, determine the corresponding scene according to the application user information, mark it as the user scene, match the corresponding virtualized data from the main database according to the user scene, and obtain the corresponding public data based on the virtualized data.

[0081] The demand analysis module is used to analyze public data, obtain corresponding analysis results, and display the obtained analysis results to application users.

[0082] In one embodiment, the method of analyzing public data is as follows: the platform performs analysis based on existing data analysis methods according to data analysis needs, such as establishing an intelligent analysis model based on neural networks, and analyzing the data analysis needs of application users through the successfully trained intelligent analysis model to obtain corresponding analysis results.

[0083] In one embodiment, a method for analyzing public data includes:

[0084] The platform presets various data analysis requirements corresponding to different potential scenarios, marks them as alternative requirements, and sets corresponding data analysis methods for each alternative requirement; sets a requirement column according to each alternative requirement, the requirement column includes function keys corresponding to each alternative requirement and function keys corresponding to additional requirements, and additional requirements refer to data analysis requirements among non-alternative requirements, and associates corresponding data analysis methods, so that analysis can be directly performed based on the corresponding data analysis method by clicking the function key corresponding to the alternative requirement;

[0085] When the user has an alternative requirement, the user clicks the function key corresponding to the corresponding alternative requirement, and analyzes the public data in the application database according to the data analysis method corresponding to the alternative requirement to obtain the corresponding analysis result;

[0086] When the user has additional requirements, the user clicks the function key corresponding to the additional requirements, enters the corresponding data analysis requirements, analyzes the public data in the application database according to the data analysis requirements, and obtains the corresponding analysis results.

[0087] In one embodiment, alternative requirements may be determined based on existing methods, such as settings by the platform, or statistics on various existing data analysis requirements may be collected to select data analysis requirements with a corresponding proportion as alternative requirements. Alternatively, the necessity of each data analysis requirement may be analyzed to determine alternative requirements based on the necessity analysis.

[0088] In one embodiment, the data analysis method can be determined according to the existing data analysis method, or it can be set by the platform itself because the alternative demand is an existing data analysis demand.

[0089] In one embodiment, the public data in the application database is analyzed according to the data analysis requirement, and the analysis can be performed based on the method in the previous embodiment, that is, based on the existing method.

[0090] In one embodiment, this embodiment differs from the previous embodiment in that no additional requirements are included.

[0091] The above formulas are all calculated by removing dimensions and taking their numerical values. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained by simulating a large amount of data.

[0092] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A public data intelligent processing and analysis system based on artificial intelligence, characterized in that: Including data terminal and application terminal; The data terminal includes a demand analysis module, a data collection module, and a subject database; The demand analysis module is used to determine the scenario scope according to the management requirements of the subject party, the scenario scope is composed of several potential scenarios, set the public data scope corresponding to the potential scenario, and set the data processing method corresponding to the potential scenario according to the public data scope; The data acquisition module is used to perform real-time data acquisition according to the public data range and data processing method of the potential scene, obtain the public data of the potential scene, perform virtualization processing on the public data, and obtain the virtualized data corresponding to the potential scene; and send the virtualized data to the main database after tagging it with the corresponding potential scene tag; The subject database is used to store and manage the received virtualized data; The application end includes a permission module, an application database and an application analysis module; The permission module is used to verify the permissions of the logged-in user. When the permission verification is successful, the logged-in user is marked as an application user and the usage permission is opened; When permission verification fails, the application is prohibited from being used; The application database is used to connect to the subject database and store public data corresponding to application users, including a matching unit and a storage unit. The matching unit is used to obtain public data and store the obtained public data in the storage unit; the storage unit is used to store public data; The demand analysis module is used to analyze public data, obtain corresponding analysis results, and display the obtained analysis results to application users.

2. According to claim 1, a public data intelligent processing and analysis system based on artificial intelligence is characterized in that: Methods for setting the common data range for potential scenarios include: Reference data ranges of several potential scenarios are obtained, and the reference data ranges corresponding to the potential scenarios are merged to form a common data range of the potential scenarios.

3. The public data intelligent processing and analysis system based on artificial intelligence according to claim 2 is characterized in that: Before merging the reference data ranges of the potential scenes, the reference data ranges are corrected, and then merged after the reference data ranges are corrected.

4. The public data intelligent processing and analysis system based on artificial intelligence according to claim 2 or 3 is characterized in that: The public data range of the potential scenario is adjusted in real time, and the adjustment method includes: obtaining the range adjustment requirement of the potential scenario in real time, and adjusting the public data range according to the range adjustment requirement.

5. The public data intelligent processing and analysis system based on artificial intelligence according to claim 1 is characterized in that: The method of virtualizing public data is as follows: Step SA1: The platform party presets the blur processing method and blur processing requirements of the potential scene, and marks the blur processing method preset by the platform party as a reference processing method; Step SA2: performing blur processing on the public data of the potential scene by using the reference processing method to obtain blur data; Step SA3: acquiring a candidate blurring method in real time according to the blurring processing requirement, comparing the candidate blurring method with a reference processing method, and determining whether the blurring effect of the candidate blurring method is better than that of the reference processing method; When it is determined that the blurring effect of the candidate blurring method is better than that of the reference processing method, the candidate blurring method replaces the reference processing method in step SA2 to form a new reference processing method; When it is determined that the blurring effect of the selected blurring method is not better than the reference processing method, no corresponding processing is performed; Step SA4: Loop step SA3.

6. The public data intelligent processing and analysis system based on artificial intelligence according to claim 5 is characterized in that: Methods for comparing a candidate blurring method with a baseline processing method include: Set optimization index items and their corresponding weight coefficients. The optimization index items include blur speed, matching speed and memory usage. Extracting features of the selected blurring method by using the optimization index item, obtaining a selected blurring speed, a selected matching speed, and a selected occupied memory corresponding to the selected blurring method; extracting features of the benchmark processing method by using the optimization index item, obtaining a benchmark blurring speed, a benchmark matching speed, and a benchmark occupied memory corresponding to the benchmark processing method; According to the formula Calculate the optimal value of the blurring method to be selected; Where: PK is the optimization value; β1, β2, β3 are the weight coefficients of the corresponding optimization index items; HV1, PV1, ZC1 are the selected blur speed, the selected matching speed and the selected occupied memory respectively; HV2, PV2, ZC2 are the benchmark blur speed, the benchmark matching speed and the benchmark occupied memory respectively; It is determined according to the optimization value that the blurring effect of the candidate blurring method is better than that of the reference processing method.

7. The public data intelligent processing and analysis system based on artificial intelligence according to claim 1 is characterized in that: The method used by the matching unit to obtain public data includes: Acquire application user information of an application user, determine a user scenario corresponding to the application user according to the application user information, match corresponding virtualized data from the subject database according to the user scenario, and acquire public data corresponding to the application user according to the virtualized data.

8. The public data intelligent processing and analysis system based on artificial intelligence according to claim 1 is characterized in that: Methods for analyzing public data include: The platform party presets several alternative requirements and sets corresponding data analysis methods for the alternative requirements; sets a requirement column according to the alternative requirements and the data analysis methods, wherein the requirement column includes function keys corresponding to the alternative requirements and function keys corresponding to additional requirements, wherein the additional requirements are data analysis requirements corresponding to non-alternative requirements; When the user has an alternative requirement, the user clicks the function key corresponding to the alternative requirement, and the public data in the application database is analyzed according to the data analysis method corresponding to the alternative requirement to obtain the analysis result; When the user has additional requirements, the user clicks the function key corresponding to the additional requirements, enters the corresponding data analysis requirements, analyzes the public data in the application database according to the data analysis requirements, and obtains the corresponding analysis results.

Images