Data processing method and device based on AI model

Abstract

The invention discloses a data processing method and device based on an AI model. Identifying data from at least one data source, and determining sample data from the identified data; performing data analysis on the sample data through an AI model to obtain a data analysis result; a classification identifier is added to the identified data through an AI model; generating a data processing rule based on an analysis result of the data analysis, writing a database design script, and synchronizing the data processing rule and the database design script to a data processing platform; and accessing the data source to a data processing platform, preprocessing the collected data by using a data processing rule through the data processing platform, generating a database table by using a database design script, importing the preprocessed data into the database table, and storing the preprocessed data based on the classification identifier. According to the invention, data acquisition and rapid analysis are carried out through AI, rapid storage of various data in the park is realized, and the problems of low data processing efficiency, low data utilization rate and insufficient data security in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method and device based on an AI model. Background Art

[0002] At present, all kinds of parks and departments within the parks are actively carrying out information construction, and information work has also laid a certain foundation. However, due to the lack of unified planning, some vertical information networks are self-contained, business systems are closed and cannot be interconnected, forming many "information islands" and making it difficult to share information resources. Due to the existence of information islands, many parks have neglected the development and utilization of information resources, which is mainly manifested in the relatively scattered data resources and the low degree of information integration and sharing. Although the park network and business system are developing rapidly, the development and utilization of information resources have not been raised to the level they should be. There is a lack of park information resource exchange platforms and centers, and there are generally problems such as small scale, narrow scope, poor quality, long update cycle, and low degree of sharing of information resources.

[0003] At present, traditional park data processing methods and devices mostly use data processing platforms with manual analysis. In the data preprocessing stage, data analysis engineers collect data samples, perform data analysis, customize data rules, etc. However, the data in the park are messy and diverse, and each park has its own unique data. Manual analysis often consumes a lot of time and manpower costs. At the same time, there is also the possibility of inaccurate data analysis.

[0004] Based on this, a method is needed to improve the efficiency of park data processing. Summary of the invention

[0005] The present invention provides a data processing method and device based on an AI model to improve the efficiency of park data processing.

[0006] According to one aspect of the present invention, there is provided a data processing method based on an AI model, comprising:

[0007] Identify data from at least one data source, and determine sample data from the identified data; perform data analysis on the sample data using an AI model to obtain a data analysis result;

[0008] Add classification identification to the identified data through the AI ​​model;

[0009] Generate data processing rules and write database design scripts based on the analysis results of the data analysis, and synchronize the data processing rules and the database design scripts to the data processing platform;

[0010] The data source is connected to the data processing platform, the collected data is preprocessed by the data processing platform using the data processing rules and a database table is generated using the database design script, the preprocessed data is imported into the database table and stored based on the classification identifier.

[0011] Optionally, the data analysis includes: data field attribute analysis, data field analysis, data null value rate analysis, data repetition rate analysis, data content analysis and data scenario analysis; accordingly, the data analysis of the sample data by the AI ​​model to obtain the data analysis result includes:

[0012] Performing field attribute analysis on the sample data through the AI ​​model to identify integers, floating point numbers, strings, and dates in each field;

[0013] Perform data field analysis on the sample data through the AI ​​model to perform statistical description, data distribution, and outlier detection;

[0014] Performing data vacancy rate analysis on the sample data through the AI ​​model to determine the impact of the vacancy value;

[0015] Performing data repetition rate analysis on the sample data through the AI ​​model, checking duplicate records and evaluating the impact of duplicate data;

[0016] Performing data content analysis on the sample data through the AI ​​model to identify sensitive information and perform semantic analysis;

[0017] The AI ​​model is used to perform data scenario analysis on the sample data to determine the relationship between the data and the data business scenario.

[0018] Optionally, adding a classification mark to the identified data by using the AI ​​model includes:

[0019] Performing field attribute analysis on the identified data, identifying the data type of the data and adding corresponding classification identifiers;

[0020] Analyze the distribution and outliers of the identified data and add corresponding classification labels;

[0021] According to the role of the identified data in the business process, the identified data is divided into different business scenario categories and corresponding classification identifiers are added;

[0022] Add corresponding classification identifiers based on the importance and urgency of the identified data to the business.

[0023] Optionally, the generating of data processing rules based on the analysis results of the data analysis includes:

[0024] Determine an outlier judgment rule based on the analysis result of the data field attribute analysis;

[0025] Determining a format conversion rule according to the analysis result of the data field analysis;

[0026] Determine a filling rule and / or a deletion rule for null value data according to the analysis result of the data null value rate analysis;

[0027] Determining a deletion rule and / or a marking rule for duplicate data according to the analysis result of the data duplication rate analysis;

[0028] Determining a data value range rule according to the analysis result of the data content analysis;

[0029] The data rationality verification rules are determined according to the analysis results of the data scenario analysis.

[0030] Optionally, the writing of the database design script includes:

[0031] Determine the table name and field according to the data analysis results, and set the primary key and foreign key;

[0032] Determine the index field based on the data query and analysis requirements; select the index type based on the field data characteristics and business requirements.

[0033] Optionally, connecting the data source to the data processing platform includes:

[0034] The configuration parameters for data source access are configured on the data processing platform. If the data source is an internal relational database of an enterprise, the configuration parameters include the address, port number, database name, user name and password of the database server; if the data source is an IoT device, the configuration parameters include the IP address, communication port, data transmission protocol and device authentication information.

[0035] A connection test is used to verify the connection status of the data source on the data processing platform.

[0036] Optionally, importing the preprocessed data into the database table includes:

[0037] Inserting or updating the preprocessed data according to the structure and primary key constraints of the database table;

[0038] If the primary key value of the data does not exist in the database table, the data will be inserted into the corresponding table as a new record;

[0039] If the primary key value already exists, determine whether to update some or all fields of the original record based on specific business needs and preset rules.

[0040] According to another aspect of the present invention, there is provided a data processing device based on an AI model, comprising:

[0041] A data analysis unit, configured to identify data from at least one data source, determine sample data from the identified data, and perform data analysis on the sample data using an AI model to obtain a data analysis result;

[0042] An identification adding unit, used to add a classification identification to the identified data through the AI ​​model;

[0043] An analysis result application unit, used to generate data processing rules and write database design scripts based on the analysis results of the data analysis, and synchronize the data processing rules and the database design scripts to the data processing platform;

[0044] The data storage unit is used to connect the data source to the data processing platform, pre-process the collected data using the data processing rules through the data processing platform and generate a database table using the database design script, import the pre-processed data into the database table and store it based on the classification identifier.

[0045] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0046] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data processing method based on the AI ​​model described in any embodiment of the present invention.

[0047] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method based on the AI ​​model described in any embodiment of the present invention when executed.

[0048] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program, and the computer program, when executed by a processor, implements the data processing method based on the AI ​​model described in any embodiment of the present invention.

[0049] The technical solution of the embodiment of the present invention uses AI to collect data, conduct rapid analysis, generate data specifications, and efficiently complete data preprocessing, so as to realize the rapid storage of various types of data in the park, and conduct in-depth analysis of data integration according to the park business. The present invention solves the problems of low data processing efficiency, low data utilization, and insufficient data security in the prior art. By adopting the high-performance data processing capabilities of the AI ​​large model, the present invention not only improves the efficiency and security of data processing, but also improves the level of park management, and has important practical value and broad market prospects.

[0050] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 is a flow chart of a data processing method based on an AI model provided in Embodiment 1 of the present invention;

[0053] Figure 2 This is a process framework diagram of data processing based on an AI model applicable to the first embodiment of the present invention;

[0054] Figure 3 is a structural diagram of a data processing device based on an AI model provided in Embodiment 2 of the present invention;

[0055] Figure 4 It is a structural schematic diagram of an electronic device that implements the data processing method based on the AI ​​model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0058] Embodiment 1

[0059] Figure 1 This is a flow chart of a data processing method based on an AI model provided in the first embodiment of the present invention. This embodiment is applicable to the case of automatically processing data in a park. The method can be executed by a data processing device based on an AI model. The data processing device based on an AI model can be implemented in the form of hardware and / or software. The data processing device based on an AI model can be configured in an electronic device. Figure 1 As shown, the method includes:

[0060] S110, identifying data from at least one data source, and determining sample data from the identified data; performing data analysis on the sample data through an AI model to obtain a data analysis result.

[0061] It is necessary to carefully identify all possible data sources in the park. The park data sources are extensive, covering policy data, enterprise data, operation data, equipment data, government data, property data, industry data and investment promotion data. Policy data may include various preferential policies, industrial support policies and other documents issued by the national and local governments for the park and related interpretation information; enterprise data involves the basic information of each enterprise in the park (such as company name, registered address, business scope, etc.), business data (such as revenue, profit, tax amount, etc.), and personnel information (number of employees, educational structure, etc.); operation data includes data generated during the daily operation and management of the park, such as water and electricity consumption, traffic flow, facility maintenance records, etc.; equipment data is data collected by various intelligent devices in the park (such as monitoring equipment, sensors, intelligent lighting systems, etc.); government data may come from the information exchange between the park management department and the higher-level government departments, such as administrative approval data, statistical reports, etc.; property data covers information on property charges, property service records, property facility management, etc.; industry data includes the development scale of various industries in the park, industrial structure, and distribution of upstream and downstream enterprises in the industrial chain; investment promotion data includes information on potential investment promotion targets, progress of investment promotion projects, investment promotion channels and effects, etc.

[0062] Extract representative sample data from each identified data source. These sample data should be able to reflect the main characteristics of the data source and the distribution of data types. For example, extract some data records from enterprises of different sizes and industries from enterprise data; select data samples collected by different types of equipment in different time periods from device data. The number and selection method of sample data need to be determined according to the characteristics of the data source and the needs of subsequent analysis. It must ensure sufficient representativeness, but not be too large to affect the collection efficiency.

[0063] The extracted sample data will serve as the basic material for subsequent AI data analysis, data classification identification, generation of data preprocessing rules and database design scripts. Through in-depth research on these sample data, we can understand the overall structure of the data, data quality status, potential data relationships, etc., thus providing a basis for formulating reasonable data processing strategies and rules.

[0064] S120. Add classification identification to the identified data through the AI ​​model.

[0065] In the embodiment of the present invention, the data analysis includes: data field attribute analysis, data field analysis, data null value rate analysis, data repetition rate analysis, data content analysis and data scenario analysis; accordingly, the data analysis of the sample data by the AI ​​model to obtain the data analysis result includes:

[0066] Performing field attribute analysis on the sample data through the AI ​​model to identify integers, floating point numbers, strings, and dates in each field;

[0067] Perform data field analysis on the sample data through the AI ​​model to perform statistical description, data distribution, and outlier detection;

[0068] Performing data vacancy rate analysis on the sample data through the AI ​​model to determine the impact of the vacancy value;

[0069] Performing data repetition rate analysis on the sample data through the AI ​​model, checking duplicate records and evaluating the impact of duplicate data;

[0070] Performing data content analysis on the sample data through the AI ​​model to identify sensitive information and perform semantic analysis;

[0071] The AI ​​model is used to perform data scenario analysis on the sample data to determine the relationship between the data and the data business scenario.

[0072] The AI ​​model is used to analyze the field attributes of the collected park sample data, which can accurately determine the data type of each field, such as integer, floating point, string, date, etc., as well as its corresponding format. For example, in the company's employee information table, the "employee ID" field may be identified as an integer type, and the "entry date" field is a date type, and the format may be "YYYY-MM-DD". This helps to correctly read, store and calculate the data during subsequent data processing. In addition to basic data types and formats, potential data relationships between fields can also be discovered. For example, in the company's sales data, there is a calculation relationship between the "order amount" field and the "product unit price" and "sales quantity" fields. Through the identification of this relationship, data verification and association analysis can be performed during data processing and analysis to ensure data accuracy and consistency.

[0073] Data field analysis and statistics include calculating statistical indicators such as mean, median, standard deviation, etc., to understand the central tendency and dispersion of data. At the same time, analyze the distribution of data to determine whether it conforms to common distribution rules (such as normal distribution, skewed distribution, etc.). Taking the annual profit data of enterprises in the park as an example, statistical description can be used to know the average profit, median profit and distribution range of profit data. If the data is found to be skewed, it may indicate that a small number of high-profit enterprises have a greater impact on the overall data, and further analysis of the reasons is needed. Identify outliers in the data, which may be caused by data entry errors, special circumstances or abnormal events. For example, in the operation data of park equipment, if a monitoring indicator of a certain equipment suddenly appears to be far beyond the normal range, it may indicate that the equipment is faulty or in abnormal operation state, which needs to be checked in time. At the same time, ensure that the format of all fields is unified, such as the unified date format of "YYYY-MM-DD", the numerical unit of "yuan" or "10,000 yuan", etc., to facilitate data comparison and analysis.

[0074] Data null rate analysis is to calculate the proportion of null values ​​in park data, that is, the null rate, through an AI model. The presence of null values ​​may affect the integrity of the data and the accuracy of subsequent analysis results. For example, in the contact information field of park enterprises, if the null rate is high, it may lead to an inability to communicate effectively with some enterprises, affecting investment promotion or enterprise service work. Depending on the null rate and the importance of the data, decide whether to process the null values, such as filling in reasonable default values, estimating and filling based on other relevant data, or directly deleting records containing null values ​​(if the data volume is large enough and the null value records have little impact on the overall analysis).

[0075] Data duplication rate analysis is to check the duplicate records in the park data and calculate the duplication rate. Duplicate data may lead to deviations in data analysis results. For example, when counting the number of enterprises in the park, if there are duplicate records, the number of enterprises will be inaccurate, which will affect other analyses based on the number of enterprises (such as enterprise density calculation, industrial structure analysis, etc.). For duplicate data, it is necessary to decide the processing method according to the specific situation, such as deleting duplicate records or retaining one of them and marking the duplication, to ensure the uniqueness of the data and the reliability of the analysis results.

[0076] Data content analysis uses AI technology to identify sensitive information in data, such as the company's business secrets and employees' personal privacy information (ID number, bank card number, etc.). During data processing and storage, sensitive information is specially processed, such as encrypted storage, access permission restrictions, etc., to ensure data security and privacy protection. Semantic understanding of text data, such as semantic analysis of the company's business description and the content of park policy documents. Semantic analysis can be used to extract key information, classify information topics, and mine potential semantic relationships. For example, extract core business keywords from the company's business description to conduct industry classification and enterprise association analysis; analyze policy orientation and scope of application from policy documents to provide accurate policy interpretation and services for park enterprises.

[0077] Data scenario analysis is to classify data into different scenario categories based on the business scenarios of park data. For example, data related to the entry of enterprises (such as investment promotion negotiation records, enterprise registration information, etc.) are classified as investment promotion scenario data; park facility operation and maintenance data (such as equipment maintenance records, water and electricity consumption data, etc.) are classified as operation management scenario data; industrial development trend analysis data (such as industry reports, enterprise innovation results data, etc.) are classified as industrial planning scenario data, etc. Through this classification, we can better understand the purpose and value of data, and provide a basis for subsequent targeted processing and analysis. Clarify the specific uses of data in different scenarios. For example, in the investment promotion scenario, some data can be used to evaluate the investment potential and adaptability of potential enterprises; in the operation management scenario, some data is used to optimize resource allocation and facility maintenance plans; in the industrial planning scenario, relevant data helps to formulate industrial development strategies and support policies, etc.

[0078] S130, generating data processing rules based on the analysis results of the data analysis and writing a database design script, and synchronizing the data processing rules and the database design script to a data processing platform.

[0079] Based on the data analysis results, data processing rules are defined and implemented on the data processing platform to complete the automated data processing process.

[0080] Write database design scripts based on the database structure of the park data characteristics to prepare for data entry, including table structure design, index optimization, etc.

[0081] In an embodiment of the present invention, generating data processing rules based on the analysis results of data analysis includes:

[0082] Determine an outlier judgment rule based on the analysis result of the data field attribute analysis;

[0083] Determining a format conversion rule according to the analysis result of the data field analysis;

[0084] Determine a filling rule and / or a deletion rule for null value data according to the analysis result of the data null value rate analysis;

[0085] Determining a deletion rule and / or a marking rule for duplicate data according to the analysis result of the data duplication rate analysis;

[0086] Determining a data value range rule according to the analysis result of the data content analysis;

[0087] The data rationality verification rules are determined according to the analysis results of the data scenario analysis.

[0088] Determine the processing method based on the results of the data null value rate analysis. If the null value rate of a field is low and has little impact on the overall analysis, you can use a filling strategy, such as filling with the field average, median, or mode; if the null value rate is high and the data importance is relatively low, consider deleting records containing null values. For example, in the park enterprise employee information table, if the null value rate of the "employee email" field is high and does not affect the core business analysis, you can choose to delete these null value records.

[0089] According to the data duplication analysis, if the duplication rate is low and the duplicate data has little impact on the analysis results, one of the records can be retained and the duplication can be marked; if the duplication rate is high and seriously affects the data accuracy, the duplicate records need to be deleted. For example, in the list of participants in the park activities, if there are a small number of duplicate records caused by operational errors and do not affect the statistical analysis of the activities, one copy can be retained and marked as duplicate, while the duplicate records in the enterprise registration information must be strictly deleted.

[0090] Combine the data distribution and business scenario knowledge in the data field analysis to judge the outliers. If the outliers are caused by explainable reasons such as data entry errors or equipment failures, they need to be corrected to reasonable values; if the outliers deviate from the normal range but conform to the business logic (such as the company's high profits due to special business), they can be retained but need to be marked. For example, in the park equipment operation data, if the temperature monitoring value of a certain device suddenly exceeds the normal range and it is verified that it is a sensor failure, it should be corrected to a reasonable temperature value; if a company's revenue increases significantly during a specific project cycle and it is a normal business fluctuation, it can be retained and the reason can be noted.

[0091] According to the results of data field attribute analysis, data in different formats are uniformly converted. Date data is unified into the "YYYY-MM-DD" format, numerical data is unified into units (such as equipment power units are unified into "kilowatts"), and text data is encoded and standardized (such as converting enterprise industry category names into industry classification codes). For example, in the sales data of park enterprises, all date-type order date fields are converted into a unified format to facilitate time series analysis and data comparison.

[0092] Based on the data distribution, the data is standardized to meet specific statistical distribution or business requirements. For example, the data is normalized to a specific interval (such as 0-1 or -1 to 1) to facilitate comparison between different data and subsequent model calculations. For example, in the credit score data of enterprises in the park, if the original score range varies greatly, it can be standardized to the 0-1 interval to improve the accuracy of the credit assessment model.

[0093] According to the statistical description (such as mean, standard deviation) and data distribution obtained from the data field analysis, a reasonable value range is set for the data. For data outside the range, an error is prompted and verified. For example, in the age data of employees in the park enterprises, if the average age is known to be 35 years old and the standard deviation is 5 years old, the reasonable age range can be set to 25-45 years old (adjustable according to actual conditions). When the newly collected age data exceeds this range, the system issues an alarm and requires manual verification.

[0094] Verify the rationality of data based on business logic and data relationships. For example, in corporate financial data, ensure that the accounting equation "Assets = Liabilities + Owners' Equity" is established. If not, it will indicate data errors. For the logical relationship between related data, such as the reasonable proportional relationship between the number of orders and sales of a company, if an abnormal ratio is found, further verification of the data is required.

[0095] In an embodiment of the present invention, writing a database design script includes:

[0096] Determine the table name and field according to the data analysis results, and set the primary key and foreign key;

[0097] Determine the index field based on the data query and analysis requirements; select the index type based on the field data characteristics and business requirements.

[0098] According to the classification of park data and business needs, determine the table names that need to be created in the database and the fields contained in each table. For example, create a "Basic Information Table of Enterprises" containing fields such as "Enterprise ID", "Enterprise Name", "Enterprise Type", "Registered Address", and "Legal Representative"; create an "Equipment Operation Data Table" containing fields such as "Equipment ID", "Equipment Name", "Operation Time", and "Monitoring Parameters". The data type of each field is determined based on the results of the data field attribute analysis. For example, "Enterprise ID" can be set to integer type, and "Enterprise Name" can be set to string type.

[0099] To ensure the uniqueness and relevance of data, set the primary key and foreign key reasonably. Usually, a field that can uniquely identify each record is selected as the primary key, such as the "Company ID" in the "Company Basic Information Table". For tables with association relationships, set foreign keys to establish the connection. For example, set "Company ID" as a foreign key in the "Equipment Operation Data Table" and associate it with the "Company ID" in the "Company Basic Information Table" to indicate the company to which the equipment belongs.

[0100] According to the data query and analysis needs, select the fields that are often used for query conditions, sorting or grouping to create indexes to improve data retrieval speed. For example, in the "Enterprise Basic Information Table", if you often query based on "Enterprise Name", you can create an index on the "Enterprise Name" field; in the "Equipment Operation Data Table", if you often search and analyze data based on "Operation Time", you can create an index on the "Operation Time" field.

[0101] Select the index type according to the characteristics of the field data and business needs, such as B-tree index, hash index, etc. For range queries (such as querying equipment operation data within a certain period of time), B-tree index works better; for equivalent queries (such as querying enterprise information based on enterprise ID), hash index may be more suitable. At the same time, be careful to avoid creating too many indexes, which will cause data update and insert operation performance to decrease. It is necessary to balance query performance and data operation performance.

[0102] S140, connecting the data source to the data processing platform, preprocessing the collected data using the data processing rules through the data processing platform and generating a database table using the database design script, importing the preprocessed data into the database table and storing it based on the classification identifier.

[0103] Through the data processing platform, database tables are created on the data storage through database design scripts.

[0104] Configure data source access and test that the data source can be effectively connected to the data processing platform and successfully access the data.

[0105] Set up scheduled tasks for data entry to ensure that data can be automatically updated and loaded according to the predetermined schedule.

[0106] Start the data warehousing operation, load the preprocessed data into the database, and complete the final data warehousing.

[0107] In an embodiment of the present invention, connecting a data source to a data processing platform includes:

[0108] The configuration parameters for data source access are configured on the data processing platform. If the data source is an internal relational database of an enterprise, the configuration parameters include the address, port number, database name, user name and password of the database server; if the data source is an IoT device, the configuration parameters include the IP address, communication port, data transmission protocol and device authentication information.

[0109] A connection test is used to verify the connection status of the data source on the data processing platform.

[0110] It is necessary to accurately configure various parameters for data source access on the data processing platform. For different types of data sources, the configuration parameters are different. For example, if the data source is an internal relational database (such as MySQL or Oracle) of the enterprise, it is necessary to provide information such as the address of the database server (such as IP address or domain name), port number (common MySQL port is 3306, Oracle port is 1521, etc.), database name, user name and password. These parameters enable the data processing platform to locate and connect to the target database. If the data source is an IoT device, it may be necessary to configure the device's IP address, communication port, data transmission protocol (such as MQTT, HTTP, etc.) and device authentication information (such as device ID and password) to establish a data communication link with the device. For government data interfaces, it is necessary to clarify the interface address, request method (such as GET, POST), request parameter format, and authentication method (such as APIKey), etc., to ensure that the data processing platform can correctly call the interface to obtain data.

[0111] After completing the parameter configuration, you can perform a connection test to verify whether the data source can be successfully connected on the data processing platform. The data processing platform will try to establish a connection with the data source according to the configured parameters and send a test request (such as querying a small amount of sample data in the database, obtaining a data sample from an IoT device, or sending a simple test request to the government data interface). If the returned data can be successfully received and the data format is correct, it means that the data source connection is normal; conversely, if a connection failure prompt appears (such as network timeout, authentication error, database connection rejection, etc.), you need to carefully check whether the configured parameters are correct, whether the network environment is stable, and whether the data source itself is operating normally. For example, if the database username or password is incorrect, the database connection will be rejected. At this time, you need to check and correct the username and password; if the network is unstable and the connection timeout occurs, you can check whether the network equipment (such as routers, switches) is working properly and whether the network cables are firmly connected.

[0112] In an embodiment of the present invention, importing the preprocessed data into the database table includes:

[0113] Inserting or updating the preprocessed data according to the structure and primary key constraints of the database table;

[0114] If the primary key value of the data does not exist in the database table, the data will be inserted into the corresponding table as a new record;

[0115] If the primary key value already exists, determine whether to update some or all fields of the original record based on specific business needs and preset rules.

[0116] Data will be inserted or updated according to the structure and primary key constraints of the database table. If the primary key value of the data does not exist in the database table, the data will be inserted into the corresponding table as a new record; if the primary key value already exists, it will be determined whether to update some or all fields of the original record based on specific business needs and preset rules. For example, for the basic information table of park enterprises, if the enterprise ID is the primary key, when the enterprise ID in the newly collected enterprise data does not exist in the table, a complete new enterprise record will be inserted; if the enterprise ID already exists, only the latest contact information or other changed fields of the enterprise may be updated to maintain the real-time and accuracy of the data. During the insertion and update operations, the database management system will automatically maintain the index structure of the data (if an index is set in the table) to ensure the efficiency of data query.

[0117] Figure 2 This is a flowchart of a data processing process based on an AI model applicable to the first embodiment of the present invention. Figure 2 As shown in the figure, starting from identifying the data source, through data collection and a series of data analysis steps using AI big models, including data field attribute analysis, data field analysis, data null value rate analysis, data repetition rate analysis, data content analysis and data scenario analysis. These steps are all to ensure the quality of the data and prepare for subsequent processing and analysis.

[0118] Data labeling is the process of associating data with specific labels, which usually involves manual or automated classification and identification processes. After data preprocessing, data processing rules and database design scripts will be formed. These rules and scripts will guide the data processing platform on how to correctly process and store data.

[0119] Synchronize the data processing rules and database design scripts to the data processing platform to automatically complete the execution of database scripts and the creation of database structure. After the database table is created, you can start accessing the data source.

[0120] After the data source is connected, data collection configuration can be performed on the data processing platform. These configurations ensure that data can be automatically collected and processed according to the predetermined plan and rules. Finally, the data storage operation is performed to store the preprocessed data in the database, completing the entire data preprocessing and storage process.

[0121] The purpose of the entire process is to ensure that data can be accurately and efficiently processed and stored for subsequent analysis and application. The data processing platform plays a central role in this process, providing the necessary tools and functions to support data collection, processing and storage.

[0122] Embodiment 2

[0123] Figure 3 Schematic diagram of a data processing device based on an AI model provided in Embodiment 2 of the present invention. Figure 3 As shown, the device comprises:

[0124] The data analysis unit 310 is used to identify data from at least one data source, determine sample data from the identified data, and perform data analysis on the sample data through an AI model to obtain a data analysis result;

[0125] An identification adding unit 320, configured to add a classification identification to the identified data through the AI ​​model;

[0126] An analysis result application unit 330 is used to generate data processing rules and write database design scripts based on the analysis results of the data analysis, and synchronize the data processing rules and the database design scripts to the data processing platform;

[0127] The data storage unit 340 is used to connect the data source to the data processing platform, pre-process the collected data using the data processing rules through the data processing platform and generate a database table using the database design script, import the pre-processed data into the database table and store it based on the classification identifier.

[0128] The AI ​​model-based data processing device provided in an embodiment of the present invention can execute the AI ​​model-based data processing method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0129] Embodiment 3

[0130] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0131] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0132] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0133] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data processing method based on an AI model.

[0134] In some embodiments, the data processing method based on the AI ​​model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method based on the AI ​​model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the data processing method based on the AI ​​model in any other appropriate manner (e.g., by means of firmware).

[0135] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0137] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0140] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0141] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0142] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method based on an AI model, characterized in that: include: Identify data from at least one data source, and determine sample data from the identified data; perform data analysis on the sample data using an AI model to obtain a data analysis result; Add classification identification to the identified data through the AI ​​model; Generate data processing rules and write database design scripts based on the analysis results of the data analysis, and synchronize the data processing rules and the database design scripts to the data processing platform; The data source is connected to the data processing platform, the collected data is preprocessed by the data processing platform using the data processing rules and a database table is generated using the database design script, the preprocessed data is imported into the database table and stored based on the classification identifier.

2. The method according to claim 1, characterized in that: The data analysis includes: data field attribute analysis, data field analysis, data null value rate analysis, data repetition rate analysis, data content analysis and data scenario analysis; accordingly, the data analysis results obtained by performing data analysis on the sample data through the AI ​​model include: Performing field attribute analysis on the sample data through the AI ​​model to identify integers, floating point numbers, strings, and dates in each field; Perform data field analysis on the sample data through the AI ​​model to perform statistical description, data distribution, and outlier detection; Performing data vacancy rate analysis on the sample data through the AI ​​model to determine the impact of the vacancy value; Performing data repetition rate analysis on the sample data through the AI ​​model, checking duplicate records and evaluating the impact of duplicate data; Performing data content analysis on the sample data through the AI ​​model to identify sensitive information and perform semantic analysis; The AI ​​model is used to perform data scenario analysis on the sample data to determine the relationship between the data and the data business scenario.

3. The method according to claim 1, characterized in that Adding classification identification to the identified data by using the AI ​​model includes: Performing field attribute analysis on the identified data, identifying the data type of the data and adding corresponding classification identifiers; Analyze the distribution and outliers of the identified data and add corresponding classification labels; According to the role of the identified data in the business process, the identified data is divided into different business scenario categories and corresponding classification identifiers are added; Add corresponding classification identifiers based on the importance and urgency of the identified data to the business.

4. The method according to claim 2, characterized in that: The data processing rules are generated based on the analysis results of the data analysis, including: Determine an outlier judgment rule based on the analysis result of the data field attribute analysis; Determining a format conversion rule according to the analysis result of the data field analysis; Determine a filling rule and / or a deletion rule for null value data according to the analysis result of the data null value rate analysis; Determining a deletion rule and / or a marking rule for duplicate data according to the analysis result of the data duplication rate analysis; Determining a data value range rule according to the analysis result of the data content analysis; The data rationality verification rules are determined according to the analysis results of the data scenario analysis.

5. The method according to claim 2, characterized in that: The writing of database design scripts includes: Determine the table name and field according to the data analysis results, and set the primary key and foreign key; Determine the index field based on the data query and analysis requirements; select the index type based on the field data characteristics and business requirements.

6. The method according to any one of claims 1 to 5, characterized in that: Connecting the data source to the data processing platform includes: The configuration parameters for data source access are configured on the data processing platform. If the data source is an internal relational database of an enterprise, the configuration parameters include the address, port number, database name, user name and password of the database server; if the data source is an IoT device, the configuration parameters include the IP address, communication port, data transmission protocol and device authentication information. A connection test is used to verify the connection status of the data source on the data processing platform.

7. The method according to any one of claims 1 to 6, characterized in that: The step of importing the preprocessed data into the database table comprises: Inserting or updating the preprocessed data according to the structure and primary key constraints of the database table; If the primary key value of the data does not exist in the database table, the data will be inserted into the corresponding table as a new record; If the primary key value already exists, determine whether to update some or all fields of the original record based on specific business needs and preset rules.

8. A data processing device based on an AI model, characterized in that: include: A data analysis unit, configured to identify data from at least one data source, determine sample data from the identified data, and perform data analysis on the sample data using an AI model to obtain a data analysis result; An identification adding unit, used to add a classification identification to the identified data through the AI ​​model; An analysis result application unit, used to generate data processing rules and write database design scripts based on the analysis results of the data analysis, and synchronize the data processing rules and the database design scripts to the data processing platform; The data storage unit is used to connect the data source to the data processing platform, pre-process the collected data using the data processing rules through the data processing platform and generate a database table using the database design script, import the pre-processed data into the database table and store it based on the classification identifier.

9. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the AI ​​model-based data processing method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method based on the AI ​​model described in any one of claims 1 to 7 when executed.

11. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the data processing method based on the AI ​​model according to any one of claims 1 to 7.

Images