Time standardization governance method and system based on AI training model and SQL governance

Abstract

The invention discloses a time standardization governance method and system based on an AI training model and SQL governance, and relates to the technical field of data governance and artificial intelligence, and the method comprises the steps: pre-training an AI large model to recognize and analyze irregular time data, and outputting a standardized time format; associating the time data analyzed by the AI large model with the original time data, and storing the time data and the original time data in an SQL (Structured Query Language) database; and constructing an SQL statement according to the analyzed time data format, and operating the time data in the database through the constructed SQL statement so as to calculate and obtain the standardized end time. According to the method, various irregular time data can be automatically and accurately converted into a uniform format, time is screened and calculated in batches through sql statements, standardized processing of the time data is achieved, and therefore the efficiency and accuracy of data governance are improved.

Description

Technical Field

[0001] The present invention relates to the field of data management and artificial intelligence technology, and specifically to a time standardization management method and system based on AI training model and SQL management. Background Art

[0002] In the modern information society, time data is an indispensable part of various application systems. However, due to the diversity of data sources and the irregularities in the collection process, as well as the fact that time data from different systems and sources have inconsistent formats and precision, and the collection methods are also different, time data formats vary and are difficult to process uniformly. For example, in the data collection process, time data often appears in irregular and inconsistent formats, such as "two years", "18 months", "20 calendar days", etc., and we usually need to add these data to the start time to get the final end time. However, these data bring great difficulties to the data governance and subsequent analysis of the final time. Therefore, these data need to be standardized for unified analysis and application.

[0003] Traditional data governance methods often rely on manual correction and optimization, which is inefficient and prone to errors. Therefore, this paper designs and proposes a time standardization governance method and system based on AI training model and SQL governance, which can efficiently and accurately process time data of various formats and sources, and has important practical significance and application value. Summary of the invention

[0004] In response to the needs and shortcomings of current technological development, the present invention provides a time standardization management method and system based on AI training model and SQL management.

[0005] In the first aspect, the present invention provides a time standardization management method based on AI training model and SQL management. The technical solution adopted to solve the above technical problems is as follows:

[0006] A time standardization management method based on AI training model and SQL management includes the following steps:

[0007] S1. Pre-train an AI model to identify and parse irregular time data and output a standardized time format;

[0008] S2. Associate the time data parsed by the AI ​​big model with the original time data and store them in the SQL database;

[0009] S3. Construct an SQL statement according to the parsed time data format, and operate the time data in the database through the constructed SQL statement to calculate the standardized end time.

[0010] Optionally, the pre-training process of the AI ​​large model involved is as follows:

[0011] Collect text information containing irregular time data, perform manual annotation, and convert the time data into a unified format as training data for the AI ​​large model;

[0012] Use the training data to train the AI ​​big model. The trained AI big model can recognize and parse irregular time data and output a standardized time format.

[0013] The trained AI big model is applied to the actual time data parsing task. The AI ​​big model outputs the parsed standardized time format based on the input irregular time data.

[0014] Preferably, text information containing irregular time data is collected from the website through a python script.

[0015] Preferably, a large AI model suitable for natural language processing is pre-trained.

[0016] Optionally, execute step S3, construct an SQL statement according to the parsed time data format, execute the constructed SQL statement in the database, add or subtract the time parsed by the AI ​​large model from the original time, and obtain a standardized end time.

[0017] In the second aspect, the present invention provides a time standardization management system based on AI training model and SQL management. The technical solutions adopted to solve the above technical problems are as follows:

[0018] A time standardization management system based on AI training model and SQL management, which includes:

[0019] The pre-training module is used to pre-train a large AI model to identify and parse irregular time data and output a standardized time format;

[0020] The association storage module is used to associate the time data parsed by the AI ​​big model with the original time data and store them in the SQL database;

[0021] An execution module is constructed to construct an SQL statement according to the time data format parsed by the AI ​​large model, and the time data in the database is operated through the constructed SQL statement to calculate the standardized end time.

[0022] Optionally, the pre-training modules involved include:

[0023] The information collection submodule is used to collect text information containing irregular time data;

[0024] Data annotation submodule, used to assist manual annotation of time data;

[0025] The format conversion submodule is used to convert manually annotated time data into a unified format as training data for the AI ​​large model;

[0026] The model training submodule is used to train the AI ​​big model using the training data. The trained AI big model can recognize and parse irregular time data and output a standardized time format.

[0027] Preferably, the information collection submodule involved collects text information containing irregular time data from the website through a python script.

[0028] Preferably, the pre-training module involved pre-trains a large AI model suitable for natural language processing.

[0029] Optionally, the construction and execution module involved first constructs an SQL statement according to the parsed time data format, and then executes the constructed SQL statement in the database, adding or subtracting the time parsed by the AI ​​large model from the original time to obtain a standardized end time.

[0030] The time standardization management method and system based on AI training model and SQL management of the present invention have the following beneficial effects compared with the prior art:

[0031] 1. The present invention can automatically and accurately convert various irregular time data into a unified format, and screen and batch calculate the time through SQL statements to achieve standardized processing of time data, thereby improving the efficiency and accuracy of data management;

[0032] 2. The present invention combines AI big model parsing and SQL governance methods to achieve automatic, accurate parsing and standardized processing of various irregular time data, and achieves standardized governance and acquisition of end time data through start time and project cycle or construction period. This not only improves the efficiency and accuracy of data governance, but also provides a reliable foundation for subsequent data analysis and application. It has high flexibility and scalability and can adapt to the needs of different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Attached Figure 1 is a flow chart of a method according to Embodiment 1 of the present invention;

[0034] Attached Figure 2 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0036] Embodiment 1:

[0037] Combined with Figure 1 This embodiment proposes a time standardization management method based on AI training model and SQL management, which includes the following steps:

[0038] S1. Pre-train an AI large model suitable for natural language processing to identify and parse irregular time data and output standardized time format.

[0039] The AI ​​big model can be BERT or GPT.

[0040] The pre-training process of the AI ​​large model is as follows:

[0041] Use Python scripts to collect text information containing irregular time data from websites, perform manual annotation, and convert the time data into a unified format as training data for the AI ​​model; for example, "this study lasts for 9 months", "for 2 years", and "20 calendar days" are converted into "9 months", "2 years", and "20 days";

[0042] Use the training data to train the AI ​​big model. The trained AI big model can recognize and parse irregular time data and output a standardized time format.

[0043] The trained AI big model is applied to the actual time data parsing task. The AI ​​big model outputs the parsed standardized time format based on the input irregular time data.

[0044] S2. Associate the time data parsed by the AI ​​big model with the original time data and store them in a SQL database.

[0045] S3. Construct an SQL statement according to the parsed time data format, execute the constructed SQL statement in the database, add or subtract the time parsed by the AI ​​large model from the original time, and obtain a standardized end time.

[0046] For example, for time data in the format of "day", you can use the following SQL statement:

[0047] 1. If the start time is in the format of YYYY-MM-DD and YYYY.MM.DD, the unified start time format is YYYY-MM-DD (if there are other irregular formats, the SQL may need to be rewritten).

[0048] UPDATE table_time

[0049] SET

[0050] START_TIME_deal=CONCAT(SUBSTR(START_TIME,1,4),'-',SUBSTR(START_TIME,6,2),'-',SUBSTR(START_TIME,9,2)).

[0051] 2. Use the DATE_ADD function to add the start time and the number of days to get the end time.

[0052] UPDATE table_time SET endTime=DATE_ADD(startTime,INTERVAL SUBSTR(parsed time,1,INSTR(parsed time,'day')-1)DAY)

[0053] WHERE parsed time LIKE'%day%' AND trim(SUBSTR(parsed time,1,INSTR(parsed time,'month')-1)) REGEXP'[^0-9]'=0;

[0054] It should be noted that the purpose of trim(SUBSTR(parsed time, 1, INSTR(parsed time, 'month')-1))REGEXP'[^0-9]'=0 is to ensure that the number of days is pure digits.

[0055] Execute the SQL statements constructed in 1 and 2 above in the database to operate the time data to obtain the standardized end time.

[0056] Embodiment 2:

[0057] Combined with Figure 2 This embodiment proposes a time standardization management system based on AI training model and SQL management, which includes:

[0058] The pre-training module is used to pre-train a large AI model suitable for natural language processing to identify and parse irregular time data and output a standardized time format;

[0059] The association storage module is used to associate the time data parsed by the AI ​​big model with the original time data and store them in the SQL database;

[0060] Build an execution module to construct an SQL statement according to the time data format parsed by the AI ​​big model, and operate the time data in the database through the constructed SQL statement to calculate the standardized end time.

[0061] In this embodiment, the AI ​​big model can be BERT or GPT.

[0062] In this embodiment, the pre-training module involved also includes:

[0063] The information collection submodule is used to collect text information containing irregular time data from the website through Python scripts;

[0064] Data annotation submodule, used to assist manual annotation of time data;

[0065] The format conversion submodule is used to convert manually annotated time data into a unified format as training data for the AI ​​large model; for example, "this study lasts for 9 months", "for a period of 2 years", and "20 calendar days" are converted into "9 months", "2 years", and "20 days";

[0066] The model training submodule is used to train the AI ​​big model using the training data. The trained AI big model can recognize and parse irregular time data and output a standardized time format.

[0067] In this embodiment, the construction and execution module involved first constructs an SQL statement according to the parsed time data format, and then executes the constructed SQL statement in the database, adding or subtracting the time parsed by the AI ​​large model and the original time to obtain a standardized end time.

[0068] For example, for time data in the format of "day", you can use the following SQL statement:

[0069] For example, for time data in the format of "day", you can use the following SQL statement:

[0070] 1. If the start time is in the format of YYYY-MM-DD and YYYY.MM.DD, the unified start time format is YYYY-MM-DD (if there are other irregular formats, the SQL may need to be rewritten).

[0071] UPDATE table_time

[0072] START_TIME_deal=CONCAT(SUBSTR(START_TIME,1,4),'-',SUBSTR(START_TIME,6,2),'-',SUBSTR(START_TIME,9,2)).

[0073] 2. Use the DATE_ADD function to add the start time and the number of days to get the end time.

[0074] UPDATE table_time SET endTime=DATE_ADD(startTime,INTERVAL SUBSTR(parsed time,1,INSTR(parsed time,'day')-1)DAY)

[0075] WHERE parsed time LIKE'%day%' AND trim(SUBSTR(parsed time,1,INSTR(parsed time,'month')-1)) REGEXP'[^0-9]'=0;

[0076] It should be noted that the purpose of trim(SUBSTR(parsed time, 1, INSTR(parsed time, 'month')-1))REGEXP'[^0-9]'=0 is to ensure that the number of days is pure digits.

[0077] Execute the SQL statements constructed in 1 and 2 above in the database to operate the time data to obtain the standardized end time.

[0078] In summary, the time standardization management method and system based on AI training model and SQL management of the present invention can identify and parse irregular time data, and perform further calculations, screening and other operations on these data in combination with SQL statements, thereby realizing standardized processing of time data and improving the efficiency and accuracy of data management.

[0079] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A time standardization management method based on AI training model and SQL management, characterized in that: The steps include: S1. Pre-train an AI model to identify and parse irregular time data and output a standardized time format; S2. Associate the time data parsed by the AI ​​big model with the original time data and store them in the SQL database; S3. Construct an SQL statement according to the parsed time data format, and operate the time data in the database through the constructed SQL statement to calculate the standardized end time.

2. According to claim 1, the time standardization management method based on AI training model and SQL management is characterized in that: The pre-training process of the AI ​​large model is as follows: Collect text information containing irregular time data, perform manual annotation, and convert the time data into a unified format as training data for the AI ​​large model; Use the training data to train the AI ​​big model. The trained AI big model can recognize and parse irregular time data and output a standardized time format. The trained AI big model is applied to the actual time data parsing task. The AI ​​big model outputs the parsed standardized time format based on the input irregular time data.

3. The time standardization management method based on AI training model and SQL management according to claim 2 is characterized in that: Use Python scripts to collect text information containing irregular time data from websites.

4. The time standardization management method based on AI training model and SQL management according to claim 2 is characterized in that: Pre-train a large AI model suitable for natural language processing.

5. The time standardization management method based on AI training model and SQL management according to claim 1 is characterized in that: Execute step S3, construct an SQL statement according to the parsed time data format, execute the constructed SQL statement in the database, add or subtract the time parsed by the AI ​​large model from the original time, and obtain a standardized end time.

6. A time standardization management system based on AI training model and SQL management, characterized in that: It includes: The pre-training module is used to pre-train a large AI model to identify and parse irregular time data and output a standardized time format; The association storage module is used to associate the time data parsed by the AI ​​big model with the original time data and store them in the SQL database; Build an execution module to construct an SQL statement according to the time data format parsed by the AI ​​big model, and operate the time data in the database through the constructed SQL statement to calculate the standardized end time.

7. The time standardization management system based on AI training model and SQL management according to claim 6 is characterized in that: The pre-training module further comprises: The information collection submodule is used to collect text information containing irregular time data; Data annotation submodule, used to assist manual annotation of time data; The format conversion submodule is used to convert manually annotated time data into a unified format as training data for the AI ​​large model; The model training submodule is used to train the AI ​​big model using the training data. The trained AI big model can recognize and parse irregular time data and output a standardized time format.

8. The time standardization management system based on AI training model and SQL management according to claim 7 is characterized in that: The information collection submodule collects text information containing irregular time data from the website through a python script.

9. The time standardization management system based on AI training model and SQL management according to claim 7 is characterized in that: The pre-training module pre-trains a large AI model suitable for natural language processing.

10. The time standardization management system based on AI training model and SQL management according to claim 6 is characterized in that: The construction and execution module first constructs an SQL statement according to the parsed time data format, and then executes the constructed SQL statement in the database, adding or subtracting the time parsed by the AI ​​large model and the original time to obtain a standardized end time.

Images