Adaptive time format conversion method based on HyperSCAN

Abstract

The invention provides an adaptive time format conversion method based on HyperSCAN. The adaptive time format conversion method comprises the following steps: calling an interface provided by the HyperSCAN to carry out multi-mode matching; using the successfully matched regular expressions to obtain category and time format character strings from the Map; a step of creating a time format converter using the category and the time format character string; performing format conversion on the data; and outputting a conversion result. The method provided by the invention can ensure that time data from different data sources and in various formats can be received and processed, time stamps in units of seconds or milliseconds, specific character string formats or time information in other self-defined formats can be accurately identified and converted, and the time information can be accurately identified and converted. The application range of time format conversion is greatly widened, various complex time data sources can be seamlessly connected, and the problem that in the prior art, time data formats are poor in compatibility is solved.

Description

Technical Field

[0001] This invention relates to the field of time format conversion methods, and in particular to an adaptive time format conversion system and method based on HyperSCAN. Background Technology

[0002] Currently, there are various implementation methods in the field of time format conversion technology, but when faced with complex and ever-changing application scenarios, the existing implementation methods have revealed many limitations.

[0003] Traditional time conversion schemes often rely on fixed rules and algorithms. For example, some simple systems simply convert local time to Coordinated Universal Time (UTC) or other specific time zones based on a preset time zone offset. Alternatively, a specific time can be specified; other systems only support setting a regular expression or a specific string format.

[0004] Some time conversion devices rely on specific hardware chips, which are often designed for a particular type of time conversion requirement and have poor flexibility.

[0005] There are currently some technologies that can support the conversion of multiple time formats at the same time; however, they use multiple regular expressions to match each one individually.

[0006] Hyperscan is a high-performance multi-pattern matching engine developed by Intel. It is based on automata technology and achieves fast regular expression matching by preprocessing regular expressions into a series of bitmaps and then performing matching through bitwise operations.

[0007] Dates in China are commonly formatted as "yyyy / mm / dd", while in North America they are typically formatted as "mm / dd / yyyy". For example, "2010 / 11 / 20" in North American format is "11 / 20 / 2010". In programming, if a date string in "yyyy / mm / dd" format is given, converting it to "mm / dd / yyyy" format can be achieved through string manipulation.

[0008] In Excel, to convert timestamps to a normal time format, first set the target column (e.g., column B) to time format. In cell B1, enter the formula "=(A1+8*3600) / 86400+70*365+19", press Enter, select B1, and double-click the "+" in the lower left corner. This will convert the timestamps in the column to time format. The satellite system time format is 48-bit 0.1ms counters, with Beijing time January 1, 2000 00:00:00 as the starting point. Each increment of 1 in the 0.1ms counter represents an increase of 0.1ms in the satellite system time.

[0009] In applications, different time formats may be encountered, and these different time formats need to be converted to a specified time format for use. However, the relevant time format conversion technologies have at least the following problems:

[0010] Most traditional time conversion techniques only support one explicit conversion rule and cannot support simultaneous conversion of multiple time formats. While this approach is relatively simple to implement, it lacks adaptability to dynamic changes in time data and diverse input formats.

[0011] Conversion devices that rely on specific hardware chips, such as some early time-to-digital converters, are hardware-based solutions that are expensive and can significantly increase the overall system cost in scenarios that require a large number of time conversion functions.

[0012] The traditional method of matching each regular expression line by line requires time and system resources that increase linearly with the number of regular expressions. In the field of data processing and analysis, with the explosive growth of data volume and the increasing demands for real-time data processing, the high computational complexity may lead to slow processing speeds, making it impossible to provide accurate time conversion results for subsequent data processing flows in a timely manner, thus affecting the overall system efficiency.

[0013] Traditional time conversion technologies require on-site implementation or maintenance personnel to understand the regular expressions or specific expression strings corresponding to most time formats. When data sources are inconsistent (e.g., some timestamps are represented in seconds, while others are represented in a specific string format) or when the time format of the source data is adjusted, on-site maintenance personnel need to readjust the program code and restart the system as needed, resulting in high operational complexity. Summary of the Invention

[0014] This invention aims to address a series of problems exposed by existing time conversion technologies when facing complex and ever-changing application scenarios, and to provide an adaptive time format conversion method based on HyperSCAN.

[0015] The technical solution adopted by this invention to achieve its technical objective is: an adaptive time format conversion method based on HyperSCAN, comprising the following steps:

[0016] Step 1: Call the interface provided by HyperSCAN to perform multi-mode matching;

[0017] Step 2: Use the successfully matched regular expression to retrieve the category and time format strings from the Map;

[0018] Step 3: Create a time format converter using the category and time format string;

[0019] Step 4: Convert the data format;

[0020] Step 5: Output the conversion result.

[0021] Furthermore, in the above-mentioned HyperSCAN-based adaptive time format conversion method:

[0022] Before step 1, check if there is a preferred time format converter. If so, call the preferred time format converter to perform the format conversion; otherwise, proceed to step 1.

[0023] Furthermore, in the above-mentioned HyperSCAN-based adaptive time format conversion method:

[0024] The preferred time format converter is invoked from the rules database.

[0025] Furthermore, in the above-mentioned HyperSCAN-based adaptive time format conversion method:

[0026] The rules database contains over 100 different time formats and supports customizable extension interfaces.

[0027] Furthermore, in the above-mentioned HyperSCAN-based adaptive time format conversion method:

[0028] Users add custom time format conversion rules to the rule database by calling the add() method:

[0029] timeFormatter.add("\d{4}-(0[1-9]|1[0-2])-(0[1-9]|

[12] \d|3

[01] )","old","yyyy-MM-dd");

[0030] The parameters are as follows: the first parameter is a regular expression; the second parameter is the category; and the third parameter is the corresponding time format string. The categories include three types: built-in, old, and new. Old represents the time format before Java 8, and the corresponding time format converter is SimpleDateTimeFormat or FastDateFormat. New represents the time format after Java 8, and the corresponding time format converter is DateTimeFormatter. Built-in represents the user-extended time format.

[0031] Furthermore, in the above-mentioned HyperSCAN-based adaptive time format conversion method: in the rule database: the later the conversion rule is added, the higher the priority; for conversion rules with the same regular expression, the later added conversion rule will override the earlier added conversion rule.

[0032] Furthermore, in the above-mentioned HyperSCAN-based adaptive time format conversion method, a new time format is added by setting three fields: regular expression, category, and time format string.

[0033] Furthermore, in the above-mentioned HyperSCAN-based adaptive time format conversion method, constructing the rule database includes the following steps:

[0034] Step 01: Read the built-in time format conversion rules and process them one by one;

[0035] Step 02: Add the regular expression to the list, and add the regular expression and the correspondence between the category and the time format string to the Map;

[0036] Step 03: Read the extended time format conversion rules and process them one by one;

[0037] Step 04: Determine if a regular expression already exists in the Map. If it does, proceed to step 05; otherwise, proceed to step 06.

[0038] Step 05: Replace the category and time format characters corresponding to the regular expressions in Map, then proceed to step 07;

[0039] Step 06: Add the regular expression to the list, and add the regular expression and the correspondence between the category and the time format string to the Map;

[0040] Step 07: Construct a rule database using lists.

[0041] Furthermore, in the above-mentioned HyperSCAN-based adaptive time format conversion method:

[0042] If the preferred time format converter is successfully invoked for format conversion, proceed to step 5; otherwise, proceed to step 1.

[0043] The method of this invention can ensure the reception and processing of time data from different data sources and in various formats. Whether it's a timestamp in seconds or milliseconds, a specific string format (such as yyyy-MM-dd HH:mm:ss), or other custom-formatted time information, it can accurately identify and convert it. This greatly expands the application scope of time format conversion, enabling seamless integration with various complex time data sources and solving the problem of poor time data format compatibility in existing technologies. For example, in log acquisition systems, different software versions use different time formats. This method can quickly and accurately process these different time data formats simultaneously and convert them into a unified format, providing support for subsequent data analysis.

[0044] This invention utilizes HyperScan's high-performance multi-pattern matching engine to perform in-depth analysis of input time data. Through more than 100 predefined time formats, it can quickly scan data and accurately match the corresponding time format, which significantly improves the efficiency of time format conversion and reduces the time required for time format conversion.

[0045] This invention also provides a preferred time format converter function, which can further reduce the number of regular expression matches in scenarios with fewer time formats, avoid frequent creation and destruction of time format converters, and further improve the efficiency of time format conversion.

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the time format conversion of the present invention;

[0048] Figure 2 A flowchart for constructing a rule database for this invention. Detailed Implementation

[0049] This embodiment aims to address a series of problems exposed by existing time conversion technologies when facing complex and ever-changing application scenarios. It is an adaptive time format conversion method based on HyperSCAN, such as... Figure 1 As shown, it includes the following steps:

[0050] Step 1: Call the interface provided by HyperSCAN to perform multi-mode matching;

[0051] Step 2: Use the successfully matched regular expression to retrieve the category and time format strings from the Map;

[0052] Step 3: Create a time format converter using the category and time format string;

[0053] Step 4: Convert the data format;

[0054] Step 5: Output the conversion result.

[0055] The details are as follows:

[0056] Time data formats are diverse

[0057] Existing traditional time conversion schemes are mostly based on fixed rules and algorithms, making it difficult to adapt to dynamic changes in time data and diverse input formats. This application collects a wide variety of time formats and their corresponding regular expressions, and has built-in support for the conversion of more than 100 different time format data, such as 10-digit seconds, 13-digit milliseconds, traditional Java date and time formats, and time formats supported by Java Time in JDK 8 and later. Furthermore, it provides extension rules, which can be freely extended by configuring regular expressions, types, and time format strings according to the provided rules.

[0058] In this embodiment, only the pre-packaged JAR file needs to be imported, and the interface provided by this application can be called to adaptively complete the format conversion of various time data formats without the need for hardware chip support.

[0059] The HyperSCAN technology used in this embodiment is a high-performance multi-pattern matching engine. Compared to matching regular expressions one by one, HyperSCAN constructs a dynamic rule database and performs parallel matching of multiple regular expressions, which can more efficiently convert time formats. Furthermore, since most data processing scenarios involve data with the same format, this application records the corresponding time format converter after a successful match. Subsequent data is then directly converted using this time format converter, greatly reducing the number of regular expressions and achieving higher conversion performance.

[0060] This embodiment supports format conversion for most time-formatted data, requiring minimal intervention from maintenance or implementation personnel to set regular expressions or time format strings. For example... Figure 2 The following steps are involved in building a rules database:

[0061] Step 01: Read the built-in time format conversion rules and process them one by one;

[0062] Step 02: Add the regular expression to the list, and add the regular expression and the correspondence between the category and the time format string to the Map;

[0063] Step 03: Read the extended time format conversion rules and process them one by one;

[0064] Step 04: Determine if a regular expression already exists in the Map. If it does, proceed to step 05; otherwise, proceed to step 06.

[0065] Step 05: Replace the category and time format characters corresponding to the regular expressions in Map, then proceed to step 07;

[0066] Step 06: Add the regular expression to the list, and add the regular expression and the correspondence between the category and the time format string to the Map;

[0067] Step 07: Construct a rule database using lists.

[0068] If the time format converter is switched, a log will be printed, recording the old and new time format strings, which helps to trace the source of the problem and make adjustments.

[0069] In the HyperSCAN-based adaptive time conversion method involved in this embodiment, the overall architecture design of the conversion device aims to fully leverage the advantages of HyperSCAN and effectively solve many problems faced by existing time conversion technologies. Its core technical solution mainly includes the following key parts:

[0070] Conversion rule configuration

[0071] This application discloses a method for configuring time format conversion rules, which adds a new time format by setting three fields: a regular expression, a category, and a time format string. The device disclosed in this application has more than 100 built-in different time formats and supports expansion.

[0072] For example:

[0073]

[0074] When the application starts, the method described in this embodiment loads all built-in and extended time format conversion rules, builds a rule database, and uses a Map to store regular expressions and their corresponding category and time format strings, such as... Figure 2 As shown. When a user-extended time format conversion rule has the same regular expression as the built-in time format conversion rule, the built-in time format conversion rule will be overridden.

[0075] Preferred Time Format Converter: Since data processing typically deals with similar data in relatively simple time formats (usually only one or a few), it eliminates the need to call the HyperSCAN interface for multi-pattern matching for every data entry, thus saving performance. The device provided in this application offers a preferred time format converter, which can persist one or more time format converters, avoiding frequent creation and destruction of time format converters and reducing the performance overhead of multi-pattern matching.

[0076] The preferred time format converter is an optional configuration. It can be turned off when performance is sufficient or when there are many data time formats.

[0077] Time format conversion: If no preferred time format converter is currently available, or if an exception is thrown when converting the time of this data using the preferred time format converter, the HyperSCAN interface will be called for multi-pattern matching. Upon successful matching, the corresponding category and time format string are obtained through the matched regular expression. Then, the factory method provided by the device described in this application is called to create the time format converter corresponding to that category and time format string. The time format converter is used to convert the time, and the result is output, setting that time format converter as the preferred time format converter.

[0078] Log printing: To facilitate traceability by operations and maintenance personnel, logs will be printed when switching the preferred time format converter. The logs include the category and time format string of the time format conversion rule before the conversion, as well as the category and time format string of the time format conversion rule after the conversion.

[0079] The technical advantages of this embodiment are as follows: This embodiment can ensure the reception and processing of time data from different data sources and in various formats. Whether it is a timestamp in seconds or milliseconds, a specific string format (such as yyyy-MM-dd HH:mm:ss), or other custom-formatted time information, it can accurately identify and convert it. This greatly expands the application scope of this device, enabling it to seamlessly connect with various complex time data sources and solves the problem of poor time data format compatibility in existing technologies. For example, in a log acquisition system, different versions of software have different time formats. This device can quickly and accurately process these different formats of time data simultaneously and convert them into a unified format, providing support for subsequent data analysis.

[0080] This embodiment utilizes HyperScan's high-performance multi-pattern matching engine to perform in-depth analysis of the input time data. With over 100 predefined time formats, it can quickly scan the data and accurately match the corresponding time format. This process significantly improves the efficiency of time format conversion and reduces the time required for time format conversion.

[0081] This embodiment also provides a preferred time format converter function, which can further reduce the number of regular expression matches in scenarios with fewer time formats, avoid frequent creation and destruction of time format converters, and further improve the efficiency of time format conversion.

[0082] The specific steps of this embodiment are as follows:

[0083] Initializing the device: First, the initialization method provided by this device needs to be called to create an adaptive time format converter. Example:

[0084] AdaptiveTimeFormatter.timeFormatter=daptiveTimeFormatter.getInstance();

[0085] Custom extended time format conversion rules (optional): This embodiment includes over 100 different time format conversion rules and provides an interface for customization. Users can add custom time format conversion rules by calling the add() method, for example:

[0086] timeFormatter.add("\d{4}-(0[1-9]|1[0-2])-(0[1-9]|

[12] \d|3

[01] )","old","yyyy-MM-dd");

[0087] Where: the first parameter is a regular expression; the second parameter is the category; and the third parameter is the corresponding time format string.

[0088] Currently, it supports three categories: built-in, old, and new. Custom extensions are currently supported for the old and new categories. The old category represents time formats prior to Java 8, and the corresponding time format converter is SimpleDateTimeFormat or FastDateFormat. The new category represents time formats after Java 8, and the corresponding time format converter is DateTimeFormatter.

[0089] It is important to note that the later a conversion rule is added, the higher its priority. For the same regular expression, the later conversion rule will override the earlier one.

[0090] Time format conversion: After configuring all time format conversion rules, proceed to the time format conversion step. This device provides several different format methods to convert the input raw string data into the specified time format:

[0091]

[0092] In this embodiment, the output format can be flexibly switched according to the called conversion method, where: toMillis(String time) outputs 13-digit milliseconds; toSeconds(String time) outputs 10-digit seconds; toDate(String time) outputs java.util.Date; toCalendar(String time) outputs java.util.Calendar; formatOld(String time, String format) outputs the string in the format specified by format before Java 8; formatNew(String time, String format) outputs the string in the format specified by format after Java 8; toLocalDate(String time) outputs java.time.LocalDate; toLocalDateTime(String time) outputs java.time.LocalDateTime; toInstant(String time) outputs java.time.Instant.

[0093] When any of the conversion methods is called for the first time, the interface provided by HyperSCAN is invoked to perform multi-pattern matching. Then, a time formatter is created according to the matched rules. After that, the time formatter is used to convert the input time string. Once the conversion is complete, the time formatter is set as the preferred time formatter.

[0094] When calling the conversion method later, first try to use the preferred time formatter for format conversion, and output the result directly if the conversion is successful.

[0095] If the conversion fails, the HyperSCAN interface will be called to perform multi-pattern matching. Then, a time formatter will be created according to the matched rules. After that, the time formatter will be used to convert the input time string. Once the conversion is complete, the time formatter will be set as the preferred time formatter.

[0096] Whether setting the preferred time formatter for the first time or changing the preferred time formatter, a log will be printed, including the category and time format string of the time format conversion rule before the conversion, and the category and time format string of the time format conversion rule after the switch.

Claims

1. An adaptive time format conversion method based on HyperSCAN, characterized in that: Includes the following steps: Step 1: Call the interface provided by HyperSCAN to perform multi-mode matching; Step 2: Use the successfully matched regular expression to retrieve the category and time format strings from the Map; Step 3: Create a time format converter using the category and time format strings; Step 4: Convert the data format; Step 5: Output the conversion result.

2. The adaptive time format conversion method based on HyperSCAN according to claim 1, characterized in that: Before step 1, it also checks if there is a preferred time format converter. If so, the preferred time format converter is called to perform the format conversion; otherwise, it proceeds to step 1.

3. The adaptive time format conversion method based on HyperSCAN according to claim 2, characterized in that: The preferred time format converter is invoked from the rules database.

4. The adaptive time format conversion method based on HyperSCAN according to claim 3, characterized in that: The rules database contains over 100 different time formats and supports customizable extension interfaces.

5. The adaptive time format conversion method based on HyperSCAN according to claim 4, characterized in that: Users add custom time format conversion rules to the rule database by calling the add() method: timeFormatter.add("\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01])","old","yyyy-MM-dd"); The parameters are as follows: the first parameter is a regular expression; the second parameter is the category; and the third parameter is the corresponding time format string. The categories include three types: built-in, old, and new. Old represents the time format before Java 8, and the corresponding time format converter is SimpleDateTimeFormat or FastDateFormat. New represents the time format after Java 8, and the corresponding time format converter is DateTimeFormatter. Built-in represents the user-extended time format.

6. The adaptive time format conversion method based on HyperSCAN according to claim 5, characterized in that: In the rule database: the later a transformation rule is added, the higher its priority. For transformation rules with the same regular expression, the later-added transformation rule will override the earlier-added transformation rule.

7. The adaptive time format conversion method based on HyperSCAN according to claim 6, characterized in that: A new time format can be added by setting three fields: regular expression, category, and time format string.

8. The adaptive time format conversion method based on HyperSCAN according to claim 7, characterized in that: Building a rules database involves the following steps: Step 01: Read the built-in time format conversion rules and process them one by one; Step 02: Add the regular expression to the list, and add the regular expression and the correspondence between the category and the time format string to the Map; Step 03: Read the extended time format conversion rules and process them one by one; Step 04: Determine if a regular expression already exists in the Map. If it does, proceed to step 05; otherwise, proceed to step 06. Step 05: Replace the category and time format characters corresponding to the regular expressions in Map, then proceed to step 07; Step 06: Add the regular expression to the list, and add the regular expression and the correspondence between the category and the time format string to the Map; Step 07: Construct a rule database using lists.

9. The adaptive time format conversion method based on HyperSCAN according to claim 2, characterized in that: If the preferred time format converter is successfully invoked for format conversion, proceed to step 5; otherwise, proceed to step 1.

Images