Systems and methods for using regexp to process input data to generate corresponding output data.

Abstract

This invention provides a system 100 including computing hardware 102 for processing input data 106 to generate corresponding output data. The system 100 includes the computing hardware 102, a data storage 104, a main search device 108, and a secondary search device 110. The system 100 stores the input data 106 in the data storage 104. The system 100 generates a group of generalized text regexps that match all regexp groups to be searched in the input data 106. The system 100 uses the main search device 108 to search the input data 106 using the generalized text regexp groups to find one or more matches of the regexps. The system 100 uses the secondary search device 110 to analyze the one or more matches of the regexps found in the input data 106 to generate the output data.

Description

Technical Field

[0001] This invention generally relates to a data processing system including computing hardware for processing input data using regexp to generate corresponding output data. The invention also relates to a method for using the system including the computing hardware to process the input data to generate the corresponding output data. Background Technology

[0002] In recent years, as public awareness of digital privacy has increased, some countries have enacted privacy regulations to protect their citizens' information or data in the digital space. For example, some countries employ privacy regulations such as the General Data Protection Regulation (GDPR) or the Personal Information Protection Law (PIPL) to protect their citizens' information in the digital space. Digital privacy awareness involves understanding how personal data is collected, used, and shared online, as well as understanding the risks and impacts of sharing personal data. Privacy regulations define the boundaries and constraints on how personal data is collected, processed, and stored in the digital space to strengthen personal privacy protection and data security. Privacy regulations can also have a significant impact on how businesses handle such personal data. Businesses and organizations that fail to comply with privacy regulations may face substantial fines. Furthermore, businesses may be unable to identify sensitive data according to privacy regulations, especially when sensitive data is in an unstructured format. Sensitive data may be personally identifiable information, known as "PII." PII is information that can be used to identify an individual in the digital space.

[0003] Pattern matching is a computational technique used in various fields such as data analysis and linguistics. It identifies specific patterns within large datasets or text sets. These specific patterns can be identified using regular expressions (hereinafter referred to as "regexp"). Regexp can be used to process given text to define search patterns to be identified within that text. Regexp consists of combinations of characters and symbols that create search patterns, allowing various operations to be performed on the text. These operations can be pattern searching, pattern matching, pattern replacement, or pattern validation based on specific conditions.

[0004] Pattern matching mechanisms can be used to detect personally identifiable information (PII) in the digital space. They detect PII by determining whether user data in the digital space contains sensitive data. Sensitive data is determined by checking whether the data matches a pattern in a predefined regular expression (regexp). The pattern in the predefined regexp can be a sensitive pattern. However, this pattern matching mechanism becomes inefficient when there are many predefined regexps to search. For example, when there are hundreds or thousands of predefined regexps, pattern matching inefficiency may occur.

[0005] Therefore, it is necessary to address the technical problems / deficiencies in processing input data to generate corresponding output data, in order to reduce the complexity of pattern matching of multiple regular expressions (regexp). Summary of the Invention

[0006] The object of this invention is to provide a system including computing hardware for processing input data to generate corresponding output data. This invention also relates to a method for using the system including the computing hardware to process the input data to generate the corresponding output data.

[0007] This objective is achieved through the features of the independent claim. Other implementations are apparent in the dependent claims, the specification, and the drawings.

[0008] According to a first aspect, a system including computing hardware is provided for processing input data to generate corresponding output data. The system is used to store the input data in a data memory of the system. The system is used to generate a group of generalized text regexps that match all groups of regular expressions (regexps) to be searched in the input data. The system is used to use a primary search device to apply the group of generalized text regexps to search the input data to find one or more matches of the regexps in the input data. The system is used to use a secondary search device to analyze the one or more matches of the regexps found in the input data to generate the output data. The output data describes the location of the regexps found in the input data.

[0009] The system combines all regular expressions (regexp) into a generalized text regexp group. (i) Efficiently match all regexp groups in the input data, rather than matching each regexp within the generalized text regexp group individually; (ii) It can detect sensitive data without explicitly specifying the exact data type; (iii) It reduces the complexity of pattern matching for multiple regexps.

[0010] The system constructs a hierarchical structure for single generalized text regexps, enabling recursive matching to locate accurate regexps in a shorter time, especially when processing a large number of regexps. This is because the hierarchical structure reduces the complexity of pattern matching. Before storing sensitive data, the system detects sensitive data to avoid regulatory violations and improves pattern matching across multiple regexps, particularly in scenarios requiring low-latency detection, such as within cloud storage systems like Object Storage Service (OBS) and Shared File Storage (SFS). The system generates smaller groups of generalized text regexps to significantly reduce the runtime of pattern matching.

[0011] Optionally, the generalized text regexp group is generated using an artificial intelligence (AI) system trained on a corpus representing the input data. The AI ​​system may optionally be implemented as a neural network.

[0012] Optionally, the auxiliary search device is used to use one or more versions of the generalized text regexp group, which are applicable to a specific type of regexp present in the input data.

[0013] Optionally, the generalized text regexp group is used to detect sensitive data in the input data without specifying the actual type of the sensitive data.

[0014] Optionally, the generalized text regexp group is used to define at least one of the following: one or more alphabetic characters, one or more numeric characters, and one or more non-alphabetic characters.

[0015] Optionally, the generalized text regexp group is used to define the range of the number of alphanumeric characters to appear.

[0016] Optionally, the primary search device and the secondary search device are used to share computational search tasks between them to optimize the rate at which the input data is processed within the system.

[0017] According to a second aspect, a method is provided for using a system including computing hardware to process input data to generate corresponding output data. The method includes configuring the system to store the input data in a data memory of the system. The method includes configuring the system to generate a group of generalized text regexps that match all groups of regular expressions (regexps) to be searched in the input data. The method includes configuring the system to use a primary search device to apply the group of generalized text regexps to search the input data to find one or more matches of the regexps in the input data. The method includes configuring the system to use a secondary search device to analyze the one or more matches of the regexps found in the input data to generate the output data. The output data describes the location of the regexps found in the input data.

[0018] This method combines all regular expressions (regexp) into a generalized text regexp group, which is: (i) Efficiently match all regexp groups in the input data, rather than matching each regexp within the generalized text regexp group individually; (ii) It can detect sensitive data without explicitly specifying the exact data type; (iii) It reduces the complexity of pattern matching for multiple regexps.

[0019] This method constructs a hierarchical structure for generalized text registrar's (regexp) sets, enabling recursive matching to locate precise registrar's in a shorter time, especially when processing a large number of registrar's, because the hierarchical structure reduces the complexity of pattern matching. For example, the use of this hierarchical structure allows computing hardware, including arrays of processes, to run simultaneously to process input data to find registrar's matches. Before storing sensitive data, the method detects sensitive data to avoid regulatory violations and improves pattern matching of multiple registrar's, especially in scenarios requiring low-latency detection, such as in cloud storage systems like Object Storage Service (OBS) and Shared File Storage (SFS). This method generates a smaller set of generalized text registrar's, significantly reducing the runtime of pattern matching.

[0020] Optionally, the generalized text regexp group is generated using an artificial intelligence (AI) system trained on a corpus representing the input data.

[0021] Optionally, the auxiliary search device is used to use one or more versions of the generalized text regexp group, which are applicable to a specific type of regexp present in the input data.

[0022] Optionally, the generalized text regexp group is used to detect sensitive data in the input data without specifying the actual type of the sensitive data.

[0023] Optionally, the generalized text regexp group is used to define at least one of the following: one or more alphabetic characters, one or more numeric characters, and one or more non-alphabetic characters.

[0024] Optionally, the generalized text regexp group is used to define the range of the number of alphanumeric characters to appear.

[0025] Optionally, the primary search device and the secondary search device are used to share computational search tasks between them to optimize the rate at which the input data is processed within the system.

[0026] According to a third aspect, a software product including computer-executable instructions is provided, which can be executed on computing hardware to implement all steps of the method using the system.

[0027] Therefore, unlike existing solutions, the methods and systems used to process input data to generate corresponding output data: (i) are able to detect sensitive data without explicitly specifying the exact data type; and (ii) generate a reduced-size generalized text regexp group to significantly reduce the runtime of pattern matching.

[0028] These and other aspects of the invention will become apparent from the implementation described below. Attached Figure Description

[0029] The various implementations of the present invention will now be described by way of example and in conjunction with the accompanying drawings, in which: Figure 1 It is a block diagram illustrating a system including an implementation according to the present invention; Figure 2A and Figure 2BIt is a flowchart illustrating the steps of a method for processing input data to generate corresponding output data according to an implementation of the present invention; Figure 3 It is a description of a computer system (e.g., computing hardware) in which various architectures and functions of various previously implemented methods can be implemented. Detailed Implementation

[0030] This invention provides a system including computing hardware for processing input data to generate corresponding output data. The invention also relates to a method for using a system including computing hardware to process input data to generate corresponding output data.

[0031] To enable those skilled in the art to more easily understand the present invention, the following implementation of the present invention is described in conjunction with the accompanying drawings.

[0032] The terms “first,” “second,” “third,” and “fourth” (if any) used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular sequence or order. It should be understood that such terms are interchangeable where appropriate, for example, to allow implementations of the invention described herein to be implemented in sequences other than those shown or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to the explicitly listed steps or units, but may include other steps or units not explicitly listed or inherent to the aforementioned process, method, product, or apparatus.

[0033] Figure 1 This is a block diagram illustrating a system 100 according to an implementation of the present invention. System 100 includes computing hardware 102, data storage 104, a main search device 108, and a secondary search device 110. The computing hardware 102 processes input data to generate corresponding output data. System 100 stores input data 106 in the data storage 104 of system 100. System 100 generates a group of generalized text regexps that match all regexp groups to be searched in input data 106. System 100 uses the main search device 108 to search the input data 106 using the generalized text regexp groups to find one or more matches of the regexps. System 100 uses the secondary search device 110 to analyze one or more matches of the regexps found in input data 106 to generate output data. The output data describes the location of the regexps found in input data 106.

[0034] System 100 combines all regular expressions (regexp) into a generalized text regexp group, which is: (i) Efficiently match all regexp groups in the input data 106, instead of matching each regexp within the generalized text regexp group individually; (ii) It can detect sensitive data without explicitly specifying the exact data type; (iii) It reduces the complexity of pattern matching for multiple regexps.

[0035] System 100 constructs a hierarchical structure of generalized text registrar groups, enabling recursive matching to locate precise registrars in a shorter time, especially when processing a large number of registrars, because the hierarchical structure reduces the complexity of pattern matching. Before storing sensitive data, System 100 detects sensitive data to avoid regulatory violations and improves pattern matching of multiple registrars, especially in scenarios requiring low-latency detection, such as in cloud storage systems like Object Storage Service (OBS) and Shared File Storage (SFS). System 100 generates smaller sets of generalized text registrar groups to significantly reduce the runtime of pattern matching.

[0036] Optionally, the generalized text regexp sets are generated using an artificial intelligence (AI) system trained on a corpus representing the input data 106. The corpus is generated using a dynamic algorithm. The corpus includes a set of generalized regular expressions (regexps) and corresponding generalized data. Optionally, this set of generalized regexps is manually generated by the user; or, it is generated by the AI ​​computing engine. The AI ​​system searches within the generalized regexp set for generalized text regexp sets that include all regexp sets. These regexp sets can be manually entered by the user. If the AI ​​system cannot find a generalized text regexp set within the generalized regexp set, it generates a new set of generalized regexps or modifies the existing set to match all regexp sets.

[0037] Optionally, the user defines a character subset in system 100. For example, the character subset includes: The character subset A is [az], which includes all lowercase letters from a to z.

[0038] The character subset B is [AZ], which includes all uppercase letters from A to Z.

[0039] The character subset C is [0-9], which includes all digits from 0 to 9.

[0040] The character subset D is [ ], which represents a space as a subset.

[0041] These regexp groups are divided into one or more regexp subsets. System 100 initializes two empty sets to generate a generalized text regexp group that matches all regexp groups in the input data 106. For example, the first empty set can be called the abstract set, and the second empty set can be called the generalized set. System 100 iterates over each regexp subset in the one or more regexp subsets to generate a general candidate. The general candidate is generated by replacing each character in the regexp subset with the corresponding character subset in the character subset. For example, if the user-provided regexp subset is “[az]{2}[0-4][ ][AZ]{2,4}”, then the generated general candidate is 'A{2}CDB{2,4}'. System 100 omits or removes one or more matches from the general candidate to generate an abstract element. “One or more matches” can be {2} and {2,4}. The abstract element is “ACDB”. System 100 checks whether the abstract element is contained in the abstract set. If the abstract element is contained in the abstract set, then System 100 checks whether the abstract element is contained in the generalized set. If an abstract element exists in a generalized set, the abstract element is represented as a generalized text regexp group, meaning that the generalized text regexp group matches all regexps in a subset of regexps. All regexps in each subset of regexps are represented by a generalized text regexp group. System 100 checks whether "one or more matches" in the "general candidate" are included in "one or more matches" in the generalized text regexp group to locate all regexps in one or more subsets of regexps. For example, if the generalized candidate is "A{2}CDB{2,4}" and the "abstract element" or generalized text regexp group is "ACDB{3,4}", this means that "one or more matches" in the "general candidate" are already included in "one or more matches" in the generalized text regexp group.

[0042] If one or more matches in the "General Candidates" do not exist in one or more matches in the "Generalized Text Regexp Group", then System 100 modifies one or more matches in the "Generalized Text Regexp Group" to include one or more matches from the "General Candidates". For example, if the generalized candidate is 'A{2-3}CD' and the generalized text regexp group is 'A{1-4}CD{2}', then the new generalized text regexp group becomes 'A{1-4}CD{1-2}'.

[0043] If an abstract element does not exist in the abstract set, system 100 adds the abstract element to the abstract set. Additionally, system 100 adds generalized candidates to the generalized set.

[0044] Optionally, the auxiliary search device 110 is used to use one or more versions of a generalized text regexp group, which are suitable for a specific type of regexp present in the input data 106.

[0045] Optionally, generalized text regexp groups are used to detect sensitive data present in the input data 106 without specifying the actual type of the sensitive data. System 100 improves pattern matching by organizing the generalized text regexp groups of each subset of regexps into a hierarchical structure (e.g., a tree structure or a B+ tree) to describe the location of the regexps found in the input data 106.

[0046] Optionally, the generalized text regexp group is used to define at least one of the following: one or more alphabetic characters, one or more numeric characters, and one or more non-alphabetic characters.

[0047] Optionally, the generalized text regexp group is used to define the range of the number of alphanumeric characters to appear.

[0048] Optionally, the primary search device 108 and the secondary search device 110 are used to share computational search tasks between them to optimize the rate at which input data 106 is processed within the system 100.

[0049] Figure 2A and Figure 2BThis is a flowchart illustrating the steps of a method according to an implementation of the present invention, using a system (e.g., the aforementioned system 100) to process input data to generate corresponding output data. In step 202, the method includes configuring the system to store input data in the system's data memory. In step 204, the method includes configuring the system to generate a group of generalized text regexps that match all regexp groups to be searched in the input data. In step 206, the method includes configuring the system to use a primary search device to apply the generalized text regexp groups to search the input data to find one or more matches. In step 208, the method includes configuring the system to use a secondary search device to analyze one or more matches of the regexps found in the input data to generate output data. The output data describes the location of the regexps found in the input data.

[0050] This method combines all regular expressions (regexp) into a generalized text regexp group, which is: (i) Efficiently match all regexp groups in the input data, rather than matching each regexp within the generalized text regexp group individually; (ii) It can detect sensitive data without explicitly specifying the exact data type; (iii) It reduces the complexity of pattern matching for multiple regexps.

[0051] This method constructs a hierarchical structure of generalized text registrar groups, enabling recursive matching to locate precise registrars in a shorter time, especially when processing a large number of registrars, because the hierarchical structure reduces the complexity of pattern matching. Before storing sensitive data, the method detects sensitive data to avoid regulatory violations and improves pattern matching across multiple registrars, particularly in scenarios requiring low-latency detection, such as in cloud storage systems like Object Storage Service (OBS) and Shared File Storage (SFS). This method generates a smaller set of generalized text registrar groups, significantly reducing the runtime of pattern matching.

[0052] Optionally, the generalized text regexp group is generated using an artificial intelligence (AI) system trained on a corpus of data representing the input data.

[0053] Optionally, the auxiliary search device is used to employ one or more versions of a generalized text regexp group, which are applicable to a specific type of regexp present in the input data.

[0054] Optionally, the generalized text regexp group is used to detect sensitive data in the input data without specifying the actual type of the sensitive data.

[0055] Optionally, the generalized text regexp group is used to define at least one of the following: one or more alphabetic characters, one or more numeric characters, and one or more non-alphabetic characters.

[0056] Optionally, the generalized text regexp group is used to define the range of the number of alphanumeric characters to appear.

[0057] Optionally, the primary search unit and the secondary search unit are used to share computational search tasks between them to optimize the rate of processing input data within the system.

[0058] Figure 3 This is an illustration of a computer system 300 (e.g., computing hardware, data storage, primary search unit, and secondary search unit) in which various architectures and functions of various prior implementations can be implemented; the computer system 300 is advantageously used to implement the aforementioned system 100. As shown, the computer system 300 includes at least one processor 304 connected to a bus 302, wherein the computer system 300 can be implemented using any suitable protocol, such as peripheral component interconnect (PCI), PCI-Express, accelerated graphics port (AGP), Hyper Transport, or any other bus or one or more point-to-point communication protocols. The computer system 300 also includes a memory 306.

[0059] The control logic (software) and data are stored in memory 306, which may be random-access memory (RAM). In this invention, a single semiconductor platform can refer to a single, individual semiconductor-based integrated circuit or chip. It should be noted that the term "single semiconductor platform" can also refer to a multi-chip module with enhanced connectivity. Multi-chip modules offer enhanced connectivity, simulating on-chip modules and on-chip operations, resulting in significant improvements compared to implementations using traditional central processing units (CPUs) and buses. Of course, various modules can also be installed individually or combined on different semiconductor platforms according to user needs.

[0060] Computer system 300 also includes secondary storage device 310. Secondary storage device 310 includes hard disk drives and removable storage drives, such as floppy disk drives, magnetic tape drives, compact disk drives, digital versatile disk (DVD) drives, recording devices, and universal serial bus (USB) flash memory. The removable storage drive drives reading from and / or writing to the removable storage unit in a well-known manner.

[0061] Computer programs or computer control logic algorithms may be stored in at least one of memory 306 and secondary storage device 310. When executed, these computer programs cause computer system 300 to perform the various functions described above. Memory 306, secondary storage device 310, and any other storage device are possible examples of computer-readable media.

[0062] In one implementation, the architecture and functionality described in the preceding figures can be implemented within the context of processor 304, a graphics processor coupled to communication interface 312, an integrated circuit (not shown) capable of simultaneously possessing at least some of the functions of processor 304 and graphics processor, a chipset (i.e., a set of integrated circuits designed to function and be sold as units performing related functions, etc.).

[0063] Furthermore, the architecture and functions described in the previous figures can also be implemented in the context of general-purpose computer systems, circuit board systems, game console systems dedicated to entertainment, and special-purpose systems. For example, computer system 300 can be a desktop computer, a laptop computer, a server, a workstation, a game console, or an embedded system.

[0064] In addition, the computer system 300 can also be various other devices, including but not limited to personal digital assistant (PDA) devices, mobile phone devices, smartphones, televisions, etc. Furthermore, although not shown in the figure, the computer system 300 can be coupled to a network (e.g., telecommunications networks, local area networks (LANs), wireless networks, wide area networks (WANs) such as the Internet, peer-to-peer networks, cable networks, etc.) to communicate via I / O interface 308.

[0065] It should be understood that the arrangement of components shown in the described figures is exemplary, and other arrangements are possible. It should also be understood that the various system components (and elements) defined by the claims, described below, and shown in the various block diagrams represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and elements) may be implemented wholly or partially by at least some of the components illustrated in the arrangements shown in the described figures.

[0066] In addition, while at least one of these components is implemented at least partially as an electronic hardware component and thus constitutes a machine, the other components may be implemented in software, which, when included in the execution environment, constitutes a machine, hardware, or a combination of software and hardware.

[0067] While the invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations may be made herein without departing from the scope and spirit of the invention as defined by the appended claims.

Claims

1. A system (100) including computing hardware (102), characterized in that, The computing hardware (102) is used to process input data (106) to generate corresponding output data, and the system (100) is used to: (i) The input data (106) is stored in the data storage (104) of the system (100); (ii) Generate a generalized text regexp group that matches all regexp groups to be searched in the input data (106); (iii) Use the main search device (108) to search the input data (106) using the generalized text regexp group to find one or more matches of regexp in the input data (106); (iv) Using an auxiliary search device (110) to analyze the one or more matches of the regexp found in the input data (106) to generate the output data, wherein the output data describes the location of the regexp found in the input data (106).

2. The system (100) according to claim 1, characterized in that, The generalized text regexp group is generated using an artificial intelligence system trained on a corpus representing the input data (106).

3. The system (100) according to claim 1 or 2, characterized in that, The auxiliary search device (110) is used to use one or more versions of the generalized text regexp group, which are applicable to a specific type of regexp present in the input data (106).

4. The system (100) according to claim 1, 2 or 3, characterized in that, The generalized text regexp group is used to detect sensitive data in the input data (106) without specifying the actual type of the sensitive data.

5. The system (100) according to any one of claims 1 to 4, characterized in that, The generalized text regexp group is used to define at least one of the following: one or more alphabetic characters, one or more numeric characters, and one or more non-alphabetic characters.

6. The system (100) according to claim 5, characterized in that, The generalized text regexp group is used to define the range of the number of alphanumeric characters to appear.

7. The system (100) according to any one of the preceding claims, characterized in that, The primary search device (108) and the secondary search device (110) are used to share computational search tasks between them to optimize the rate at which the input data (106) is processed within the system (100).

8. A method for using a system (100) including computing hardware (102) to process input data (106) to generate corresponding output data, characterized in that, The method includes: (i) Configure the system (100) to store the input data (106) in the data storage (104) of the system (100); (ii) Configure the system (100) to generate generalized text regexp groups that match all regexp groups to be searched in the input data (106); (iii) Configure the system (100) to use the main search device (108) to apply the generalized text regexp group to search the input data (106) to find one or more matches of regexp in the input data (106); (iv) Configure the system (100) to use an auxiliary search device (110) to analyze the one or more matches of the regexp found in the input data (106) to generate the output data, wherein the output data describes the location of the regexp found in the input data (106).

9. The method according to claim 8, characterized in that, The generalized text regexp group is generated using an artificial intelligence system trained on a corpus representing the input data (106).

10. The method according to claim 8 or 9, characterized in that, The auxiliary search device (110) is used to use one or more versions of the generalized text regexp group, which are applicable to a specific type of regexp present in the input data (106).

11. The method according to claim 8, 9 or 10, characterized in that, The generalized text regexp group is used to detect sensitive data in the input data (106) without specifying the actual type of the sensitive data.

12. The method according to any one of claims 8 to 11, characterized in that, The generalized text regexp group is used to define at least one of the following: one or more alphabetic characters, one or more numeric characters, and one or more non-alphabetic characters.

13. The method according to claim 12, characterized in that, The generalized text regexp group is used to define the range of the number of alphanumeric characters to appear.

14. The method according to any one of claims 8 to 13, characterized in that, The primary search device (108) and the secondary search device (110) are used to share computational search tasks between them to optimize the rate at which the input data (106) is processed within the system (100).

15. A software product comprising computer-executable instructions, characterized in that, The computer-executable instructions can be executed on computing hardware (102) to implement the method of any one of claims 8 to 14 using the system (100) of any one of claims 1 to 7.

Images