Data processing method, device and equipment

Abstract

The embodiments of this specification disclose a data processing method, apparatus, and device. The method includes: obtaining data to be processed; generalizing the data to be processed to obtain generalized data; performing template extraction on the generalized data to obtain a substructure of the generalized data, wherein the substructure of the generalized data satisfies a preset data pattern; and performing template refinement on the substructure of the generalized data to obtain a data pattern of the data to be processed.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data processing method, device, and equipment. Background Art

[0002] Automatic pattern recognition or automatic pattern discovery means that a machine can automatically classify specific samples into a certain pattern without human intervention. Existing automatic pattern discovery technology mainly targets structured data and automatically specifies the pattern of a string. However, because existing automatic pattern discovery technology targets structured data and assumes that structured data does not contain nested substructures, and treats structured data as the smallest atomic unit, it is obvious that existing technology cannot automatically discover patterns for key-value type data. In addition, the time consumption of existing automatic pattern discovery technology usually increases exponentially with the increase of data length. Therefore, existing technology cannot automatically discover patterns for long text data.

[0003] Based on this, a new method is needed that can discover the internal structure of key-value type data and can solve the problem of discovering the internal structure of long text data. Summary of the Invention

[0004] The embodiments of this specification provide a data processing method, apparatus, and device for solving the following technical problems: Existing automatic pattern discovery techniques target structured data, assuming that structured data does not contain nested substructures and treating structured data as the smallest atomic unit. Clearly, existing techniques are unable to perform automatic pattern discovery on key-value data. Furthermore, the time consumed by existing automatic pattern discovery techniques typically increases exponentially with increasing data length, making them incapable of automatically discovering patterns on long text data.

[0005] To solve the above technical problems, the embodiments of this specification are implemented as follows:

[0006] The present invention provides a data processing method, including:

[0007] Get the data to be processed;

[0008] Generalizing the data to be processed to obtain generalized data;

[0009] Performing template extraction on the generalized data to obtain a substructure of the generalized data, where the substructure of the generalized data satisfies a preset data pattern;

[0010] Template refinement is performed on the substructure of the generalized data to obtain a data pattern of the data to be processed.

[0011] The present invention also provides a data processing device, including:

[0012] Get module to get data to be processed;

[0013] A generalization module generalizes the data to be processed to obtain generalized data;

[0014] a template extraction module, performing template extraction on the generalized data to obtain a substructure of the generalized data, wherein the substructure of the generalized data satisfies a preset data pattern;

[0015] The template refinement module performs template refinement on the substructure of the generalized data to obtain the data pattern of the data to be processed.

[0016] An embodiment of this specification further provides an electronic device, including:

[0017] at least one processor; and,

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0020] Get the data to be processed;

[0021] Generalizing the data to be processed to obtain generalized data;

[0022] Performing template extraction on the generalized data to obtain a substructure of the generalized data, where the substructure of the generalized data satisfies a preset data pattern;

[0023] Template refinement is performed on the substructure of the generalized data to obtain a data pattern of the data to be processed.

[0024] The embodiments of this specification obtain data to be processed; generalize the data to be processed to obtain generalized data; perform template extraction on the generalized data to obtain a substructure of the generalized data, wherein the substructure of the generalized data satisfies a preset data pattern; and perform template refinement on the substructure of the generalized data to obtain a data pattern of the data to be processed. This can achieve rapid division of data patterns of key-value type data, solve the problem of long texts, automatically generate data with high accuracy and high recall, and save memory and computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1 A schematic diagram of a data processing method provided in an embodiment of this specification;

[0027] Figure 2 A flow chart of another data processing method provided in an embodiment of this specification;

[0028] Figure 3 A data processing method framework diagram provided in an embodiment of this specification;

[0029] Figure 4 A schematic diagram of a data processing device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0030] In order to help those skilled in the art better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0031] Regular expressions are a method for concisely specifying string patterns. These patterns can then be used by specialized engines to extract matching strings from data streams. Regular expressions are a mature technology widely applicable in various application areas, including text processing, and are a commonly used tool due to their expressiveness and flexibility. Since many real-world entities follow a basic grammatical pattern that can be described using regular expressions, a wide range of entity extraction tasks can be addressed using regular expressions. However, constructing regular expressions that guarantee high precision and recall for a given extraction task is tedious, difficult, and requires specific technical skills.

[0032] Existing automatic pattern discovery technologies mainly include PotterWheel, FIDEX, and Xsystem. PotterWheel is an open-source method that automatically generates patterns based on data characteristics. This method generates regular expressions that meet the data requirements and then sorts them using a minimum length criterion. FIDEX is a method that generates regular expressions from examples and uses them to filter data from tables. This method uses enumeration to generate a graph structure and then performs pruning on the graph structure. Xsystem is a method for extracting semantic patterns from databases. This method uses a branching and merging method to generate regular expressions.

[0033] Based on this, the embodiments of this specification provide a new method. Figure 1 A schematic diagram of a data processing method provided in an embodiment of this specification is shown as follows: Figure 1 As shown, the embodiment of this specification provides a data processing method, which specifically includes the following steps:

[0034] Step S101: Obtain data to be processed.

[0035] In the embodiments of this specification, the data to be processed is key-value type data. Of course, the data to be processed can also be data that can generate key-value type data. If the data to be processed is data that can generate key-value type data, it is necessary to first process the data to be processed to generate key-value type data.

[0036] The source of the data to be processed may be data from the blockchain or other data. The source of the data to be processed does not constitute a limitation on this application.

[0037] Key-value data is stored as key-value pairs. Key-value stores are a form of data storage that is easy to find and retrieve efficiently, and can store a variety of different types of data. Therefore, they are widely used in data storage.

[0038] It should be noted that, in the embodiments of this specification, the data to be processed may be long text data.

[0039] Step S103: generalize the data to be processed to obtain generalized data.

[0040] In the embodiment of this specification, generalizing the data to be processed to obtain generalized data specifically includes:

[0041] Taking each character in the data to be processed as a leaf node of the generalization tree;

[0042] Mapping the leaf nodes to upper-layer branch nodes to generalize the characters in the data to be processed and obtain generalized characters;

[0043] Based on the generalized characters of the data to be processed, a greedy algorithm is adopted to select the optimal generalization level to obtain the generalized data.

[0044] Generalization is to absorb the common points of two or more classes and combine them into a generalized superclass so that subclasses can share the properties, relationships, and methods of the superclass.

[0045] In the embodiment of this specification, the method of using a greedy algorithm to determine the generalization level and obtain the generalization data specifically includes:

[0046] The semantic distance is used as the optimization measure of the greedy algorithm, and the optimal generalization level is selected to obtain the generalization data.

[0047] A greedy algorithm (also known as a greedy algorithm) means that when solving a problem, the best choice is always made at the moment. In other words, without considering the overall optimal solution, the algorithm obtains a local optimal solution in a certain sense. Semantic distance is a metric. In the embodiments of this specification, semantic distance is used to measure the distance between leaf nodes in the generalization tree. The specific processes of semantic distance, greedy algorithm, and generalization can all be implemented using the processes of the existing technology and will not be repeated here.

[0048] It should be noted that the purpose of using the generalization tree method to generate generalized data is to generate regular expressions. Compared with the traditional enumeration method for generating regular expressions, this method can significantly reduce the consumption of computing memory and time.

[0049] Step S105: performing template extraction on the generalized data to obtain a substructure of the generalized data, where the substructure of the generalized data satisfies a preset data pattern.

[0050] In the embodiment of this specification, the template extraction includes cyclic structure division and vertical structure division, and the template extraction of the generalized data to obtain the substructure of the generalized data specifically includes:

[0051] Performing cyclic structure division based on special characters and spaces in the generalized data, and / or performing vertical structure division based on each data included in the generalized data, to obtain a template candidate set;

[0052] Based on the minimum distance principle, a substructure is selected from the candidate template candidate set as the substructure of the generalized data.

[0053] In the embodiment of the present specification, the step of performing cyclic structure division based on special characters and spaces in the generalized data and / or performing vertical structure division based on each data included in the generalized data to obtain a template candidate set specifically includes:

[0054] Performing cyclic structure division based on the special character set spaces in the generalized data, calculating the cyclic distance between each data in the generalized data, and when the cyclic distance is less than a preset first similarity threshold, adding the substructure obtained by the cyclic structure division as a candidate structure to the template candidate set;

[0055] and / or

[0056] Perform vertical structural division based on each data included in the generalized data, calculate the vertical distance between each data in the generalized data, and when the vertical distance is less than a preset second similarity threshold, add the substructure obtained by the vertical structural division as a candidate structure to the template candidate set.

[0057] In order to further understand the cyclic structure division and vertical structure division provided by the embodiments of this specification, they will be described below in conjunction with specific embodiments.

[0058] For loop structure division, given a data l1 = 2012, 2013, 2014, loop structure segmentation is to divide the data within a piece of data according to some special characters or spaces, l1 can be divided into three parts

[0059]

[0060] Similarly, given a series of data We use H(l n ) represents data l n All possible cyclic structure partitions of data can be expressed as

[0061]

[0062] We also define the cycle distance, which can be expressed as

[0063]

[0064] The algorithm enumerates all possible cyclic structure partitions of the data and calculates the cyclic distance between them. If the cyclic distance is less than a preset first similarity threshold, the cyclic structure is considered to be a true substructure of the data to be processed and is added to the template candidate set as a candidate structure. The specific value of the first similarity threshold is generally determined by the business scenario.

[0065] For vertical structure division, given a data l1 = Jan, 2013, l2 = Nov, 2014. Vertical structure division is performed between data, l1 and l2 can be divided into two parts

[0066] part1={Jan,Nov},part2={2013,2014}

[0067] Similarly, given a series of data We use H(v n ) represents data l n All possible vertical structure partitions of data can be expressed as

[0068]

[0069] The vertical distance can be expressed as

[0070]

[0071] The algorithm enumerates all possible vertical structural divisions of the data and calculates the vertical distance between them. When the vertical distance is less than a preset first similarity threshold, the vertical structure is considered to be a true substructure of the data to be processed and is added to the template candidate set as a candidate structure. The specific value of the second similarity threshold is generally determined by the business scenario. It is important to note that the specific values ​​of the first and second similarity thresholds may be the same or different.

[0072] In the embodiments of this specification, the preset data pattern is a regular expression structure. If the substructure of the generalized data obtained in the above steps does not satisfy the regular expression structure, the generalization and template extraction loop is continued on the data to be processed until the substructure of the generalized data satisfies the preset data pattern. Further generalization of the data to be processed is achieved by changing the generalization level.

[0073] Step S107: performing template refinement on the substructure of the generalized data to obtain a data pattern of the data to be processed.

[0074] In order to ensure high accuracy and recall rate, the template needs to be further refined.

[0075] In the embodiment of this specification, the template-refining of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes:

[0076] The intermediate nodes in the substructure of the generalized data are refined into leaf nodes to obtain the data pattern of the data to be processed.

[0077] In the embodiment of this specification, the template-refining of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes:

[0078] Based on the respective parts of the substructure of the generalized data, determining frequent words of the respective parts of the substructure of the generalized data using a statistical method;

[0079] The frequent words and the infrequent words form an alphabet of a regular expression;

[0080] The substructure of the generalized data is template-refined based on the alphabet to obtain a data pattern of the data to be processed.

[0081] It should be noted that when using a statistical method to determine the frequent words in each part of the substructure of the generalized data based on the various parts of the substructure of the generalized data, the statistical method is used to calculate the relative frequency of occurrence of each word, and the frequency of occurrence generally varies from one level to another. It is generally considered that a word with a frequency of occurrence of more than 10 times is defined as a frequent word.

[0082] After template refinement, character-level features can be captured.

[0083] Since the data to be processed is dynamic, the size of the data set usually increases over time. In order to save memory and computing resources, in the embodiments of this specification, after obtaining the data to be processed, the data is further processed by an incremental calculation method. Figure 2 A flow chart of another data processing method provided in an embodiment of this specification, the method comprising:

[0084] Step S201: Obtain data to be processed.

[0085] Step S203: performing incremental calculation on the data to be processed and the data to be processed at the previous time point, and determining the data to be processed belonging to the current time point in the data to be processed.

[0086] In the embodiment of this specification, performing incremental calculation on the data to be processed and the data to be processed at the previous time point to determine the data to be processed belonging to the current time point in the data to be processed specifically includes:

[0087] The data constraint status of the to-be-processed data at the previous time point is defined as the data submission status of the to-be-processed data, and the to-be-processed data belonging to the current time point in the to-be-processed data is determined.

[0088] Incremental computing is stateful. In batch computing, each output result is specific to the data scanned in the current batch, and the computation is idempotent. In incremental computing, each batch's result is calculated using the current batch's data and the previous batch's results, i.e., newValue = function(currentBatchValue, oldValue). The newValue calculated for the current batch is then used as the oldValue in the calculation of the next batch of data. Incremental computing eliminates the need to recalculate already calculated data, significantly saving memory and computing resources.

[0089] Step S205: generalize the data to be processed to obtain generalized data.

[0090] Step S207: performing template extraction on the generalized data to obtain a substructure of the generalized data.

[0091] Step S209: If the substructure of the generalized data does not satisfy the preset data pattern, the data to be processed is subjected to a loop of generalization and template extraction according to steps S205 and S207 until the substructure of the generalized data satisfies the preset data pattern.

[0092] Step S211: performing template refinement on the substructure of the generalized data to obtain the data pattern of the data to be processed.

[0093] In order to further understand the data processing method provided in the embodiments of this specification, the embodiments of this specification also provide a data processing method framework diagram. Figure 3 This is a data processing method framework diagram provided in the embodiment of this specification. Figure 3 As shown in the figure, after data input, data generalization and template extraction are performed, and then the appropriate structure is determined. If the preset data pattern does not exist, data generalization and template extraction are continued. If the preset data pattern is present, template refinement is further performed before output. To save memory and computing resources, after data input, incremental calculation can be performed first, followed by data generalization and other operations.

[0094] The method provided in the embodiments of this specification can be used to quickly divide the data patterns of key-value type data, solve the problem of long text, automatically generate data with high accuracy and high recall rate, and save memory and computing resources.

[0095] The above content describes in detail a data processing method. Correspondingly, this specification also provides a data processing device, such as Figure 4 shown. Figure 4 A schematic diagram of a data processing device provided in an embodiment of this specification, the device comprising:

[0096] Acquisition module 401, acquires data to be processed;

[0097] A generalization module 403 generalizes the data to be processed to obtain generalized data;

[0098] The template extraction module 405 performs template extraction on the generalized data to obtain a substructure of the generalized data, where the substructure of the generalized data satisfies a preset data pattern;

[0099] The template refinement module 407 performs template refinement on the substructure of the generalized data to obtain a data pattern of the data to be processed.

[0100] Furthermore, the obtaining of data to be processed further includes:

[0101] The incremental calculation module 409 performs incremental calculation on the data to be processed and the data to be processed at the previous time point to determine the data to be processed belonging to the current time point in the data to be processed.

[0102] Furthermore, performing incremental calculation on the data to be processed and the data to be processed at the previous time point to determine the data to be processed belonging to the current time point in the data to be processed specifically includes:

[0103] The data constraint status of the to-be-processed data at the previous time point is defined as the data submission status of the to-be-processed data, and the to-be-processed data belonging to the current time point in the to-be-processed data is determined.

[0104] Furthermore, generalizing the data to be processed to obtain generalized data specifically includes:

[0105] Taking each character in the data to be processed as a leaf node of the generalization tree;

[0106] Mapping the leaf nodes to upper-layer branch nodes to generalize the characters in the data to be processed and obtain generalized characters;

[0107] Based on the generalized characters of the data to be processed, a greedy algorithm is adopted to select the optimal generalization level to obtain the generalized data.

[0108] Furthermore, the method of using a greedy algorithm to determine the generalization level and obtain the generalization data specifically includes:

[0109] The semantic distance is used as the optimization measure of the greedy algorithm, and the optimal generalization level is selected to obtain the generalization data.

[0110] Furthermore, the template extraction includes cyclic structure division and vertical structure division, and the template extraction of the generalized data to obtain the substructure of the generalized data specifically includes:

[0111] Performing cyclic structure division based on special characters and spaces in the generalized data, and / or performing vertical structure division based on each data included in the generalized data, to obtain a template candidate set;

[0112] Based on the minimum distance principle, a substructure is selected from the candidate template candidate set as the substructure of the generalized data.

[0113] Furthermore, performing cyclic structure division based on special characters and spaces in the generalized data, and / or performing vertical structure division based on each data included in the generalized data to obtain a template candidate set specifically includes:

[0114] Performing cyclic structure division based on the special character set spaces in the generalized data, calculating the cyclic distance between each data in the generalized data, and when the cyclic distance is less than a preset first similarity threshold, adding the substructure obtained by the cyclic structure division as a candidate structure to the template candidate set;

[0115] and / or

[0116] Perform vertical structural division based on each data included in the generalized data, calculate the vertical distance between each data in the generalized data, and when the vertical distance is less than a preset second similarity threshold, add the substructure obtained by the vertical structural division as a candidate structure to the template candidate set.

[0117] Furthermore, the preset data pattern is a regular expression structure, and the template refinement of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes:

[0118] The intermediate nodes in the substructure of the generalized data are refined into leaf nodes to obtain the data pattern of the data to be processed.

[0119] Furthermore, the template-refining of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes:

[0120] Based on the respective parts of the substructure of the generalized data, determining frequent words of the respective parts of the substructure of the generalized data using a statistical method;

[0121] The frequent words and the infrequent words form an alphabet of a regular expression;

[0122] The substructure of the generalized data is template-refined based on the alphabet to obtain a data pattern of the data to be processed.

[0123] Furthermore, the data to be processed is key-value type data or data that can be converted into key-value type data.

[0124] An embodiment of this specification further provides an electronic device, including:

[0125] at least one processor; and,

[0126] a memory communicatively connected to the at least one processor; wherein,

[0127] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0128] Get the data to be processed;

[0129] Generalizing the data to be processed to obtain generalized data;

[0130] Performing template extraction on the generalized data to obtain a substructure of the generalized data, where the substructure of the generalized data satisfies a preset data pattern;

[0131] Template refinement is performed on the substructure of the generalized data to obtain a data pattern of the data to be processed.

[0132] Furthermore, the obtaining of data to be processed further includes:

[0133] Incremental calculation is performed on the data to be processed and the data to be processed at a previous time point to determine the data to be processed belonging to the current time point in the data to be processed.

[0134] Furthermore, performing incremental calculation on the data to be processed and the data to be processed at the previous time point to determine the data to be processed belonging to the current time point in the data to be processed specifically includes:

[0135] The data constraint status of the to-be-processed data at the previous time point is defined as the data submission status of the to-be-processed data, and the to-be-processed data belonging to the current time point in the to-be-processed data is determined.

[0136] Furthermore, generalizing the data to be processed to obtain generalized data specifically includes:

[0137] Taking each character in the data to be processed as a leaf node of the generalization tree;

[0138] Mapping the leaf nodes to upper-layer branch nodes to generalize the characters in the data to be processed and obtain generalized characters;

[0139] Based on the generalized characters of the data to be processed, a greedy algorithm is adopted to select the optimal generalization level to obtain the generalized data.

[0140] Furthermore, the method of using a greedy algorithm to determine the generalization level and obtain the generalization data specifically includes:

[0141] The semantic distance is used as the optimization measure of the greedy algorithm, and the optimal generalization level is selected to obtain the generalization data.

[0142] Furthermore, the template extraction includes cyclic structure division and vertical structure division, and the template extraction of the generalized data to obtain the substructure of the generalized data specifically includes:

[0143] Performing cyclic structure division based on special characters and spaces in the generalized data, and / or performing vertical structure division based on each data included in the generalized data, to obtain a template candidate set;

[0144] Based on the minimum distance principle, a substructure is selected from the candidate template candidate set as the substructure of the generalized data.

[0145] Furthermore, performing cyclic structure division based on special characters and spaces in the generalized data, and / or performing vertical structure division based on each data included in the generalized data to obtain a template candidate set specifically includes:

[0146] Performing cyclic structure division based on the special character set spaces in the generalized data, calculating the cyclic distance between each data in the generalized data, and when the cyclic distance is less than a preset first similarity threshold, adding the substructure obtained by the cyclic structure division as a candidate structure to the template candidate set;

[0147] and / or

[0148] Perform vertical structural division based on each data included in the generalized data, calculate the vertical distance between each data in the generalized data, and when the vertical distance is less than a preset second similarity threshold, add the substructure obtained by the vertical structural division as a candidate structure to the template candidate set.

[0149] Furthermore, the preset data pattern is a regular expression structure, and the template refinement of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes:

[0150] The intermediate nodes in the substructure of the generalized data are refined into leaf nodes to obtain the data pattern of the data to be processed.

[0151] Furthermore, the template-refining of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes:

[0152] Based on the respective parts of the substructure of the generalized data, determining frequent words of the respective parts of the substructure of the generalized data using a statistical method;

[0153] The frequent words and the infrequent words form an alphabet of a regular expression;

[0154] The substructure of the generalized data is template-refined based on the alphabet to obtain a data pattern of the data to be processed.

[0155] Furthermore, the data to be processed is key-value type data or data that can be converted into key-value type data.

[0156] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0157] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences between the other embodiments. In particular, the device, electronic device, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0158] The apparatus, electronic device, and non-volatile computer storage medium provided in the embodiments of this specification correspond to the method. Therefore, the apparatus, electronic device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device, and non-volatile computer storage medium will not be repeated here.

[0159] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0160] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0161] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0162] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0163] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0167] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0168] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0169] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0170] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0171] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0172] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0173] The foregoing is merely an embodiment of the present invention and is not intended to limit the present application. For those skilled in the art, various modifications and variations may be made to the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. A data processing method, comprising: Get the data to be processed; Generalizing the data to be processed to obtain generalized data; Performing template extraction on the generalized data to obtain a substructure of the generalized data, where the substructure of the generalized data satisfies a preset data pattern; Performing cyclic structure division based on special characters and spaces in the generalized data, and / or performing vertical structure division based on each data included in the generalized data, to obtain a substructure of the generalized data; Performing template refinement on the substructure of the generalized data to obtain a data pattern of the data to be processed; The preset data pattern is a regular expression structure, and the template refinement of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes: refining the intermediate nodes in the substructure of the generalized data into leaf nodes to obtain the data pattern of the data to be processed; or, The template-refining of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes: determining frequent words in each part of the substructure of the generalized data using a statistical method based on each part of the substructure of the generalized data; forming an alphabet of a regular expression with the frequent words and infrequent words; and template-refining the substructure of the generalized data based on the alphabet to obtain the data pattern of the data to be processed.

2. The method according to claim 1, wherein obtaining the data to be processed further comprises: Incremental calculation is performed on the data to be processed and the data to be processed at a previous time point to determine the data to be processed belonging to the current time point in the data to be processed.

3. The method according to claim 2, wherein performing incremental calculation on the data to be processed and the data to be processed at the previous time point to determine the data to be processed belonging to the current time point in the data to be processed specifically comprises: The data constraint status of the to-be-processed data at the previous time point is defined as the data submission status of the to-be-processed data, and the to-be-processed data belonging to the current time point in the to-be-processed data is determined.

4. The method according to claim 1, wherein generalizing the data to be processed to obtain generalized data specifically comprises: Taking each character in the data to be processed as a leaf node of the generalization tree; Mapping the leaf nodes to upper-layer branch nodes to generalize the characters in the data to be processed and obtain generalized characters; Based on the generalized characters of the data to be processed, a greedy algorithm is adopted to select the optimal generalization level to obtain the generalized data.

5. The method according to claim 4, wherein the method of selecting the optimal generalization level using a greedy algorithm to obtain the generalization data comprises: The semantic distance is used as the optimization measure of the greedy algorithm, and the optimal generalization level is selected to obtain the generalization data.

6. The method according to claim 1, wherein the template extraction includes cyclic structure division and vertical structure division, and the performing template extraction on the generalized data to obtain the substructure of the generalized data specifically includes: Performing cyclic structure division based on special characters and spaces in the generalized data, and / or performing vertical structure division based on each data included in the generalized data, to obtain a template candidate set; Based on the minimum distance principle, a substructure is selected from the template candidate set as the substructure of the generalized data.

7. The method according to claim 6, wherein performing cyclic structure division based on special characters and spaces in the generalized data and / or performing vertical structure division based on individual data included in the generalized data to obtain a template candidate set specifically comprises: Performing cyclic structure division based on the special character set spaces in the generalized data, calculating the cyclic distance between each data in the generalized data, and when the cyclic distance is less than a preset first similarity threshold, adding the substructure obtained by the cyclic structure division as a candidate structure to the template candidate set; and / or Perform vertical structural division based on each data included in the generalized data, calculate the vertical distance between each data in the generalized data, and when the vertical distance is less than a preset second similarity threshold, add the substructure obtained by the vertical structural division as a candidate structure to the template candidate set.

8. The method according to claim 1, wherein the data to be processed is key-value type data or data that can be converted into key-value type data.

9. A data processing device, comprising: Get module to get data to be processed; A generalization module generalizes the data to be processed to obtain generalized data; a template extraction module, performing template extraction on the generalized data to obtain a substructure of the generalized data, wherein the substructure of the generalized data satisfies a preset data pattern; Performing cyclic structure division based on special characters and spaces in the generalized data, and / or performing vertical structure division based on each data included in the generalized data, to obtain a substructure of the generalized data; A template refinement module, performing template refinement on the substructure of the generalized data to obtain a data pattern of the data to be processed; The preset data pattern is a regular expression structure, and the template refinement of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes: refining the intermediate nodes in the substructure of the generalized data into leaf nodes to obtain the data pattern of the data to be processed; or, The template-refining of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes: determining frequent words in each part of the substructure of the generalized data using a statistical method based on each part of the substructure of the generalized data; forming an alphabet of a regular expression with the frequent words and infrequent words; and template-refining the substructure of the generalized data based on the alphabet to obtain the data pattern of the data to be processed.

10. An electronic device comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Get the data to be processed; Generalizing the data to be processed to obtain generalized data; performing template extraction on the generalized data to obtain a substructure of the generalized data, wherein the substructure of the generalized data satisfies a preset data pattern; performing cyclic structure division based on special characters and spaces in the generalized data, and / or performing vertical structure division based on individual data included in the generalized data, to obtain the substructure of the generalized data; Performing template refinement on the substructure of the generalized data to obtain a data pattern of the data to be processed; The preset data pattern is a regular expression structure, and the template refinement of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes: refining the intermediate nodes in the substructure of the generalized data into leaf nodes to obtain the data pattern of the data to be processed; or, The template-refining of the substructure of the generalized data to obtain the data pattern of the data to be processed specifically includes: determining frequent words in each part of the substructure of the generalized data using a statistical method based on each part of the substructure of the generalized data; forming an alphabet of a regular expression with the frequent words and infrequent words; and template-refining the substructure of the generalized data based on the alphabet to obtain the data pattern of the data to be processed.

Images