An unstructured data analysis method and apparatus

Abstract

The application provides a non-structured data analysis method and device. The method comprises the following steps: S1, receiving front-end data, and converting the data into a large text format and transferring the data to a high-speed message queue for consumption processing; S2, dividing the data in the large text format into multiple word groups to obtain a word segmentation result; S3, performing secondary verification on the word segmentation result by using regular matching, and screening out to-be-processed elements; and S4, completing other fields of the to-be-processed elements, organizing the to-be-processed elements into structured data, and storing the structured data. The application can simplify the extraction of data elements, improve the data extraction performance, and structure long-term storage. Through early configuration of extraction rules, the application realizes the flow automatic production of data, automatic consumption and extraction, and structures the extraction result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unstructured data, in particular to an unstructured data analysis method and device. BACKGROUND

[0002] With the advent and development of the big data era, the constantly growing massive unstructured log data, due to its irregular inherent data characteristics, poses great challenges to the high-performance analysis, storage and extraction of data. Only by doing a good job of data important information extraction work and laying a good foundation for effective use of massive data and data efficiency can the massive data access necessarily require a large amount of storage consumption. In order to save storage space, it is usually necessary to set a storage period, and the expired data is automatically deleted, which results in that the data cannot be stored for a long time and the storage cost is high. The current massive unstructured data fuzzy retrieval method cannot accurately match the required information, and the effective utilization rate is not high. Since the unstructured data cannot be standardized and stored in a two-dimensional table form, it can only exist in the form of large text.

[0003] The current traditional processing method of unstructured and semi-structured data includes:

[0004] 1. Generally, no extraction is done, and the full text is directly stored in the index library for simple fuzzy retrieval. This method is the minimum data utilization, and cannot accurately extract effective data elements, usually resulting in a large amount of valuable data loss.

[0005] 2. Although regular simple matching can extract specified elements, the overall performance is not high, and the accuracy is insufficient, which cannot meet the normal extraction performance requirements.

[0006] In view of the above problems, the present application provides an unstructured data analysis method and device, which can simplify data element extraction, improve data extraction performance, and structure long-term storage. SUMMARY

[0007] In order to solve the problems of insufficient accuracy of the existing technology in extracting elements and inability to meet the normal extraction performance requirements, the present application provides an unstructured data analysis method and device to solve the above technical defects.

[0008] According to one aspect of the present application, an unstructured data analysis method is provided, which comprises the following steps:

[0009] S1, receiving front-end data, and converting the data into a large text format to a high-speed message queue for consumption processing;

[0010] S2, dividing the data in large text format into multiple word groups to obtain a word segmentation result;

[0011] S3, performing secondary verification on the word segmentation result by using regular matching, and screening out the to-be-processed element; and

[0012] S4, completing other fields of the to-be-processed element, organizing into structured data, and storing the structured data.

[0013] In a specific embodiment, the method further comprises marking the data in the large text format, and the marking is used to determine whether to consume; if the data in the large text format has been marked, the process is directly ended; if the data in the large text format has not been marked, the operation in step S2 is performed.

[0014] In a specific embodiment, in step S1, the front-end data to be processed is received by using a file transfer protocol or a program interface.

[0015] In a specific embodiment, in step S2, the data in the large text format is segmented and divided according to special characters, and is divided into a plurality of individual words.

[0016] In a specific embodiment, the special characters include a comma, a period, an equal sign, a closing quotation mark, a colon, a bracket, and a square bracket.

[0017] According to a second aspect of the present application, an unstructured data analysis device is provided, which comprises:

[0018] A receiving module receives front-end data, and converts the data into a large text format and sends the data to a high-speed message queue for consumption processing.

[0019] A word segmentation module divides the data in the large text format into a plurality of word groups, and obtains a word segmentation result.

[0020] A verification module performs secondary verification on the word segmentation result by using regular matching, and screens out a to-be-processed element.

[0021] A storage module completes other fields of the to-be-processed element, organizes into structured data, and stores the structured data.

[0022] In a specific embodiment, the method further comprises marking the data in the large text format, and the marking is used to determine whether to consume; if the data in the large text format has been marked, the process is directly ended; if the data in the large text format has not been marked, the operation in step S2 is performed.

[0023] In a specific embodiment, in the receiving module, the front-end data to be processed is received by using a file transfer protocol or a program interface.

[0024] In specific embodiments, in the word segmentation module, the large text format data is segmented and split according to special characters, and divided into multiple individual words, and the special characters include comma, period, equal sign, closing quotation mark, colon, bracket, square bracket, and, or, and not.

[0025] In a third aspect, the application provides a computer readable storage medium, the medium storing a computer program, when the computer program is executed by a processor, the method of any one of the above is implemented.

[0026] Compared with the prior art, the beneficial results of the application are:

[0027] The application aims to simplify the extraction of data elements, improve data extraction performance, and structure long-term storage. By configuring extraction rules in advance, including ID number rules, mobile phone number rules, address rules, etc., the data can be automatically produced and consumed in a streaming manner, and the extraction results can be structured. BRIEF DESCRIPTION OF DRAWINGS

[0028] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the attached drawings:

[0029] Figure 1 is a flowchart of the unstructured data analysis method according to the application;

[0030] Figure 2 is a structural diagram of the unstructured data analysis method according to the application;

[0031] Figure 3 is a structural diagram of the unstructured data analysis device according to the application;

[0032] Figure 4 is a structural diagram of the computer system of the electronic device suitable for implementing the embodiments of the application. DETAILED DESCRIPTION

[0033] The application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0034] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and embodiments.

[0035] Figure 1 shows the flowchart of the unstructured data analysis method of the application, Figure 2The structure diagram of the unstructured data analysis method of this application is shown in FIG. Figure 1 and Figure 2 , the method comprises the following steps:

[0036] S1. Receive front-end data and transfer the data in large text format to a high-speed message queue for consumption processing.

[0037] In this embodiment, the front-end massive data to be processed is received through the File Transfer Protocol (FTP) and program interface, and is transferred to the high-speed message queue in a specified large text format for consumption and processing. The format is as follows:

[0038] Field name Field description Field type Remark big_text Big text Text Unstructured or semi-structured text in_time In warehouse time DateTime Record in warehouse time out_time Out time DateTime Record last out time 1flag Flag Boolean Whether the consumption flag

[0039] S2. Divide the data in large text format into multiple phrases to obtain word segmentation results.

[0040] In this embodiment, in step S2, the large text format data can be segmented and divided into multiple individual words according to special characters. Special characters include: comma, period, equal sign, seal, colon, brackets, brackets, and, or not (!#$%^&*()_+[]\;',. / *-<>?).

[0041] Through AI training or writing non-AI automatic word segmentation programs, large texts are divided into multiple phrases. The word segmentation rules are divided into individual words according to the above special characters.

[0042] Example unstructured text:

[0043] Hm_e3b19c08b7e6f2aa7af14a2a6945aeec=1598497554;

[0044] Hm_lpvt_e3b19c08b7e6f2aa7af14a2a6945aeec=1598497554;

[0045] -=SERVERID=214ab29c07ee42633000e845b27a37f1|1598497550|1598497501

[0046] Email=abcd@sina.com&sfzh=350426xxxxxxxx0027&password=wo2momo&remember=0&verify=Verification Code

[0047] The word segmentation results are as follows:

[0048] Word group Element identification result e3b19c08b7e6f2aa7af14a2a6945aeec Unknown 1598497554 Mobile phone number 214ab29c07ee42633000e845b27a37f1 Unknown 1598497550 Mobile phone number 1598497551 Mobile phone number abcd@sina.com Email 350426xxxxxxxx0027 ID card number wo2momo Unknown Verify Unknown Hm Unknown lpvt Unknown SERVERID Unknown Verification code Chinese word group password

[0049] Between step S1 and step S2, further comprising: performing marking processing on the data in the large text format, the marking being used to determine whether the data in the large text format has been consumed, if the data in the large text format has been marked, directly ending the process, if the data in the large text format has not been marked, performing the operation of step S2.

[0050] S3, performing secondary verification on the segmentation result by using regular matching, and screening out the to-be-processed elements.

[0051] For example, the regular expression is as follows:

[0052] Mobile phone number: / ^1[3|4|5|6|7|8|9][0-9]{9}$ /

[0053] Identity card:

[0054] ^[1-9]\d{5}(18|19|(10|11|12))(([0-2][1-9])|10|20|30|31)\d{3}[0-9Xx]$

[0055] Email:

[0056] / ^[0-9a-zA-Z]+([\.\-_]*[0-9a-zA-Z]+)*@([0-9a-zA-Z]+[\-_]*[0-9a-zA-Z]+\.)+[0-9a-zA-Z]{2,6}$ /

[0057] The element types, such as names, addresses, and other elements, are self-written expressions.

[0058] S4, completing other fields of the to-be-processed elements, organizing into structured data, and storing the structured data.

[0059] According to the need, the other fields of the elements are completed, the structured data is organized, and the structured data is stored, as shown in the following table.

[0060]

[0061] The present application mainly solves the element extraction, structured storage, and data completion of unstructured data of big data, realizes one-step data, and long-term storage. It can be used for more in-depth and efficient analysis of enterprises or institutions with a large amount of data.

[0062] Further reference Figure 3 , as an implementation of the above method, the present application provides an embodiment of an unstructured data analysis device, which corresponds to the method embodiment shown in Figure 1 , the device can be applied to various electronic devices. The device 300 includes the following modules:

[0063] The receiving module 310 receives front-end data and converts the data into a large text format to a high-speed message queue for consumption processing;

[0064] The word segmentation module 320 divides the data in the large text format into multiple word groups to obtain a word segmentation result;

[0065] The checking module 330 performs secondary checking on the word segmentation result by using regular matching to screen out to-be-processed elements; and

[0066] The storage module 340 completes other fields of the to-be-processed elements, organizes structured data, and stores the structured data.

[0067] In specific embodiments, the method further includes performing marking processing on the data in the large text format, and the marking is used to determine whether to consume. If the data in the large text format has been marked, the process is directly ended. If the data in the large text format has not been marked, the operation in the word segmentation module 320 is performed.

[0068] In specific embodiments, in the receiving module 310, the front-end data to be processed is received through a file transfer protocol or a program interface.

[0069] In specific embodiments, in the word segmentation module 320, the data in the large text format is segmented and divided according to special characters, and is divided into multiple individual words. The special characters include a comma, a period, an equal sign, a closing quotation mark, a colon, a bracket, a square bracket, and and or not.

[0070] Reference is made below to Figure 4 which shows a structural schematic diagram of a computer system 400 suitable for implementing a terminal device or a server according to an embodiment of the present application. Figure 4 The terminal device or the server shown is merely an example and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0071] As shown in Figure 4 , the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or programs loaded from a storage portion 408 to a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the system 400 are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0072] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as necessary. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 410 as necessary, so that a computer program read out therefrom is installed in the storage section 408 as necessary.

[0073] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above-described functions defined in the methods of the present application are executed. It should be noted that the computer readable medium described in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer the program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF or the like, or any suitable combination of the above.

[0074] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0075] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0076] The modules described in the embodiments of the present application can be implemented through software or hardware. The described units can also be implemented as a processor, for example, a processor can be described as: a processor includes a receiving module, an obtaining module, a determining module, a calculating module and a generating module. In some cases, the names of the units do not constitute a limitation on the units themselves, for example, the receiving unit can also be described as: a module configured to obtain the configuration information of the target user according to the determination that the verification request information includes the username, the request time, the user signature code and the client application code.

[0077] As another aspect, the application also provides a computer readable medium, which can be included in the server described in the above embodiments, or can exist independently without being assembled into the server. The computer readable medium carries one or more programs, which, when executed by the server, cause the server to: receive the verification request information sent by the client of the target user; in response to determining that the verification request information includes the username, the request time, the user signature code and the client application code, obtain the preset configuration information of the target user, wherein the configuration information includes the preset user password corresponding to the username; determine whether the verification request information is valid according to the request time, and in response to determining that it is valid, determine whether the user signature code is included in the preset storage area; in response to determining that it is not included, store the user signature code in the preset storage area, and calculate the server application code based on the user password, the request time and the user signature code; and in response to determining that the server application code and the client application code match, generate the verification success information for representing that the verification request is a legal request.

[0078] In addition, the computer readable medium can be included in the terminal device described in the above embodiments, or can exist independently without being assembled into the terminal device. The computer readable medium carries one or more programs, which, when executed by the terminal device, cause the terminal device to: obtain the user information input by the target user, wherein the user information includes the username and the user password; generate the user signature code representing the target user based on the user information; determine the request time; calculate the client application code based on the user password, the request time and the user signature code; generate the verification request information including the username, the request time, the user signature code and the client application code; and send the verification request information to the server.

[0079] The above description is merely preferred embodiments of the application and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application disclosed in the application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the above inventive concept. For example, the above features can be replaced with the technical features disclosed in the application (but not limited to) having similar functions to form technical solutions.

Claims

1. A method of unstructured data analysis, characterized by, The method comprises the following steps: S1, receiving front-end data, and converting the data into a large text format to a high-speed message queue for consumption processing, performing marking processing on the large text format data, the marking being used to determine whether to consume, if the large text format data has been marked, directly ending the process, if the large text format data has not been marked, performing the operation of step S2; S2, performing segmentation and splitting of the large text format data according to special characters, dividing into multiple individual words, the special characters including: comma, period, equal sign, semicolon, colon, parentheses, square brackets, and, or, not, dividing the large text format data into multiple word groups to obtain a word segmentation result; S3, performing secondary verification on the word segmentation result by using regular matching to screen out to-be-processed elements; and S4, completing other fields of the to-be-processed elements, organizing into structured data, and storing the structured data.

2. The unstructured data analysis method of claim 1, wherein, In step S1, the front-end data to be processed is received by a file transfer protocol or a program interface.

3. An unstructured data analysis apparatus characterized by comprising: The device comprises: a receiving module, receiving front-end data, and converting the data into a large text format to a high-speed message queue for consumption processing, performing marking processing on the large text format data, the marking being used to determine whether to consume, if the large text format data has been marked, directly ending the process, if the large text format data has not been marked, performing the operation of step S2; a word segmentation module, performing segmentation and splitting of the large text format data according to special characters, dividing into multiple individual words, the special characters including: comma, period, equal sign, semicolon, colon, parentheses, square brackets, and, or, not, dividing the large text format data into multiple word groups to obtain a word segmentation result; a verification module, performing secondary verification on the word segmentation result by using regular matching to screen out to-be-processed elements; and a storage module, completing other fields of the to-be-processed elements, organizing into structured data, and storing the structured data.

4. The unstructured data analysis apparatus according to claim 3, wherein In the receiving module, the front-end data to be processed is received by a file transfer protocol or a program interface.

5. A computer readable storage medium, the medium storing a computer program, when the computer program is executed by a processor, implementing the method of any one of claims 1-2.

Images