Data import method, device, system and computer readable storage medium
By performing line-by-line field processing, parsing and verification of text data on the Elasticsearch search server, and batch import when the threshold is reached, the problems of low data import efficiency and server downtime are solved, and efficient data import and stability are achieved promote.
Patent Information
- Application Number
- CN202010964369.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-14
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-09-14
AI Technical Summary
The existing data import method is inefficient on servers with different performances, has a long execution time, and is prone to server downtime when fully loaded.
By obtaining text data, line-by-line field processing, parsing, data checksum format conversion is performed based on the preset field mapping of the Elasticsearch search server, and batch import of the Elasticsearch search server when the preset threshold is reached, improving data import efficiency and avoiding server downtime machine.
Improves the efficiency of data import methods, shortens execution time, and avoids the risk of server downtime when fully loaded.
Smart Images

Figure CN111966882B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data import method, device, system and computer-readable storage medium. Background Art
[0002] Elasticsearch is a distributed, scalable, real-time search and analysis engine based on the full-text search engine Apache Lucene(TM). It can easily enable large amounts of data to be searched, analyzed, and explored. The implementation principle of Elasticsearch is mainly divided into the following steps: first, the user submits the data to the Elasticsearch database, and then the corresponding words and sentences are segmented through the word segmentation controller, and their weights and word segmentation results are stored in the data together. When the user searches for data, the results are sorted and scored according to the weights, and then the results are presented to the user. At present, the process of submitting data to Elasticsearch usually adopts manual import, Logstash tool import, or Jdbc tool import. However, the above traditional data import methods are inefficient on servers with different performances, take a long time to execute, and are prone to server downtime when fully loaded.
[0003] Currently, no effective solution has been proposed to address the problems in related technologies, such as low efficiency of data import, long execution time, and easy server crash when fully loaded. Summary of the invention
[0004] The main purpose of the present application is to provide a data import method, device, system and computer-readable storage medium to solve the problems of low efficiency, long execution time and easy server crash when fully loaded in related technical data import methods.
[0005] In order to achieve the above objectives, in a first aspect, the present application provides a data import method.
[0006] The data import method according to the present application comprises the following steps:
[0007] Get text data;
[0008] Based on the preset field mapping of the Elasticsearch search server, the text data is processed line by line to obtain the field data of each line;
[0009] Parse the field data of each row in turn to obtain the field array corresponding to each row;
[0010] Perform data verification on each field array in turn to obtain the data of each verified field; and convert the format of each verified field data in turn to obtain the JSON data of each corresponding row;
[0011] When the amount of JSON data reaches a preset threshold, the JSON data corresponding to the preset threshold is imported into the Elasticsearch search server in batches.
[0012] In one embodiment, the step of obtaining text data includes:
[0013] The text data is obtained based on the Hive data warehouse tool; the text data is data in text format.
[0014] In one of the embodiments, the field mapping based on the preset Elasticsearch search server includes field type information corresponding to the text data;
[0015] The steps of processing the text data line by line based on the preset field mapping of the Elasticsearch search server to obtain the field data of each line include:
[0016] Based on the field type information, the text data is processed line by line to obtain the field data of each line.
[0017] In one embodiment, the field type information includes a field name, a data type, and a tokenizer.
[0018] In one embodiment, the field data includes a plurality of fields; each field is connected by a separator;
[0019] The steps of parsing the field data of each row in turn to obtain the field array corresponding to each row include:
[0020] The field data of each row is split based on the delimiter in turn to obtain the field array corresponding to each row.
[0021] In one embodiment, the step of performing data verification on each field array in turn to obtain the verified field data includes:
[0022] Data verification is performed on each field array in turn. If there are illegal characters in the fields in the field array, the illegal fields are corrected according to the corresponding fields to obtain the verified field data.
[0023] In one embodiment, the step of batch-importing JSON data corresponding to a preset threshold amount to an Elasticsearch search server includes:
[0024] The preset threshold amount of JSON data is encapsulated in sequence, and the encapsulated JSON data is imported into the Elasticsearch search server in batches.
[0025] In a second aspect, the present application also provides a data import device, comprising:
[0026] A data acquisition unit, used for acquiring text data;
[0027] A field processing unit, used to perform field processing on the text data line by line based on the preset field mapping of the Elasticsearch search server to obtain field data of each line;
[0028] The field parsing unit is used to parse the field data of each row in turn to obtain the field array corresponding to each row;
[0029] The data verification and format conversion unit is used to perform data verification on each field array in turn to obtain the data of each verified field; and perform format conversion on each verified field data in turn to obtain the JSON data of each corresponding row;
[0030] The data import unit is used to batch import the JSON data corresponding to the preset threshold amount into the Elasticsearch search server when the amount of JSON data reaches a preset threshold amount.
[0031] In a third aspect, the present application also provides a data import system, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above-mentioned data import methods when executing the computer program.
[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above-mentioned data import methods when the computer program is executed by a processor.
[0033] In an embodiment of the present application, a data import method is provided, which obtains text data; performs field processing on the text data line by line based on a preset field mapping of an Elasticsearch search server to obtain field data of each line; parses and processes the field data of each line in turn to obtain a field array corresponding to each line; performs data verification on each field array in turn to obtain each verified field data; and converts the format of each verified field data in turn to obtain JSON data corresponding to each line; when the amount of JSON data reaches a preset threshold, batch imports the JSON data corresponding to the preset threshold amount into an Elasticsearch search server, thereby realizing the import of text data into the Elasticsearch search server, thereby solving the problem of low execution efficiency on servers with different performances, and solving the technical problems of server crashes caused by full load and excessive execution time caused by too low efficiency, thereby improving the efficiency of the data import method, shortening the execution time, and not easily causing server crashes when fully loaded. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The schematic embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:
[0035] Figure 1 is a schematic diagram of an application environment of a data import method provided in an embodiment of the present application;
[0036] Figure 2 is a first flow chart of a data import method provided according to an embodiment of the present application;
[0037] Figure 3 is a second flow chart of the data import method provided in an embodiment of the present application;
[0038] Figure 4 is a third flow chart of the data import method provided according to an embodiment of the present application;
[0039] Figure 5 It is a block diagram of a data import device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0040] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0042] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.
[0043] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to specific circumstances.
[0044] In addition, the term "plurality" shall mean two or more.
[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0046] The data import method provided in this application can be applied to Figure 1 In the application environment shown. The processor 102 is connected to the server 104 by communication. The server 104 may be an Elasticsearch search server. The processor 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.
[0047] In one embodiment, Figure 2 As shown, a data import method is provided, which is applied to Figure 1 The processor in the example is used to illustrate, including the following steps:
[0048] Step S210, obtaining text data.
[0049] The text data refers to data in a text format (txt format). For example, the file data may be "ID, name, region, region code, address", etc. In one example, the text data may include multiple lines of data.
[0050] Step S220, based on the preset field mapping of the Elasticsearch search server, field processing is performed on the text data line by line to obtain field data of each line.
[0051] Among them, the Elasticsearch search service is a search server based on Lucene. Elasticsearch can be used to search various document data. Field mapping determines the storage format and query method of the field itself. Reasonable and refined definition of fields is conducive to improving query rates. Field mapping refers to a piece of json data, which defines the name, data type and tokenizer used for each field. Field data can include several fields. For example, the field data of a row includes 5 fields, namely "ID, name, region, region code, address".
[0052] For example, in order to process date fields as dates, number fields as numbers, and string fields as full-text or exact string values, Elasticsearch needs to know what type each field contains. This information about types and fields is stored (contained) in mappings.
[0053] Specifically, the field mapping of Elasticsearch can be preset and defined, and then the text data can be processed line by line based on the preset field mapping of the Elasticsearch search server to obtain the field data of each line.
[0054] Step S230, parse and process the field data of each row in turn to obtain a field array corresponding to each row.
[0055] The field array consists of the fields contained in the field data.
[0056] Specifically, the processor can process each field in segments based on the separators between the fields in the field data, thereby obtaining fields corresponding to each row. The processor can process the fields included in each row in an array, thereby obtaining a field array corresponding to each row.
[0057] Furthermore, the processor obtains a complete line of data, reads it into the memory in the form of a string, and divides the string into field arrays using a delimiter (such as a tab).
[0058] Step S240, perform data verification on each field array in turn to obtain each verified field data; and convert each verified field data in turn to obtain JSON data corresponding to each row.
[0059] Specifically, the processor can detect the field data, and if there are illegal characters in the field data, the illegal field is corrected based on the corresponding field, thereby obtaining each checked field data. The processor converts the format of each checked field data in turn, thereby obtaining the JSON data corresponding to each line.
[0060] For example, the first field is ID, which is in integer format. If non-digits such as "abc" appear here, they are illegal characters and need to be corrected. The correction method is to replace "abc" with a non-repeating value from an auto-increment number. The processing principle of other fields is similar and will not be repeated here. After all fields of the field data are verified, the verified data is converted into JSON data in JSON format and the JSON data is placed in the system queue.
[0061] Step S250, when the amount of JSON data reaches a preset threshold, batch importing the JSON data corresponding to the preset threshold into the Elasticsearch search server.
[0062] Among them, JSON (JavaScript Object Notation) data refers to a lightweight data exchange format data.
[0063] Specifically, the processor may put the JSON data into a system queue, and when the amount of the JSON data put into the system queue reaches a preset threshold, the JSON data corresponding to the preset threshold amount is batch-imported into the Elasticsearch search server.
[0064] For example, if there is a lot of data, it will be continuously written into the system queue until the memory is exhausted and the execution is terminated. Here, every time 10,000 records enter the system queue, they will be processed in batches until all the data is processed, which can effectively reduce the http connection time. After all 10,000 records in the current batch are processed in the above way, the data is submitted uniformly and the data confirmation is written into the Elasticsearch search server, so as to realize the import of text data into the Elasticsearch search server.
[0065] In the above embodiment, text data is obtained; field processing is performed on the text data line by line based on the preset field mapping of the Elasticsearch search server to obtain field data of each line; the field data of each line is parsed and processed in turn to obtain a field array corresponding to each line; data verification is performed on each field array in turn to obtain each verified field data; and each verified field data is formatted in turn to obtain JSON data corresponding to each line; when the amount of JSON data reaches a preset threshold, the JSON data corresponding to the preset threshold amount is imported into the Elasticsearch search server in batches to realize the import of text data into the Elasticsearch search server, thereby solving the problem of low execution efficiency on servers with different performances, and solving the technical problems of server crashes caused by full load and long execution time caused by too low efficiency, thereby improving the efficiency of the data import method, shortening the execution time, and not easily causing server crashes when fully loaded.
[0066] In a specific embodiment, the step of obtaining text data includes:
[0067] The text data is obtained based on the Hive data warehouse tool; the text data is data in text format.
[0068] Among them, Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading; it is a mechanism that can store, query and analyze large-scale data stored in Hadoop.
[0069] Specifically, the processor can obtain text data in text format based on the Hive data warehouse tool, and can parse, define the format, and import the data on the obtained file to realize importing the text data into the Elasticsearch search server, thereby solving the problem of low execution efficiency on servers with different performances. At the same time, it solves the technical problems of server crashes caused by full load and long execution time caused by too low efficiency, thereby improving the efficiency of the data import method, shortening the execution time, and not easily causing server crashes when fully loaded.
[0070] In a specific embodiment, the field mapping based on the preset Elasticsearch search server includes field type information corresponding to the text data. The steps of performing field processing on the text data line by line based on the field mapping of the preset Elasticsearch search server to obtain the field data of each line include:
[0071] Based on the field type information, the text data is processed line by line to obtain the field data of each line.
[0072] The field type information can be used to indicate what type the fields in each row of the text data contain. For example, based on the corresponding field type information, a date field can be processed as a date, a number field can be processed as a number, and a string field can be processed as a full-text or an exact string value. For example, field similarity information can be, but is not limited to, keyword, date, text, and double.
[0073] In a specific embodiment, the field type information includes a field name, a data type, and a word segmenter.
[0074] In a specific embodiment, the field data includes multiple fields; each field is connected by a separator. The steps of parsing the field data of each row in turn to obtain the field array corresponding to each row include:
[0075] The field data of each row is split based on the delimiter in turn to obtain the field array corresponding to each row.
[0076] The field data includes multiple fields; for example, the field data may be "ID, name, region, region code, address". Each field is connected by a separator; for example, the separator between fields may be a tab, and the text data (txt document) may contain multiple lines of such data.
[0077] Specifically, the principle of line-by-line processing is adopted, and the field data of each line is segmented and processed based on the separator in turn, so as to obtain the field array corresponding to each line.
[0078] Furthermore, when the processor obtains a complete line of data, it reads it into the memory in the form of a string and divides the string into an array using the tab character as a delimiter.
[0079] In one embodiment, Figure 3 As shown, a data import method is provided, which is applied to Figure 1 The processor in the example is used to illustrate, including the following steps:
[0080] Step S310, obtaining text data.
[0081] Step S320, based on the preset field mapping of the Elasticsearch search server, the text data is processed line by line to obtain the field data of each line.
[0082] Step S330, parse and process the field data of each row in turn to obtain a field array corresponding to each row.
[0083] Step S340, perform data verification on each field array in turn. If there are illegal characters in the fields in the field array, correct the illegal fields according to the corresponding fields to obtain the verified field data.
[0084] Specifically, illegal characters can be determined based on field type information. For example, the first field in the text data is ID, which is in the form of an integer. If a non-digit such as "abc" appears here, it is an illegal character. Then the processor can correct the illegal field according to the corresponding field by taking a non-repeating value from an auto-increment number to replace "abc". The processing principles of other fields are similar and will not be repeated here.
[0085] Step S350, convert the format of each verified field data in turn to obtain JSON data corresponding to each row.
[0086] Step S360, when the amount of JSON data reaches a preset threshold, the JSON data corresponding to the preset threshold amount is imported into the Elasticsearch search server in batches.
[0087] Among them, the specific content and process of the above steps 310, 320, 330, 350 and step S360 can be referred to the above content and will not be repeated here.
[0088] Specifically, by acquiring text data; performing field processing on the text data line by line based on the preset field mapping of the Elasticsearch search server to obtain the field data of each line; parsing and processing the field data of each line in turn to obtain the field array corresponding to each line; performing data verification on each field array in turn, and if there are illegal characters in the fields in the field array, the illegal fields are corrected according to the corresponding fields to obtain the field data after verification; and format conversion of each verified field data in turn to obtain the JSON data of each line; when the amount of JSON data reaches a preset threshold, the JSON data corresponding to the preset threshold amount is imported into the Elasticsearch search server in batches, so as to realize the import of text data into the Elasticsearch search server, thereby solving the problem of low execution efficiency on servers with different performances, and solving the technical problems of server crash caused by full load and long execution time caused by too low efficiency, thereby improving the efficiency of the data import method, shortening the execution time, and not easily causing server crash when fully loaded.
[0089] In one embodiment, Figure 4 As shown, a data import method is provided, which is applied to Figure 1 The processor in the example is used to illustrate, including the following steps:
[0090] Step S410, obtaining text data.
[0091] Step S420, based on the preset field mapping of the Elasticsearch search server, the text data is processed line by line to obtain field data of each line.
[0092] Step S430, parse and process the field data of each row in turn to obtain a field array corresponding to each row.
[0093] Step S440, performing data verification on each field array in turn. If there are illegal characters in the fields in the field array, the illegal fields are corrected according to the corresponding fields to obtain the verified field data.
[0094] Step S450, convert the format of each verified field data in turn to obtain JSON data corresponding to each row.
[0095] Among them, the specific content and process of the above steps 410, 420, 430, 440 and step S450 can be referred to the above content and will not be repeated here.
[0096] Step S460, when the amount of JSON data reaches a preset threshold, the JSON data of the preset threshold amount are encapsulated in sequence, and the encapsulated JSON data are imported into the Elasticsearch search server in batches.
[0097] Specifically, in a batch processing logic, the JOSN data is also encapsulated one by one. The encapsulation here refers to calling the internal API (bulkRequest) of the Elasticsearch search server and passing the JOSN data as a parameter to the API. After all 10,000 records in the current batch are processed in the above manner, the data is submitted once and written to the Elasticsearch search server.
[0098] Specifically, by obtaining text data; processing the text data line by line based on the preset field mapping of the Elasticsearch search server to obtain the field data of each line; parsing the field data of each line in turn to obtain the field array corresponding to each line; verifying the data of each field array in turn to obtain the verified field data; and converting the format of each verified field data in turn to obtain the JSON data of each line; when the amount of JSON data reaches the preset threshold (such as 10,000), batch processing is performed once for every 10,000 records entering the system queue (this can effectively reduce the http connection time) until all the data are processed; in a batch processing logic, the JSON data is also encapsulated one by one, and after all 10,000 records in the current batch are processed in the above manner, the data is submitted once and the data is written to the Elasticsearch search server. It realizes the import of text data into the Elasticsearch search server, thereby solving the problem of low execution efficiency on servers with different performances. It also solves the technical problems of server crashes caused by full load and long execution time caused by too low efficiency, thereby improving the efficiency of the data import method, shortening the execution time, and not easily causing server crashes when fully loaded.
[0099] In one example, in order to effectively utilize server resources, the data import method of the present application can adopt a configurable manner. When the tool is called up, the execution thread of the service startup can be informed by passing parameters. In this way, the working parameters can be reasonably adjusted according to the specific configuration of the running server to maximize the use of server resources and improve work efficiency.
[0100] It should be understood that although Figure 2-4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0101] In one embodiment, Figure 5 As shown, a data import device is provided, the device comprising:
[0102] The data acquisition unit 510 is used to acquire text data.
[0103] The field processing unit 520 is used to perform field processing on the text data line by line based on the preset field mapping of the Elasticsearch search server to obtain field data of each line.
[0104] The field parsing unit 530 is used to parse the field data of each row in turn to obtain a field array corresponding to each row.
[0105] The data verification and format conversion unit 540 is used to perform data verification on each field array in turn to obtain each verified field data; and perform format conversion on each verified field data in turn to obtain JSON data corresponding to each row.
[0106] The data importing unit 550 is used to batch import the JSON data corresponding to the preset threshold amount into the Elasticsearch search server when the amount of JSON data reaches a preset threshold amount.
[0107] For the specific definition of the data import device, please refer to the definition of the data import method above, which will not be repeated here. Each module in the above-mentioned data import device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the data import device in the form of hardware, or can be stored in the memory in the data import device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0108] In one embodiment, a data import system is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above-mentioned data import methods when executing the computer program.
[0109] Specifically, the processor can be used to perform the following steps:
[0110] Get text data;
[0111] Based on the preset field mapping of the Elasticsearch search server, the text data is processed line by line to obtain the field data of each line;
[0112] Parse the field data of each row in turn to obtain the field array corresponding to each row;
[0113] Perform data verification on each field array in turn to obtain the data of each verified field; and convert the format of each verified field data in turn to obtain the JSON data of each corresponding row;
[0114] When the amount of JSON data reaches a preset threshold, the JSON data corresponding to the preset threshold is imported into the Elasticsearch search server in batches.
[0115] In the above embodiment, the processor imports the text data into the Elasticsearch search server by parsing, formatting, importing and other processes of the text data in text format, thereby solving the problem of low execution efficiency on servers with different performances, and at the same time solving the technical problems of server crashes due to full load and long execution time due to too low efficiency, thereby improving the efficiency of the data import method, shortening the execution time, and not easily causing server crashes when fully loaded.
[0116] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data import methods are implemented.
[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0118] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0119] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.
Claims
1. A data import method, characterized in that: The following steps are involved: Get text data; Performing field processing on the text data line by line based on a preset field mapping of an Elasticsearch search server to obtain field data of each line; The field data of each row is analyzed in turn to obtain a field array corresponding to each row; Performing data verification on each of the field arrays in turn to obtain each verified field data; and converting the format of each of the verified field data in turn to obtain JSON data corresponding to each row; and putting the JSON data into the system queue; When the amount of the JSON data put into the system queue reaches a preset threshold, batch-importing the JSON data corresponding to the preset threshold amount into the Elasticsearch search server; The field data of each row is parsed in sequence to obtain a field array corresponding to each row, including: Based on the separators between the fields in the field data, each field is segmented to obtain the fields corresponding to each row; Process the fields contained in each row into an array, and then obtain the field array corresponding to each row; Wherein, the field data includes multiple fields; each field is connected by a separator; The step of sequentially verifying the data of each field array to obtain the verified field data comprises: Performing data verification on each of the field arrays in turn, if there are illegal characters in the fields in the field array, correcting the illegal characters according to the corresponding fields to obtain the verified field data; The method further comprises: In a configurable way, when the tool is called up, the execution thread started by the service is informed by passing parameters, and the working parameters are adjusted according to the specific configuration of the running server.
2. The data import method according to claim 1, characterized in that: The step of obtaining text data comprises: The text data is obtained based on a Hive data warehouse tool; the text data is data in a text format.
3. The data import method according to claim 1, characterized in that: The field mapping based on the preset Elasticsearch search server includes field type information corresponding to the text data; The step of performing field processing on the text data line by line based on the field mapping of the preset Elasticsearch search server to obtain field data of each line includes: The text data is processed line by line based on the field type information to obtain the field data of each line.
4. The data import method according to claim 3, characterized in that: The field type information includes field noun, data type and word segmenter.
5. The data import method according to claim 1, characterized in that: The step of importing the JSON data corresponding to the preset threshold amount into the Elasticsearch search server in batches comprises: The preset threshold amount of the JSON data is encapsulated in sequence, and the encapsulated JSON data is imported into the Elasticsearch search server in batches.
6. A data import device, characterized in that: include: A data acquisition unit, used for acquiring text data; A field processing unit, used to perform field processing on the text data line by line based on a preset field mapping of an Elasticsearch search server to obtain field data of each line; A field parsing unit, used to parse the field data of each row in turn to obtain a field array corresponding to each row; The data verification and format conversion unit is used to perform data verification on each of the field arrays in turn to obtain each verified field data; and perform format conversion on each of the verified field data in turn to obtain JSON data corresponding to each row; and put the JSON data into the system queue; A data importing unit, configured to import the JSON data corresponding to the preset threshold amount into the Elasticsearch search server in batches when the amount of the JSON data put into the system queue reaches a preset threshold amount; The field parsing unit is further used to perform segment processing on each field based on the separators between each field in the field data, thereby obtaining fields corresponding to each row; perform array processing on the fields contained in each row, thereby obtaining field arrays corresponding to each row; wherein the field data includes multiple fields; each field is connected by a separator; The data verification and format conversion unit is further used to perform data verification on each of the field arrays in turn. If there are illegal characters in the fields of the field array, the illegal characters are corrected according to the corresponding fields to obtain the verified field data of each field; In a configurable way, when the tool is called up, the execution thread started by the service is informed by passing parameters, and the working parameters are adjusted according to the specific configuration of the running server.
7. A data import system, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the data import method according to any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data import method according to any one of claims 1 to 5 are implemented.
