File retrieval method and system based on HBase

Abstract

The application provides a file retrieval method and system based on HBase, and relates to the field of computer systems.In the application, the field library defined by each business system in the incremental data is uploaded by each project group when uploading the file to the bank storage system through busid, and the field library extracted from the unstructured text by using an algorithm is trained by using a bert model or a chatGLM model in a natural language processing technology.The application is based on the technical scheme of hbase+es, the search feature of es is used to quickly and accurately search the unstructured data stored in the hadoop cluster, the unstructured data files meeting the conditions are searched in the storage system through the self-definable conditions, and the search capability of the unstructured data files of the storage system is enhanced.

Description

Technical Field

[0001] This invention relates to the field of computer systems, and more specifically, to a file retrieval method and system based on HBase. Background Technology

[0002] HBase can be used to store and process semi-structured data, and can process massive amounts of data in real time, dynamically providing users with data storage and read / write services. However, with increasing demands and more complex application scenarios, users' requirements for query methods are becoming more complex. In HBase's design, only the row key of each row of data serves as the unique index for data retrieval. This imposes many limitations on HBase's retrieval capabilities, requiring additional consideration of the table's usage scenario when designing the table structure. Queries often necessitate full table scans, which is an unacceptable disaster for tables with massive amounts of data. The single method of retrieving data by row key can no longer meet the needs of many applications.

[0003] With the explosive growth of unstructured data in the past decade, storage components have stored massive amounts of unstructured data files. These components use busids to upload and download unstructured data files. The file metadata is stored in HBase, while the files themselves are stored in a Hadoop cluster. Therefore, if different business systems want to query other structured data, they must build their own system, store the busid within that system, retrieve the corresponding busid using specific conditions, and then call the storage component's methods to download the corresponding unstructured data file. However, this approach, which only distinguishes unstructured data files from different business systems, results in a highly disorganized and difficult-to-distinguish unstructured data file at the storage end. Searching for specific unstructured data files based on certain conditions is particularly challenging, especially when searching files from different business systems. Therefore, the following solution is proposed to address these difficulties. Summary of the Invention

[0004] To address the needs outlined in the background, this invention provides a file retrieval method and system based on HBase. The HBase+ES technical solution leverages the search capabilities of ES to quickly and accurately retrieve unstructured data stored in a Hadoop cluster.

[0005] A file retrieval method based on HBase, the specific steps of which include:

[0006] Step 1: Define a custom field library, specifically dividing the data in the business system into existing data and incremental data;

[0007] The custom field library for incremental data consists of two parts: one part is the field library defined by each business system, and the other part is the field library extracted from unstructured text using algorithms.

[0008] The custom field library for existing data comes from a field library extracted from unstructured text using algorithms;

[0009] The significance of a custom field library is that when searching in the Elasticsearch search system later, the corresponding unstructured data files can be found by using certain conditions;

[0010] The field libraries customized by each business system in the incremental data are uploaded by each project team when uploading files to the CCB storage system via busid. The field libraries extracted from unstructured text using algorithms are trained using BERT or chatGLM models in natural language processing technology.

[0011] Specifically, the algorithm extracts a field library from unstructured text. Taking BERT model training as an example, the unstructured text paragraphs first need to be preprocessed. The preprocessing mainly involves syntactic analysis of the unstructured text paragraphs to complete sentences lacking subjects; establishing a pronoun disambiguation dictionary and using a regular expression matching algorithm to replace pronouns in the unstructured text paragraphs; and then segmenting the unstructured text paragraphs into sentences to obtain a sentence set.

[0012] Subsequently, a field recognition model is constructed based on the pre-obtained BERT model; the pre-processed unstructured text paragraph data is input into the field recognition model to obtain a high-dimensional feature vector; the high-dimensional feature vector is input into the chatGLM model to perform structured prediction on character labels, obtain the logarithm of the label sequence probability, and output the label sequence with the highest score to obtain the field recognition result;

[0013] Step 2: Store the metadata information of the custom field library and unstructured data files in an HBase table;

[0014] Step 3: Import the HBase table data into Elasticsearch using a data import script;

[0015] Step 4: Based on the HBase table data imported into Elasticsearch, use the Elasticsearch search system to search and download unstructured data files according to the conditions.

[0016] Furthermore, the specific steps for importing HBase table data into Elasticsearch via a data import script include: first, traversing all files under a specified path in the HBase database and obtaining the absolute path of each file; parsing each file into data of a preset format based on the obtained absolute path; converting the preset format data into Result format data; constructing a data import script, and batch writing the Result format data into Elasticsearch via the data import script.

[0017] Furthermore: Unstructured text paragraph data includes character encoding, position encoding, and sentence encoding in the sentence set.

[0018] Furthermore: An HBase-based file retrieval system, comprising:

[0019] The preprocessing module is used to preprocess unstructured text paragraphs;

[0020] The model building module is used to build field recognition models based on pre-obtained Bert and chatGLM models;

[0021] The field recognition module is used to input preprocessed unstructured text paragraph data into the field recognition model to obtain field recognition results;

[0022] The storage module is used to store metadata information of custom field libraries and unstructured data files in HBase tables;

[0023] The data import module is used to import HBase table data into Elasticsearch using data import scripts.

[0024] Furthermore, the terminal device may include a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions that can be executed by the processor. When the terminal device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the deep learning model training method as described in the foregoing embodiments.

[0025] Further: a storage medium storing a computer program, which, when executed by a processor, performs the steps of the method described above.

[0026] Furthermore: a computer program product comprising a computer program that is executed by a processor using the methods described above.

[0027] The beneficial effects of this invention are as follows: Based on the HBase+ES technical solution, this invention utilizes the search characteristics of ES to quickly and accurately search for unstructured data stored in the Hadoop cluster. By using customizable conditions, it searches for unstructured data files that meet the conditions in the storage system, thereby enhancing the search capability of the storage system for unstructured data files. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart of the method of the present invention is shown.

[0030] Figure 2 A schematic diagram of the system of the present invention is shown.

[0031] Figure 3 A schematic diagram of the composition of the terminal device of the present invention is shown. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in the present invention are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of the present invention. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the present invention. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0033] Furthermore, the embodiments described herein are merely some, not all, of the embodiments of the invention. The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0034] It should be noted that the term "comprising" will be used in the embodiments of the present invention to indicate the presence of a feature subsequently declared, but does not preclude the addition of other features. It should also be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of the present invention, it should also be noted that the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0035] The following is a detailed description of this case, in conjunction with the relevant accompanying drawings in the instruction manual.

[0036] like Figure 1 As shown, the present invention provides a file retrieval method based on HBase, the specific steps of which include:

[0037] Step 1: Define a custom field library, specifically dividing the data in the business system into existing data and incremental data;

[0038] The custom field library for incremental data consists of two parts: one part is the field library defined by each business system, and the other part is the field library extracted from unstructured text using algorithms.

[0039] The custom field library for existing data comes from a field library extracted from unstructured text using algorithms;

[0040] The significance of a custom field library is that when searching in the Elasticsearch search system later, the corresponding unstructured data files can be found by using certain conditions;

[0041] The field libraries customized by each business system in the incremental data are uploaded by each project team when uploading files to the CCB storage system via busid. The field libraries extracted from unstructured text using algorithms are trained using BERT or chatGLM models in natural language processing technology.

[0042] Specifically, the algorithm extracts a field library from unstructured text. Taking BERT model training as an example, the unstructured text paragraphs first need to be preprocessed. The preprocessing mainly involves syntactic analysis of the unstructured text paragraphs to complete sentences lacking subjects; establishing a pronoun disambiguation dictionary and using a regular expression matching algorithm to replace pronouns in the unstructured text paragraphs; and then segmenting the unstructured text paragraphs into sentences to obtain a sentence set.

[0043] Subsequently, a field recognition model is constructed based on the pre-obtained BERT model; the pre-processed unstructured text paragraph data is input into the field recognition model to obtain a high-dimensional feature vector; the high-dimensional feature vector is input into the chatGLM model to perform structured prediction on character labels, obtain the logarithm of the label sequence probability, and output the label sequence with the highest score to obtain the field recognition result;

[0044] The aforementioned unstructured text paragraph data includes character encoding, position encoding, and sentence encoding from the sentence set;

[0045] In addition, the above steps also require the construction of a knowledge base for a specific industry field. This embodiment is used for the banking industry, so a knowledge base related to finance and banking is constructed, and the field recognition results are matched according to the matching rules in the specific industry field knowledge base.

[0046] Step 2: Store the metadata information of the custom field library and unstructured data files in an HBase table;

[0047] Step 3: Import the HBase table data into Elasticsearch using a data import script;

[0048] Specifically, the process first iterates through all files under the specified path in the HBase database and obtains the absolute path of each file; then, based on the obtained absolute paths, each file is parsed into data in a preset format; the preset format data is converted into Result format data; a data import script is constructed, and the Result format data is batch-written into Elasticsearch through the data import script.

[0049] Step 4: Search the Elasticsearch search system according to the conditions, find the busid of the corresponding file, and return it to the business system. The business system then downloads the unstructured data file based on the busid.

[0050] like Figure 2 As shown, the present invention provides a file retrieval system based on HBase, comprising:

[0051] The preprocessing module is used to preprocess unstructured text paragraphs;

[0052] The model building module is used to build field recognition models based on pre-obtained Bert and chatGLM models;

[0053] The field recognition module is used to input preprocessed unstructured text paragraph data into the field recognition model to obtain field recognition results;

[0054] The storage module is used to store metadata information of custom field libraries and unstructured data files in HBase tables;

[0055] The data import module is used to import HBase table data into Elasticsearch using data import scripts.

[0056] like Figure 3 As shown, the terminal device 6 may include a processor 601, a storage medium 602, and a bus 603. The storage medium 602 stores machine-readable instructions executable by the processor 601. When the terminal device is running, the processor 601 communicates with the storage medium 602 via the bus 603. The processor 601 executes the machine-readable instructions to perform the steps of the deep learning model training method described in the foregoing embodiments. The specific implementation and technical effects are similar and will not be repeated here.

[0057] For ease of explanation, only one processor is described in the terminal device described above. However, it should be noted that in some embodiments, the terminal device of the present invention may also include multiple processors, and therefore the steps performed by one processor described in the present invention may also be performed jointly by multiple processors or individually.

[0058] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A file retrieval method based on HBase, characterized in that, The specific steps include: Step 1: Define a custom field library. Specifically, the data in the business system is divided into existing data and incremental data. The custom field library for incremental data consists of two parts: one part is the field library defined by each business system, and the other part is the field library extracted from unstructured text using algorithms. The custom field library for existing data comes from the field library extracted from unstructured text using algorithms. The custom field libraries defined by each business system for incremental data are uploaded by each project team when uploading files to the CCB storage system via busid. The field library extracted from unstructured text using algorithms is trained using BERT and chatGLM models in natural language processing. Step 2: Import and store the data into an HBase table. Specifically, store the metadata information of the custom field library and unstructured data files in the HBase table. Step 3: Export the HBase table data and import the HBase table data into Elasticsearch; Step 4: Based on the HBase table data imported into Elasticsearch, and using Elasticsearch's search system, search and download unstructured data files according to the specified conditions; Step one specifically uses an algorithm to extract a field library from unstructured text. First, the unstructured text paragraphs need to be preprocessed. This preprocessing involves syntactic analysis to complete sentences lacking subjects; establishing a pronoun disambiguation dictionary and using a regular expression matching algorithm to replace pronouns within the unstructured text paragraphs; segmenting the unstructured text paragraphs into sentences to obtain a sentence set; then constructing a field recognition model based on a pre-obtained BERT model; finally, inputting the preprocessed unstructured text paragraph data into the field recognition model to obtain the field recognition results. The preprocessed unstructured text paragraph data is input into the field recognition model to obtain a high-dimensional feature vector; the high-dimensional feature vector is input into the chatGLM model to perform structured prediction on character labels, obtain the logarithm of the label sequence probability, and output the label sequence with the highest score. The unstructured text paragraph data includes character encoding, position encoding, and sentence encoding from the sentence set.

2. The method according to claim 1, characterized in that, Import HBase table data into Elasticsearch using a data import script.

3. The method according to claim 1, characterized in that, Elasticsearch's search system retrieves the busid of the corresponding file based on the given conditions and returns it to the business system. The business system then uses the busid to download the unstructured data file.

4. A file retrieval system based on HBase, characterized in that, To implement the method according to any one of claims 1-3, comprising: The preprocessing module is used to preprocess unstructured text paragraphs; The model building module is used to build field recognition models based on pre-obtained Bert and chatGLM models; The field recognition module is used to input preprocessed unstructured text paragraph data into the field recognition model to obtain field recognition results; The storage module is used to store metadata information of custom field libraries and unstructured data files in HBase tables; The data import module is used to import HBase table data into Elasticsearch using data import scripts.

5. A terminal device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the terminal device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method as described in any one of claims 1 to 3.

6. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1-3.

Images