A word segmentation method and word segmenter for variable-length text based on openGemini

Abstract

This application provides a method and segmenter for variable-length text based on OpenGemini, relating to the field of text segmentation technology. The method includes: creating a text index for a field indicator when creating a table in OpenGemini; determining whether a learner segmenter exists for the field indicator with the text index; if it exists, selecting the latest version of the learner segmenter; if it does not exist, using the default segmenter and starting the learner segmenter's learning process; performing segmentation learning on the text data inserted into the field indicator, constructing a trie with frequency information, where each node of the trie records the frequency of words in the text data; generating a variable-length segmenter based on the trie, and persistently storing the variable-length segmenter according to a combination of the table name and the field indicator name; and using the variable-length segmenter to segment the new text data inserted into the field indicator.

Description

Technical Field

[0001] This application relates to the field of text segmentation technology, and in particular to a word segmentation method and word segmenter for variable-length text based on openGemini. Background Technology

[0002] Currently, time-series databases are mainly used for the analysis and storage of time-series data in areas such as monitoring data, the Internet of Things (IoT), and financial and derivatives trading. Web service operation logs are primarily analyzed and stored using storage engines such as ELK. Log data serves as a means of locating problems and monitoring system operation status, while also possessing time-series characteristics and requiring text-based retrieval. Therefore, this paper proposes a new log storage scheme and a unified storage scheme for monitoring data by establishing text indexes through fields in OpenGemini.

[0003] When inserting data, OpenGemini needs to create a full-text index for the indexed fields. This index creation process begins by segmenting the specified fields of the inserted data, followed by creating an inverted index. The default segmentation algorithm segments each string that needs to be separated. However, this results in a larger inverted index space. The unique characteristic of log-type text is that most text from the same service shares similar or consistent formatting and content features. Therefore, a segmenter can be constructed by learning these text features to obtain the segmented results.

[0004] The process of constructing a self-learning word segmenter requires sufficient construction samples. Current technologies determine whether the construction is complete based on the number of log entries inserted. This approach causes two problems:

[0005] 1. It is not possible to dynamically adjust the number of log entries required to construct the word segmenter based on different logs. Some services may use the default word segmenter for a long time because the amount of logs generated is small.

[0006] 2. The log structure cannot be dynamically adapted when it changes later.

[0007] Therefore, a technical solution is urgently needed to solve the above technical problems. Summary of the Invention

[0008] This application demonstrates a word segmentation method and word segmenter for variable-length text based on openGemini.

[0009] Firstly, this application discloses a word segmentation method for variable-length text based on openGemini, the method comprising:

[0010] Step 101: When creating a table in OpenGemini, create a text index for the field indicator field;

[0011] Step 102: Determine whether there is an available learner tokenizer for the field indexed by the text. If it exists, select the latest version of the learner tokenizer. If it does not exist, use the default tokenizer and start learning the learner tokenizer.

[0012] Step 103: Perform word segmentation learning on the text data inserted into the field indicator field, and construct a trie with frequency information. Each node of the trie records the frequency of word occurrence in the text data.

[0013] Step 104: Generate a variable-length word segmenter based on the trie, and persistently store the variable-length word segmenter according to the combination of table name and field index field name;

[0014] Step 105: Use the variable-length word segmenter to perform word segmentation on the new text data inserted into the field indicator field. Specifically, this includes: starting from the first position of a log data entry in the new text data, searching for the longest substring that matches the trie. If no matching longest substring is found, then take the substring with a length of 1 starting from the current position as the word segmentation result.

[0015] Furthermore, prior to step 103, a step is included where text data is inserted into the field indicator field:

[0016] Parse the text index definition of the field indicator field in the table;

[0017] Determine the tokenizer version to use based on the field name and table name in the field indicator;

[0018] The corresponding version of the learning-based word segmenter is used for word segmentation, and the results are stored in the openGemini index store.

[0019] Furthermore, it also includes steps for managing the tokenizer version:

[0020] Maintain an independent learning-based tokenizer version for each combination of table and field indicator fields;

[0021] When the trie hit rate is lower than the preset threshold, the new version of the learning-based word segmenter is triggered.

[0022] After the new version of the learning-based word segmenter completes its learning process, the version number is stored in the openGemini metadata.

[0023] Furthermore, it also includes the query steps:

[0024] Parse the field indicator in the query conditions;

[0025] Get the tokenizer version corresponding to the field indicator;

[0026] Use the appropriate version of the word segmenter to segment the query conditions;

[0027] Searching is performed in the text index based on word segmentation results.

[0028] Furthermore, constructing a trie with frequency information also includes a batch write step:

[0029] After a batch of text data is written, update the frequency of all nodes on the path from the root node to the leaf node;

[0030] If there are no leaf nodes on the path, add leaf nodes and edges, and mark the edges with the end marker '#';

[0031] The last character of the string is processed separately.

[0032] Secondly, this application discloses a word segmenter for variable-length text based on openGemini, the word segmenter comprising:

[0033] The Create Index module is used to create text indexes for field indicator fields when creating tables in OpenGemini;

[0034] The judgment module is used to determine whether there is an available learning tokenizer for the field indexed by the text. If it exists, the latest version of the learning tokenizer is selected. If it does not exist, the default tokenizer is used and the learning tokenizer is started.

[0035] A trie construction module is used to perform word segmentation learning on the text data inserted into the field indicator field and construct a trie with frequency information. Each node of the trie records the frequency of word occurrence in the text data.

[0036] The storage module is used to generate a variable-length word segmenter based on the trie and to persistently store the variable-length word segmenter according to the combination of table name and field index field name;

[0037] The word segmentation module is used to perform word segmentation on the new text data inserted into the field index field using the variable-length word segmenter. Specifically, it includes: starting from the first position of a log data entry in the new text data, searching for the longest substring that matches the trie; if no matching longest substring is found, then taking the substring of length 1 starting from the current position as the word segmentation result.

[0038] Furthermore, prior to constructing the trie module, a data insertion module for inserting text data into the field indicator field is also included, specifically:

[0039] Parse the text index definition of the field indicator field in the table;

[0040] Determine the tokenizer version to use based on the field name and table name in the field indicator;

[0041] The corresponding version of the learning-based word segmenter is used for word segmentation, and the results are stored in the openGemini index store.

[0042] Furthermore, it also includes a word segmenter version management module, specifically:

[0043] Maintain an independent learning-based tokenizer version for each combination of table and field indicator fields;

[0044] When the trie hit rate is lower than the preset threshold, the new version of the learning-based word segmenter is triggered.

[0045] After the new version of the learning-based word segmenter completes its learning process, the version number is stored in the openGemini metadata.

[0046] Furthermore, it also includes a query module, specifically:

[0047] Parse the field indicator in the query conditions;

[0048] Get the tokenizer version corresponding to the field indicator;

[0049] Use the appropriate version of the word segmenter to segment the query conditions;

[0050] Searching is performed in the text index based on word segmentation results.

[0051] Furthermore, it also includes a batch write module, specifically:

[0052] After a batch of text data is written, update the frequency of all nodes on the path from the root node to the leaf node;

[0053] If there are no leaf nodes on the path, add leaf nodes and edges, and mark the edges with the end marker '#';

[0054] The last character of the string is processed separately.

[0055] Thirdly, this application discloses an electronic device comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to perform the method as described in any of the preceding aspects.

[0056] Fourthly, this application discloses a non-transitory computer-readable storage medium in which, when the instructions in the storage medium are executed by a processor of an electronic device, enable the electronic device to perform the methods described in any of the preceding aspects.

[0057] Fifthly, this application discloses a computer program product in which, when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in any of the preceding aspects.

[0058] The technical solution provided in this application may include the following beneficial effects:

[0059] 1. Variable-length word segmenters can save more than 50% of the storage space occupied by text indexes in relatively standardized text data such as logs;

[0060] 2. It can automatically decide whether to rebuild the segmenter based on the current hit rate of the segmenter in the dictionary tree, and record the segmenter version. It can use the appropriate segmenter for different data, resulting in better segmentation effect.

[0061] 3. Based on the word segmenter, the full-text indexing function of the time-series database openGemini was implemented. Attached Figure Description

[0062] Figure 1 This is a flowchart of the steps of the method described in this application;

[0063] Figure 2 This is a structural block diagram of a vehicle positioning coordinate encryption device based on clock intervention according to this application;

[0064] Figure 3 This is a flowchart of the log writing process in this application;

[0065] Figure 4 This is a frequency-based trie graph generated through learning in this application;

[0066] Figure 5 This is a dictionary storage structure diagram of this application;

[0067] Figure 6 This is a block diagram of an electronic device according to this application;

[0068] Figure 7 This is a block diagram of a computer-readable storage medium according to this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] The technical terms appearing in this application are explained as follows:

[0071] openGemini: Huawei's cloud-native distributed time-series database, open-sourced globally in 2022, is based on InfluxDB and features deep architectural optimizations in areas such as high performance, high scalability, low maintenance costs, and low storage costs.

[0072] Field: This refers to the measurement metrics used to describe the data source. These metrics typically change over time. For example, sensor devices may contain fields such as temperature and humidity.

[0073] VictoriaMetrics mergeSet: The index data storage module uses VictoriaMetrics' MergeSet module, which provides functions such as trie storage, compression, merging, and prefix lookup.

[0074] Reference Figure 1 The diagram illustrates a step-by-step flowchart of a variable-length text segmentation method based on openGemini, which can be applied to electronic devices. Specifically, the method may include the following steps:

[0075] Step 101: When creating a table in OpenGemini, create a text index for the field indicator field;

[0076] Step 102: Determine whether there is an available learner tokenizer for the field indexed by the text. If it exists, select the latest version of the learner tokenizer. If it does not exist, use the default tokenizer and start learning the learner tokenizer.

[0077] The specific steps for inserting text data into the field index are as follows: parse the text index definition of the field index in the table; determine the word segmenter version to be used based on the field index name and the table name; perform word segmentation using the corresponding version of the learning word segmenter, and store the results in the index storage of openGemini.

[0078] Step 103: Perform word segmentation learning on the text data inserted into the field indicator field, such as... Figure 4As shown, a trie with frequency information is constructed, where each node of the trie records the frequency of word occurrence in the text data;

[0079] Step 104: Generate a variable-length word segmenter based on the trie, and persistently store the variable-length word segmenter according to the combination of table name and field indicator field name, such as... Figure 5 As shown;

[0080] Step 105: Use the variable-length segmenter to segment the new text data inserted into the field index field. Specifically, this includes: if a segmenter is present, starting from the first position of a log data entry in the new text data, searching for the longest substring that matches the trie. If no matching longest substring is found, then take the substring of length 1 starting from the current position as the segmentation result.

[0081] For example, following this method to partition the log "get images hm.gif http 1.1" using a dictionary, the following path can be partitioned in dictionary D: [get images, hm.gif, http 1.1]

[0082] For each variable-length index in the resulting base VG set, when inserting it into the trie tree, the leaf node needs to be attached to its corresponding inverted linked list. However, based on the partitioning above, the substrings starting with "images" and "1.1" are missing. If a query is performed directly based on this partitioning structure, there will be a case of missing solutions, that is, querying "images" and "1.1" will not find these substrings. For each substring generated above, further suffix partitioning is performed. For example, if the substring length is len, it is further partitioned into [1,len], [2,len], ..., [i,len], ..., [len-1,len], where i is the index subscript.

[0083] Specifically, this also includes the steps for managing the word segmenter version:

[0084] Maintain an independent learning-based tokenizer version for each combination of table and field indicator fields;

[0085] When the trie hit rate is lower than the preset threshold, the new version of the learning-based word segmenter is triggered.

[0086] After the new version of the learning-based word segmenter completes its learning process, the version number is stored in the openGemini metadata.

[0087] Specifically, it also includes the query steps:

[0088] Parse the field indicator in the query conditions;

[0089] Get the tokenizer version corresponding to the field indicator;

[0090] Use the appropriate version of the word segmenter to segment the query conditions;

[0091] Searching is performed in the text index based on word segmentation results.

[0092] Specifically, constructing a trie with frequency information also includes a batch writing step:

[0093] After a batch of text data is written, update the frequency of all nodes on the path from the root node to the leaf node;

[0094] If there are no leaf nodes on the path, add leaf nodes and edges, and mark the edges with the end marker '#';

[0095] The last character of the string is processed separately.

[0096] Specifically, the learner-based tokenizer required for OpenGemini's text indexing starts learning, pruning the tree to obtain a trie. During the learning process, the frequency of all nodes along the path from the root node to a leaf node is incremented by 1, including the leaf node itself. If a leaf node does not exist along this path, a leaf node and an edge are added, marked with the closing marker '#'. Characters at the end of the string are processed separately, and the insertion operation described above for the trie is repeated. This ultimately constructs a trie where each node has a frequency of occurrence. The frequency of occurrence is recorded at each node in the trie, and during construction, it is distinguished whether the log string is a new string, thus recording the current trie's completion level.

[0097] like Figure 3 As shown below, an example of word segmentation is performed using a variable-length text segmentation method based on openGemini:

[0098] 1. This invention provides word segmentation support for full-text indexing of field fields in OpenGemini, enabling OpenGemini to obtain log storage capabilities and providing word segmentation capabilities based on dynamic content changes in processes such as log text insertion and querying:

[0099] - Create text indexes on fields of type field when creating a new table:

[0100] create measurement mst(clientip string tag,logs string field,commentstring field,index idex1 logs type text)

[0101] When creating the table, a full-text index was created for the `logs` field, but no index was created for the `comment` field.

[0102] - Insert data, example:

[0103] Insert mst,clientip=221.200.76.250logs="POST / USA / images / space.gif",comment="It is also a garbage-collected runtime system."1671141203000000000

[0104] Initially, the adaptive dictionary is empty, and the `logs` field has a text index created. Therefore, the default tokenizer needs to be called to perform word segmentation: [POST,USA,IMAGES,space.gif]. Based on the word segmentation results, a trie is inserted, and the initial frequency is recorded. As data continues to be inserted, a trie with frequency information is formed. Then, the trie is pruned to create a new version of the tokenizer. Subsequently, when inserting data, the new version of the tokenizer is called, the word segmentation results are recorded, and an index is created. The index stores the tokenizer's version information.

[0105] 2. The word segmentation dictionary is a prefix tree based on the word segmentation results. In actual storage, it can be divided into a list of variable-length substrings stored row by row. Therefore, VictoriaMetrics' MergeSet is chosen to store the dictionary row by row, with the storage format: prifix+measurementName+fieldName+dicVersion+suffix+tokens. The corresponding dictionary data is found by using a specific prefix + table name + field name + word segmenter version, and then the list of variable-length substrings is retrieved. See the attached diagram: "Dictionary Storage Structure" for details.

[0106] The dictVersion field indicates the dictionary's version information;

[0107] When dictVersion = 0 or is empty, the default tokenizer is used for word segmentation;

[0108] If dictVersion>0, the first step is to find the corresponding version of the tokenizer, load the tokenizer into memory, and use the corresponding version of the tokenizer. The corresponding index information needs to be saved for the dictionary version.

[0109] like Figure 2 The diagram shows a block diagram of a variable-length text segmenter based on OpenGemini, according to this application. The segmenter includes:

[0110] The Create Index module is used to create text indexes for field indicator fields when creating tables in OpenGemini;

[0111] The judgment module is used to determine whether there is an available learning tokenizer for the field indexed by the text. If it exists, the latest version of the learning tokenizer is selected. If it does not exist, the default tokenizer is used and the learning tokenizer is started.

[0112] A trie construction module is used to perform word segmentation learning on the text data inserted into the field indicator field and construct a trie with frequency information. Each node of the trie records the frequency of word occurrence in the text data.

[0113] The storage module is used to generate a variable-length word segmenter based on the trie and to persistently store the variable-length word segmenter according to the combination of table name and field index field name;

[0114] The word segmentation module is used to perform word segmentation on the new text data inserted into the field index field using the variable-length word segmenter. Specifically, it includes: starting from the first position of a log data entry in the new text data, searching for the longest substring that matches the trie; if no matching longest substring is found, then taking the substring of length 1 starting from the current position as the word segmentation result.

[0115] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0116] Optionally, this application also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0117] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0118] Figure 6This is a block diagram illustrating an electronic device 800. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0119] Reference Figure 6 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0120] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0121] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, images, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0122] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0123] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0124] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0125] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0126] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0127] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast operation information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0128] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0129] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0130] Figure 7 This is a block diagram illustrating a computer-readable storage medium 1900. For example, the computer-readable storage medium 1900 can be provided as a server.

[0131] Reference Figure 7 The computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by memory 1932 for storing instructions executable by the processing component 1922, such as an application program. The application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0132] The computer-readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer-readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer-readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer-readable storage medium 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0133] Fifthly, this application discloses a computer program product in which, when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in any of the preceding aspects.

[0134] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0136] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0139] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0142] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0143] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A word segmentation method for variable-length text based on openGemini, characterized in that, The method includes: Step 101: When creating a table in OpenGemini, create a text index for the field indicator field; Step 102: Determine whether there is an available learner tokenizer for the field indexed by the text. If it exists, select the latest version of the learner tokenizer. If it does not exist, use the default tokenizer and start learning the learner tokenizer. Step 103: Perform word segmentation learning on the text data inserted into the field indicator field, and construct a trie with frequency information. Each node of the trie records the frequency of word occurrence in the text data. Step 104: Generate a variable-length word segmenter based on the trie, and persistently store the variable-length word segmenter according to the combination of table name and field index field name; Step 105: Use the variable-length word segmenter to perform word segmentation on the new text data inserted into the field indicator field. Specifically, this includes: starting from the first position of a log data entry in the new text data, searching for the longest substring that matches the trie. If no matching longest substring is found, then take the substring with a length of 1 starting from the current position as the word segmentation result.

2. The word segmentation method for variable-length text based on openGemini as described in claim 1, characterized in that, The step preceding step 103 also includes inserting text data into the field indicator field: Parse the text index definition of the field indicator field in the table; Determine the tokenizer version to use based on the field name and table name in the field indicator; The corresponding version of the learning-based word segmenter is used for word segmentation, and the results are stored in the openGemini index store.

3. The word segmentation method for variable-length text based on openGemini as described in claim 1, characterized in that, It also includes the steps for tokenizer version management: Maintain an independent learning-based tokenizer version for each combination of table and field indicator fields; When the trie hit rate is lower than the preset threshold, the new version of the learning-based word segmenter is triggered. After the new version of the learning-based word segmenter completes its learning process, the version number is stored in the openGemini metadata.

4. The word segmentation method for variable-length text based on openGemini as described in claim 1, characterized in that, It also includes the query steps: Parse the field indicator field in the query conditions; Get the tokenizer version corresponding to the field indicator; Use the appropriate version of the word segmenter to segment the query conditions; Searching is performed in the text index based on word segmentation results.

5. The word segmentation method for variable-length text based on openGemini as described in claim 1, characterized in that, Building a trie with frequency information also includes a batch write step: After a batch of text data is written, update the frequency of all nodes on the path from the root node to the leaf node; If there are no leaf nodes on the path, add leaf nodes and edges, and mark the edges with the end marker '#'; The last character of the string is processed separately.

6. A word segmenter for variable-length text based on openGemini, characterized in that, The word segmenter includes: The Create Index module is used to create text indexes for field indicator fields when creating tables in OpenGemini; The judgment module is used to determine whether there is an available learning tokenizer for the field indexed by the text. If it exists, the latest version of the learning tokenizer is selected. If it does not exist, the default tokenizer is used and the learning tokenizer is started. A trie construction module is used to perform word segmentation learning on the text data inserted into the field indicator field and construct a trie with frequency information. Each node of the trie records the frequency of word occurrence in the text data. The storage module is used to generate a variable-length word segmenter based on the trie and to persistently store the variable-length word segmenter according to the combination of table name and field index field name; The word segmentation module is used to perform word segmentation on the new text data inserted into the field index field using the variable-length word segmenter. Specifically, it includes: starting from the first position of a log data entry in the new text data, searching for the longest substring that matches the trie; if no matching longest substring is found, then taking the substring of length 1 starting from the current position as the word segmentation result.

7. A word segmenter for variable-length text based on openGemini as described in claim 6, characterized in that, It also includes a word segmenter version management module, specifically: Maintain an independent learning-based tokenizer version for each combination of table and field indicator fields; When the trie hit rate is lower than the preset threshold, the new version of the learning-based word segmenter is triggered. After the new version of the learning-based word segmenter completes its learning process, the version number is stored in the openGemini metadata.

8. A word segmenter for variable-length text based on openGemini as described in claim 6, characterized in that, It also includes a query module, specifically: Parse the field indicator field in the query conditions; Get the tokenizer version corresponding to the field indicator; Use the appropriate version of the word segmenter to segment the query conditions; Searching is performed in the text index based on word segmentation results.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images