Word segmentation method and word segmentation device for realizing variable-length text based on openGemini

Abstract

The invention provides a word segmentation method and a word segmentation device for realizing a variable-length text based on openGemini, and relates to the technical field of text word segmentation, the method comprises the following steps: creating a text index for a field index field when an openGemini creates a table; judging whether an available learning type word segmentation device exists in the field index field with the created text index or not, if yes, selecting a learning type word segmentation device of the latest version, and if not, using a default word segmentation device and starting learning of the learning type word segmentation device; word segmentation learning is carried out on the text data inserted into the field index field, a dictionary tree with frequency information is constructed, and each node of the dictionary tree records the occurrence frequency of words in the text data; generating a variable-length word segmentation device based on the dictionary tree, and carrying out persistent storage on the variable-length word segmentation device according to a combination of a table name and a field index field name; and performing word segmentation processing on the new text data inserted into the field index field by using the variable-length word segmentation device.

Description

Technical Field

[0001] The present application relates to the technical field of text segmentation, and in particular to a variable-length text segmentation method and a segmenter based on openGemini. Background Art

[0002] At present, time series databases are mainly used for the analysis and storage of time series data in the fields of monitoring data, Internet of Things, finance and its derivatives trading, etc. The operation logs of web services are mainly based on storage engines such as ELK combination practice to realize the analysis and storage of log data. Log data is used as a means to locate problems and monitor the operation status of the system. It also has the characteristics of time series and the need for text retrieval. Therefore, in openGemini, text indexes are established through field fields to realize log storage, and a new log storage solution and a unified storage solution with monitoring data are obtained.

[0003] When inserting data, openGemini needs to create a full-text index for the indexed field. When creating an index, the specified field of the inserted data must first be segmented, and then an inverted index is created. The segmenter implemented by the default segmentation algorithm will segment each string that needs to be separated by default. On this basis, the inverted index takes up more space. The particularity of log text is that the format and content of most texts of the same service have similar or consistent features. Therefore, the segmentation result can be obtained by learning text features and constructing a segmenter.

[0004] The process of constructing a self-learning word segmenter requires sufficient construction samples. The existing technology implementation determines whether the construction is complete based on the number of log entries inserted. This solution causes two problems:

[0005] 1. It is not possible to dynamically adjust the number of logs required to construct a tokenizer based on different logs. Some services may use the default tokenizer for a long time because of the small amount of logs generated.

[0006] 2. When the log structure changes later, it cannot be adapted dynamically.

[0007] Therefore, a technical solution is urgently needed to solve the above technical problems. Summary of the invention

[0008] The present application shows a word segmentation method and a word segmenter for implementing variable-length text based on openGemini.

[0009] In a first aspect, the present application shows a word segmentation method for variable-length text based on openGemini, the method comprising:

[0010] Step 101, when creating a table in openGemini, create a text index for the field indicator field;

[0011] Step 102, determine whether there is an available learning word segmenter for the field indicator field where the text index is created. If so, select the latest version of the learning word segmenter. If not, use the default word segmenter and start learning the learning word segmenter.

[0012] Step 103, performing word segmentation learning on the text data inserted into the field indicator field, and constructing a dictionary tree with frequency information, wherein each node of the dictionary tree records the frequency of occurrence of a word in the text data;

[0013] Step 104, generating a variable-length word segmenter based on the dictionary tree, and persistently storing the variable-length word segmenter according to a combination of a table name and a field indicator field name;

[0014] Step 105, using the variable-length word segmenter to perform word segmentation on the new text data inserted into the field indicator field, specifically including: starting from the first position of a log data of the new text data, searching for the longest substring matching the dictionary tree, if the matching longest substring is not found, taking the substring with a length of 1 starting from the current position as the word segmentation result.

[0015] Furthermore, before step 103, the method further includes the step of inserting text data into the field indicator field:

[0016] Parse the text index definition of the field indicator field in the table;

[0017] Determine the tokenizer version to be used based on the field name and table name;

[0018] Use the corresponding version of the learning tokenizer to perform token segmentation and store the results in the index storage of openGemini.

[0019] Furthermore, it also includes the word segmenter version management steps:

[0020] Maintain an independent version of the learning tokenizer for each combination of table and field indicator fields;

[0021] When the hit rate of the dictionary tree is lower than the preset threshold, the learning of the new version of the learning word segmenter is triggered;

[0022] After the new version of the learning tokenizer completes learning, the version number is stored in the metadata of openGemini.

[0023] Furthermore, the query step is also included:

[0024] Parse the field indicator field in the query condition;

[0025] Get the tokenizer version corresponding to the field indicator field;

[0026] Use the corresponding version of the word segmenter to segment the query conditions;

[0027] Search in the text index based on the word segmentation results.

[0028] Furthermore, constructing a dictionary tree with frequency information also includes a batch writing step:

[0029] After a batch of text data is written, update the frequencies of all nodes on the path from the root node to the leaf node;

[0030] If there is no leaf node on the path, add a leaf node and an edge, and mark the end mark '#' on the edge;

[0031] The characters at the end of the string are processed separately.

[0032] In a second aspect, the present application shows a word segmenter for implementing variable-length text based on openGemini, the word segmenter comprising:

[0033] Create an index module, which is used to create a text index for the field indicator field when creating a table in openGemini;

[0034] The judgment module is used to judge whether there is an available learning word segmenter for the field indicator field where the text index is created. If so, the latest version of the learning word segmenter is selected. If not, the default word segmenter is used and the learning of the learning word segmenter is started.

[0035] Construct a dictionary tree module, which is used to perform word segmentation learning on the text data inserted into the field indicator field, and construct a dictionary tree with frequency information, wherein each node of the dictionary tree records the frequency of occurrence of a word in the text data;

[0036] A storage module, used for generating a variable-length word segmenter based on the dictionary tree, and persistently storing the variable-length word segmenter according to a combination of a table name and a field indicator field name;

[0037] The word segmentation module is used to use the variable-length word segmenter to perform word segmentation processing on the new text data inserted into the field indicator field, specifically including: starting from the first position of a log data of the new text data, searching for the longest substring matching the dictionary tree; if the matching longest substring is not found, taking the substring with a length of 1 starting from the current position as the word segmentation result.

[0038] Furthermore, before constructing the dictionary tree module, a data insertion module for inserting text data into the field indicator field is also included, specifically:

[0039] Parse the text index definition of the field indicator field in the table;

[0040] Determine the tokenizer version to be used based on the field name and table name;

[0041] Use the corresponding version of the learning tokenizer to perform token segmentation and store the results in the index storage of openGemini.

[0042] Furthermore, it also includes a word segmenter version management module, specifically:

[0043] Maintain an independent version of the learning tokenizer for each combination of table and field indicator fields;

[0044] When the hit rate of the dictionary tree is lower than the preset threshold, the learning of the new version of the learning word segmenter is triggered;

[0045] After the new version of the learning tokenizer completes learning, the version number is stored in the metadata of openGemini.

[0046] Furthermore, it also includes a query module, specifically:

[0047] Parse the field indicator field in the query condition;

[0048] Get the tokenizer version corresponding to the field indicator field;

[0049] Use the corresponding version of the word segmenter to segment the query conditions;

[0050] Search in the text index based on the word segmentation results.

[0051] Furthermore, it also includes a batch writing module, specifically:

[0052] After a batch of text data is written, update the frequencies of all nodes on the path from the root node to the leaf node;

[0053] If there is no leaf node on the path, add a leaf node and an edge, and mark the end mark '#' on the edge;

[0054] The characters at the end of the string are processed separately.

[0055] In a third aspect, the present application shows an electronic device, which includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in any of the above aspects.

[0056] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a method as described in any of the above aspects.

[0057] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in any of the above aspects.

[0058] The technical solution provided by this application may have the following beneficial effects:

[0059] 1. The variable-length tokenizer can save more than 50% of the storage space occupied by text indexes in relatively standardized text data such as logs;

[0060] 2. It can automatically decide whether to rebuild the word segmenter according to the current hit situation of the word segmenter in the dictionary tree, and record the word segmenter version. It can use the appropriate word segmenter for different data, and the word segmentation effect is better;

[0061] 3. The full-text indexing function of the time series database openGemini is implemented based on the word segmenter. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flowchart of the steps of the application method;

[0063] Figure 2 It is a structural block diagram of a vehicle positioning coordinate encryption device based on clock intervention of the present application;

[0064] Figure 3 This is the log writing flow chart of this application;

[0065] Figure 4 It is a dictionary tree diagram with frequency generated by learning in this application;

[0066] Figure 5 It is a dictionary storage structure diagram of this application;

[0067] Figure 6 is a block diagram of an electronic device of the present application;

[0068] Figure 7 It is a block diagram of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0070] The technical terms appearing in this application are explained as follows:

[0071] openGemini: A cloud-native distributed time series database that Huawei will open source globally in 2022. Its kernel is derived from InfluxDB, and it has carried out deep architectural optimization in many aspects such as high performance, high scalability, operation and maintenance costs, and storage costs.

[0072] Field: A field is used to describe the measurement indicators of a data source. It usually changes over time. For example, a sensor device contains fields such as temperature and humidity.

[0073] VictoriaMetrics mergeSet: The index data storage module uses the VictoriaMetrics MergeSet module, which provides dictionary tree storage, compression, merging, prefix query and other functions.

[0074] Reference Figure 1 , shows a flow chart of the steps of a word segmentation method for variable-length text based on openGemini of the present application, which can be applied to electronic devices, wherein the method can specifically include the following steps:

[0075] Step 101, when creating a table in openGemini, create a text index for the field indicator field;

[0076] Step 102, determine whether there is an available learning word segmenter for the field indicator field where the text index is created. If so, select the latest version of the learning word segmenter. If not, use the default word segmenter and start learning the learning word segmenter.

[0077] Insert text data into the field indicator field. The specific steps are: parse the text index definition of the field indicator field in the table; determine the word segmenter version to be used according to the field indicator field name and the table name; use the corresponding version of the learning word segmenter to perform word segmentation, and store the result in the index storage of openGemini.

[0078] Step 103, perform word segmentation learning on the text data inserted into the field indicator field, such as Figure 4As shown, a dictionary tree with frequency information is constructed, and each node of the dictionary tree records the frequency of occurrence of a word in the text data;

[0079] Step 104, generate a variable-length word segmenter based on the dictionary tree, and store the variable-length word segmenter persistently according to the combination of the table name and the field indicator field name, such as Figure 5 As shown;

[0080] Step 105, using the variable-length word segmenter to perform word segmentation processing on the new text data inserted into the field indicator field, specifically including: when there is a word segmenter, starting from the first position of a log data of the new text data, searching for the longest substring matching the dictionary tree, if the matching longest substring is not found, taking the substring of length 1 starting from the current position as the word segmentation result.

[0081] For example, according to the present method, the log "get images hm.gif http 1.1" is divided based on the dictionary, and the following paths can be divided in the dictionary D: [get images, hm.gif, http 1.1]

[0082] When inserting each variable-length index in the basic VG set obtained by the partition into the trie tree, the leaf node needs to hang its corresponding inverted linked list. However, from the above partitioning, the substrings starting with images and 1.1 are missing. If the query is directly based on the partitioning structure, there will be a loss of solution, that is, the substring cannot be found when querying "images" and "1.1". For each substring generated above, further suffix partitioning is performed. If the substring length is len, it is further divided into [1,len], [2,len], ... [i,len], ... [len-1,len], where i is the index subscript.

[0083] Specifically, it also includes the word segmenter version management steps:

[0084] Maintain an independent version of the learning tokenizer for each combination of table and field indicator fields;

[0085] When the hit rate of the dictionary tree is lower than the preset threshold, the learning of the new version of the learning word segmenter is triggered;

[0086] After the new version of the learning tokenizer completes learning, the version number is stored in the metadata of openGemini.

[0087] Specifically, the query steps include:

[0088] Parse the field indicator field in the query condition;

[0089] Get the tokenizer version corresponding to the field indicator field;

[0090] Use the corresponding version of the word segmenter to segment the query conditions;

[0091] Search in the text index based on the word segmentation results.

[0092] Specifically, building a dictionary tree with frequency information also includes batch writing steps:

[0093] After a batch of text data is written, update the frequencies of all nodes on the path from the root node to the leaf node;

[0094] If there is no leaf node on the path, add a leaf node and an edge, and mark the end mark '#' on the edge;

[0095] The characters at the end of the string are processed separately.

[0096] Specifically, the learning tokenizer required for the text index of openGemini starts learning, and the learning is pruned to obtain a dictionary tree. During the learning process, the frequencies of all nodes on the path from the root node to the leaf node are increased by 1, including the leaf node. If there is no leaf node at this path node, a leaf node and an edge are added, and the end mark '#' is marked on the edge. The characters at the end of the string are processed separately, and then the insertion operation of the dictionary tree described above is repeated. Finally, a dictionary tree with a frequency of occurrence for each node is constructed. The frequency of occurrence of each node in the dictionary tree is recorded, and during the construction process, it is distinguished whether the log string is a new string, and then the completeness of the current dictionary is recorded.

[0097] like Figure 3 As shown, the following is an example of word segmentation using a word segmentation method based on openGemini for variable-length text, as shown below:

[0098] 1. The present invention provides word segmentation support for full-text indexing of field fields in openGemini, so that openGemini has log storage capabilities and provides word segmentation capabilities based on dynamic content changes in the processes of inserting and querying log texts:

[0099] -Create a text index for the field type when creating a new table:

[0100] create measurement mst(clientip string tag,logs string field,commentstring field,index idex1 logs type text)

[0101] When creating a table, create a full-text index for the logs field, but no index for the comment field.

[0102] -Insert data, example:

[0103] Insert mst,clientip=221.200.76.250logs="POST / USA / images / space.gif",comment="It is also a garbage-collected runtime system."1671141203000000000

[0104] Initially, the adaptive dictionary is empty, and the logs field has a text index created, so the default tokenizer needs to be called to perform tokenization: [POST, USA, IMAGES, space.gif]. The tokenizer is inserted into the trie based on the tokenization result, and the initial frequency is recorded. As data continues to be inserted, a trie with frequency information is formed, and then the trie is pruned to form the latest version of the tokenizer. When inserting data later, the latest version of the tokenizer is called, the tokenization result is recorded, and an index is created. The index stores the tokenizer version information.

[0105] 2. The word segmentation dictionary is a prefix tree based on the word segmentation results. In actual storage, it can be divided into a list of variable-length substrings stored by row. Therefore, VictoriaMetrics's MergeSet is selected to store the dictionary by row. The storage format is: prifix+measurementName+fieldName+dicVersion+suffix+tokens. The corresponding dictionary data is found through the specific prefix+table name+field name+word segmentation version, and then the variable-length substring list is found. For details, see the attached figure: "Dictionary Storage Structure"

[0106] The dictVersion field is used to indicate the version information of the dictionary;

[0107] When dictVersion=0 or empty, the default tokenizer is used for tokenization;

[0108] dictVersion>0, first find the corresponding version of the word segmenter, load the word segmenter into memory, use the corresponding version of the word segmenter, and the corresponding index information needs to save the dictionary version.

[0109] like Figure 2 , shows a structural block diagram of a word segmenter for implementing variable-length text based on openGemini in the present application, the word segmenter includes:

[0110] Create an index module, which is used to create a text index for the field indicator field when creating a table in openGemini;

[0111] The judgment module is used to judge whether there is an available learning word segmenter for the field indicator field where the text index is created. If so, the latest version of the learning word segmenter is selected. If not, the default word segmenter is used and the learning of the learning word segmenter is started.

[0112] Construct a dictionary tree module, which is used to perform word segmentation learning on the text data inserted into the field indicator field, and construct a dictionary tree with frequency information, wherein each node of the dictionary tree records the frequency of occurrence of a word in the text data;

[0113] A storage module, used for generating a variable-length word segmenter based on the dictionary tree, and persistently storing the variable-length word segmenter according to a combination of a table name and a field indicator field name;

[0114] The word segmentation module is used to use the variable-length word segmenter to perform word segmentation processing on the new text data inserted into the field indicator field, specifically including: starting from the first position of a log data of the new text data, searching for the longest substring matching the dictionary tree; if the matching longest substring is not found, taking the substring with a length of 1 starting from the current position as the word segmentation result.

[0115] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0116] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0117] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0118] Figure 6800 is a block diagram of an electronic device 800 shown in the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0119] Reference Figure 6 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0120] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0121] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0122] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0123] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0124] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0125] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0126] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0127] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0128] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0129] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0130] Figure 7 19 is a block diagram of a computer-readable storage medium 1900 shown in the present application. For example, the computer-readable storage medium 1900 may be provided as a server.

[0131] Reference Figure 7 , the computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0132] The computer readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer readable storage medium 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0133] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in any of the above aspects.

[0134] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0136] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0137] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0142] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A word segmentation method for variable-length text based on openGemini, characterized in that: The method comprises: Step 101, when creating a table in openGemini, create a text index for the field indicator field; Step 102, determine whether there is an available learning word segmenter for the field indicator field where the text index is created. If so, select the latest version of the learning word segmenter. If not, use the default word segmenter and start learning the learning word segmenter. Step 103, performing word segmentation learning on the text data inserted into the field indicator field, and constructing a dictionary tree with frequency information, wherein each node of the dictionary tree records the frequency of occurrence of a word in the text data; Step 104, generating a variable-length word segmenter based on the dictionary tree, and persistently storing the variable-length word segmenter according to a combination of a table name and a field indicator field name; Step 105, using the variable-length word segmenter to perform word segmentation on the new text data inserted into the field indicator field, specifically including: starting from the first position of a log data of the new text data, searching for the longest substring matching the dictionary tree, if the matching longest substring is not found, taking the substring with a length of 1 starting from the current position as the word segmentation result.

2. A method for implementing word segmentation of variable-length text based on openGemini as claimed in claim 1, characterized in that: Before step 103, the step of inserting text data into the field indicator field is also included: Parse the text index definition of the field indicator field in the table; Determine the tokenizer version to be used based on the field name and table name; Use the corresponding version of the learning tokenizer to perform token segmentation and store the results in the index storage of openGemini.

3. The word segmentation method for variable-length text based on openGemini as claimed in claim 1, characterized in that: It also includes the tokenizer version management steps: Maintain an independent version of the learning tokenizer for each combination of table and field indicator fields; When the hit rate of the dictionary tree is lower than the preset threshold, the learning of the new version of the learning word segmenter is triggered; After the new version of the learning tokenizer completes learning, the version number is stored in the metadata of openGemini.

4. The word segmentation method for variable-length text based on openGemini as claimed in claim 1, characterized in that: Also includes the query step: Parse the field indicator field in the query condition; Get the tokenizer version corresponding to the field indicator field; Use the corresponding version of the word segmenter to segment the query conditions; Search in the text index based on the word segmentation results.

5. The word segmentation method for variable-length text based on openGemini as claimed in claim 1, characterized in that: Building a dictionary tree with frequency information also includes a batch write step: After a batch of text data is written, update the frequencies of all nodes on the path from the root node to the leaf node; If there is no leaf node on the path, add a leaf node and an edge, and mark the end mark '#' on the edge; The characters at the end of the string are processed separately.

6. A word segmenter for variable-length text based on openGemini, characterized in that: The word segmenter comprises: Create an index module, which is used to create a text index for the field indicator field when creating a table in openGemini; The judgment module is used to judge whether there is an available learning word segmenter for the field indicator field where the text index is created. If so, the latest version of the learning word segmenter is selected. If not, the default word segmenter is used and the learning of the learning word segmenter is started. Construct a dictionary tree module, which is used to perform word segmentation learning on the text data inserted into the field indicator field, and construct a dictionary tree with frequency information, wherein each node of the dictionary tree records the frequency of occurrence of a word in the text data; A storage module, used for generating a variable-length word segmenter based on the dictionary tree, and persistently storing the variable-length word segmenter according to a combination of a table name and a field indicator field name; The word segmentation module is used to use the variable-length word segmenter to perform word segmentation processing on the new text data inserted into the field indicator field, specifically including: starting from the first position of a log data of the new text data, searching for the longest substring matching the dictionary tree; if the matching longest substring is not found, taking the substring with a length of 1 starting from the current position as the word segmentation result.

7. A word segmenter for implementing variable-length text based on openGemini as claimed in claim 6, characterized in that: It also includes a word segmenter version management module, specifically: Maintain an independent version of the learning tokenizer for each combination of table and field indicator fields; When the hit rate of the dictionary tree is lower than the preset threshold, the learning of the new version of the learning word segmenter is triggered; After the new version of the learning tokenizer completes learning, the version number is stored in the metadata of openGemini.

8. The word segmenter for implementing variable-length text based on openGemini as claimed in claim 6, characterized in that: It also includes query modules, specifically: Parse the field indicator field in the query condition; Get the tokenizer version corresponding to the field indicator field; Use the corresponding version of the word segmenter to segment the query conditions; Search in the text index based on the word segmentation results.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 5 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images