IK segmented word library updating method and related equipment

Abstract

One or more embodiments of the invention provide an IK segmented word library updating method and related equipment. The method comprises the following steps: in response to starting of any item, automatically querying all entries in an IK database table and loading the entries into a cache; the IK word segmentation device loads the vocabulary entries in the cache to an IK word library in a data stream form through a remote calling interface; wherein the IK database table, the cache and the IK word bank are deployed on a server side. According to the method, the real-time performance and the flexibility of the system are improved, it is ensured that the word bank is always kept in the latest state, the waiting time of manual periodic updating is shortened, and the overall efficiency is improved through a simplified system architecture.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of computer technology, and in particular, to an IK word segmentation library updating method and related devices. Background Art

[0002] In modern information retrieval systems, especially when processing Chinese text, word segmentation accuracy is crucial to ensure the relevance of retrieval results. As a widely used full-text search engine, Elasticsearch (ES) integrates with the IK word segmenter to improve the efficiency of Chinese text processing. The IK word segmenter relies on its word library for word segmentation, so updating the IK word library in the ES system is an important technical task.

[0003] The IK tokenizer guides the tokenization process through its lexicon, which contains a large number of words and domain-specific terms that directly affect the performance of the search engine. As language and vocabulary are constantly evolving, the emergence of new words requires frequent updates to the lexicon to capture industry terms, company names, or product names.

[0004] In the related art, the ES system needs to be restarted to update the IK vocabulary, which will lead to instability of the ES system and high operation and maintenance costs. Summary of the invention

[0005] In view of this, an object of one or more embodiments of the present application is to propose an IK word segmentation library updating method and related devices to solve the problem.

[0006] Based on the above purpose, one or more embodiments of the present application provide an IK word segmentation library update method, including:

[0007] In response to any project startup, automatically query all entries in the IK database table and load the entries into the cache;

[0008] The IK word segmenter loads the entries in the cache into the IK word library in the form of data stream through a remote call interface;

[0009] The IK database table, the cache and the IK vocabulary are deployed on the server side.

[0010] Optionally, it also includes:

[0011] In response to receiving the change query request sent by the client, the server detects whether there is a word change in the client according to the status identifier of the header information in the change query request;

[0012] In response to determining that the client has a word change, sending an upload instruction to the client;

[0013] In response to receiving the upload instruction sent by the server, the client uploads the changed entry to the server via the remote call interface;

[0014] The server side updates the IK database table with the changed terms to ensure that the IK word segmenter can dynamically load the latest terms.

[0015] Optionally, it also includes:

[0016] In response to detecting that there is a change in an entry in the database table, updating the cache according to the database table;

[0017] According to the updated cache, updating the word library status value on the server side;

[0018] In response to the change in the state value of the word library, the remote call interface is triggered, and the IK word segmenter loads the entries in the cache into the IK word library in the form of data stream.

[0019] Optionally, the vocabulary status value is represented by an Etag entity tag header and a Last-Modified header field.

[0020] Optionally, the change query request is generated according to an HTTP request protocol, and the change query request includes an Etag entity tag header and a Last-Modified header field.

[0021] Optionally, the PostConstruct annotation feature in Java is used to automatically query all entries in the IK database table and load the entries into the cache when the project is started.

[0022] Based on the same inventive concept, one or more embodiments of the present application further provide an IK word segmentation library updating system, including:

[0023] The server is configured to automatically query all entries in the IK database table and load the entries into a cache in response to any project startup;

[0024] An IK word segmenter is configured to load the entries in the cache into the IK word library in the form of data stream through a remote call interface;

[0025] The IK database table, the cache and the IK vocabulary are deployed on the server side.

[0026] Optionally, also include the client:

[0027] The server is further configured to, in response to receiving a change query request sent by a client, detect whether the client has a term change according to a status identifier in the header information of the change query request; in response to determining that the client has a term change, send an upload instruction to the client; and

[0028] Taking the changed terms to update the IK database table to ensure that the IK word segmenter can dynamically load the latest terms;

[0029] The client is configured to upload the changed entry to the server via a remote call interface in response to receiving an upload instruction sent by the server.

[0030] Based on the same inventive concept, one or more embodiments of the present application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an IK word segmentation library updating method as described in any one of the above items is implemented.

[0031] Based on the same inventive concept, one or more embodiments of the present application further provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute any of the above-mentioned IK word segmentation library updating methods.

[0032] From the above description, it can be seen that the IK word segmentation library update method provided in one or more embodiments of the present application, in response to the start of any project, automatically queries all entries in the IK database table and loads the entries into the cache; the IK word segmenter loads the entries in the cache into the IK word library in the form of a data stream through a remote call interface; wherein the IK database table, the cache and the IK word library are deployed on the server side.

[0033] When the project is started, this application updates the IK vocabulary through a remote call interface to ensure that the ES service is implemented based on the latest IK vocabulary in real time, avoids frequent restarts of the ES system, and improves the ES service experience.

[0034] The IK word segmentation library updating system, electronic device and computer-readable storage medium provided in the present application can implement the steps of the above-mentioned IK word segmentation library updating method, and therefore also have the beneficial effects of the above-mentioned IK word segmentation library updating method. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic diagram of a flow chart of a method for updating an IK word segmentation database according to one or more embodiments of the present application;

[0037] Figure 2 A schematic diagram of the structure of an IK word segmentation library update system in one or more embodiments of the present application;

[0038] Figure 3 This is a schematic diagram of the hardware structure of an electronic device according to one or more embodiments of the present application. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0040] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in one or more embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0041] As described in the background technology, the related art uses the method of exporting the client IK vocabulary (including the word segmentation vocabulary and the stop word vocabulary) to a file with a dic suffix, and then uploading the file to the ik configuration under the server-side ES service. In this case, the ES service is required for the configuration to take effect.

[0042] However, frequent export of word segmentation files will lead to occasional partial loss of data, which is risky. Since the IK vocabulary is updated frequently and the update time of the IK vocabulary is not fixed, the above technical solution has the problems of untimely updates, high maintenance costs and difficult updates. This further leads to the problem of poor ES service experience.

[0043] Therefore, this application proposes an IK vocabulary update method, which updates the IK vocabulary through a remote call interface, avoiding the steps of file export and ES system restart, improving the IK vocabulary update efficiency, enhancing the ES system stability, and further improving the user experience.

[0044] refer to Figure 1 The IK word segmentation library updating method of one or more embodiments of the present application includes the following steps:

[0045] S101: In response to any project being started, all entries in the IK database table are automatically queried and loaded into a cache;

[0046] S102: The IK word segmenter loads the entries in the cache into the IK word library in the form of data stream through a remote call interface; wherein the IK database table, the cache and the IK word library are deployed on the server side.

[0047] In step S102, the IK word segmenter loads entries from the cache to the IK word library in the form of data stream through the remote call interface, making the update process simple and easy without complex configuration or manual intervention.

[0048] In an embodiment of the present application, the PostConstruct annotation feature in Java can be used to automatically query all entries in the IK database table and load the entries into the cache when the project is started.

[0049] In the technical solution of this application, the system automatically queries the IK database table and loads entries into the cache when the project is started, without manual intervention, eliminating the need to manually trigger data loading, reducing the possibility of human error, and reducing maintenance workload, allowing technicians to focus on higher-value tasks. Users only need to initialize the project to complete the vocabulary update.

[0050] In addition, loading the vocabulary at project startup improves the real-time and flexibility of the system, ensures that the vocabulary is always up-to-date, reduces the waiting time for manual periodic updates, and the simplified system architecture improves overall efficiency.

[0051] The IK database table in step S101 can be updated according to the instructions of the client. Specifically, in the embodiment of the present application, the method for updating the IK database table includes: the server side responds to receiving the change query request sent by the client, and detects whether there is a change in the entry of the client according to the status identifier of the header information in the above change query request; in response to determining that there is a change in the entry of the client, an upload instruction is sent to the above client; the client responds to receiving the upload instruction sent by the above server, and uploads the changed entry to the above server through the remote call interface; the above server adopts the above changed entry to update the above IK database table to ensure that the IK word segmenter can dynamically load the latest entry.

[0052] The status identifier of the header information in the above query request may be the Etag entity tag header and the Last-Modified header field under the HTTP protocol.

[0053] ETag is an effective way to confirm resource versions, which is achieved through a unique identifier generated by the server for each resource. When a client requests a resource for the first time, the server returns the resource and its corresponding ETag. Subsequently, when the client requests the same resource when the cache expires or the resource is updated, it includes this ETag in the request header (usually through the If-None-Match header). After receiving the request, the server compares this ETag with the ETag of the current resource. If the ETags are consistent, it means that the resource has not changed, and the server returns a 304 Not Modified status instead of returning the resource data, thereby saving bandwidth and reducing loading time. If the ETags are different, it means that the resource has been updated, and the server returns the latest resource and the new ETag. This mechanism ensures that the client is always processing the latest version of the resource by effectively verifying the version of the resource, while improving network performance.

[0054] Last-Modified helps the client confirm the latest version of a resource by recording the last modification time of the resource. When the client first obtains a resource, the server response header contains a Last-Modified timestamp. The client caches the timestamp and sends it to the server via the If-Modified-Since header in subsequent requests. The server compares the timestamp with the current modification time of the resource. If it has not changed, it returns a 304 Not Modified status to avoid re-downloading the same content, saving bandwidth and loading time. If the resource has been updated, the latest resource data and its updated Last-Modified time are returned. In this way, Last-Modified effectively supports resource version control and cache validation.

[0055] Through the above operations, the synchronization between the client and the server can be guaranteed without the need to update the IK database table on the server through complex and tedious manual operations.

[0056] In order to ensure that the latest entries are stored in the IK vocabulary, when the IK word segmenter is in use, the system detects whether the content of the above database table has changed. The system can perform regular detection according to a preset time interval, or preset trigger conditions. This application does not make specific restrictions on the detection conditions.

[0057] In an embodiment of the present application, in response to detecting that there are changes in entries in the above-mentioned database table, the above-mentioned cache is updated according to the above-mentioned database table; according to the updated above-mentioned cache, the vocabulary status value on the server side is updated; in response to the change in the above-mentioned vocabulary status value, the above-mentioned remote call interface is triggered, and the above-mentioned IK word segmenter loads the entries in the above-mentioned cache into the IK vocabulary in the form of a data stream.

[0058] In the above steps, the IK vocabulary content may be updated by reloading all the data in the cache to the IK vocabulary, or only the updated part of the content may be uploaded to the IK vocabulary.

[0059] By calling the remote interface to update data, the IK word segmenter and the ES system are allowed to share data and functions, promote modularity and scalability, and enhance the interoperability and flexibility of the application.

[0060] Transmitting data in the form of data streams is an efficient and scalable way of data processing. Streaming can achieve gradual processing of data and instant response, reducing latency and memory usage.

[0061] It can be understood that the method can be executed by any device, equipment, platform, or device cluster having computing and processing capabilities.

[0062] It should be noted that the method of one or more embodiments of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of one or more embodiments of the present application, and the multiple devices will interact with each other to complete the described method.

[0063] It should be noted that the above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0064] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an IK word segmentation library update system. Figure 2 As shown, the system includes:

[0065] The server 11 is configured to automatically query all entries in the IK database table and load the above entries into the cache in response to any project startup;

[0066] The IK word segmenter 12 is configured to load the entries in the cache into the IK word library in the form of data stream via a remote call interface;

[0067] The IK database table, the cache and the IK vocabulary are deployed on the server side.

[0068] The above-mentioned IK word segmenter 12 can be deployed in Elasticsearch plug-ins, Java applications, natural language processing projects, Web applications, etc.

[0069] The embodiment of the present application further includes a client 13:

[0070] The server 11 is further configured to, in response to receiving a change query request sent by the client 13, detect whether the client 13 has a term change according to a status identifier in the header information of the change query request; in response to determining that the client 13 has a term change, send an upload instruction to the client 13; and,

[0071] The above-mentioned changed terms are used to update the above-mentioned IK database table to ensure that the IK word segmenter can dynamically load the latest terms;

[0072] The client 13 is configured to, in response to receiving an upload instruction sent by the server 11 , upload the changed entry to the server 11 via a remote call interface.

[0073] In an embodiment of the present application, the server 11 is further configured to update the cache according to the database table in response to detecting that there is a change in the entry in the database table; and update the server-side word library status value according to the updated cache;

[0074] In response to the change in the state value of the thesaurus, the remote call interface is triggered, and the IK word segmenter loads the entries in the cache into the IK thesaurus in the form of data stream.

[0075] In an embodiment of the present application, the above-mentioned vocabulary status value is represented by the Etag entity tag header and the Last-Modified header field.

[0076] In an embodiment of the present application, the above-mentioned change query request is generated according to the HTTP request protocol, and the above-mentioned change query request includes an Etag entity tag header and a Last-Modified header field.

[0077] The embodiment of the present application utilizes the PostConstruct annotation feature in Java to automatically query all entries in the IK database table and load the above entries into the cache when the above project is started.

[0078] For the convenience of description, the above devices are described in terms of functions and modules. Of course, when implementing one or more embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0079] The apparatus of the above-mentioned embodiment is used to implement the corresponding method in the above-mentioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0080] Figure 3 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0081] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0082] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0083] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0084] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0085] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0086] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiment of the present application, and does not necessarily include all the components shown in the figure.

[0087] The electronic device of the above embodiment is used to implement the corresponding method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0088] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0089] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0090] In addition, to simplify the description and discussion, and in order not to make one or more embodiments of the present application difficult to understand, the known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making one or more embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present disclosure, it is obvious to those skilled in the art that one or more embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0091] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0092] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of this disclosure.

Claims

1. A method for updating an IK word segmentation database, characterized in that: include: In response to any project startup, automatically query all entries in the IK database table and load the entries into the cache; The IK word segmenter loads the entries in the cache into the IK word library in the form of data stream through a remote call interface; The IK database table, the cache and the IK vocabulary are deployed on the server side.

2. The method according to claim 1, characterized in that Also includes: In response to receiving the change query request sent by the client, the server detects whether there is a word change in the client according to the status identifier of the header information in the change query request; In response to determining that the client has a word change, sending an upload instruction to the client; In response to receiving the upload instruction sent by the server, the client uploads the changed entry to the server via the remote call interface; The server side updates the IK database table with the changed terms to ensure that the IK word segmenter can dynamically load the latest terms.

3. The method according to claim 2, characterized in that Also includes: In response to detecting that there is a change in an entry in the database table, updating the cache according to the database table; According to the updated cache, updating the word library status value on the server side; In response to the change in the state value of the word library, the remote call interface is triggered, and the IK word segmenter loads the entries in the cache into the IK word library in the form of data stream.

4. The method according to claim 3, characterized in that The word library status value is represented by the Etag entity tag header and the Last-Modified header field.

5. The method according to claim 4, characterized in that The change query request is generated according to the HTTP request protocol, and the change query request includes an Etag entity tag header and a Last-Modified header field.

6. The method according to claim 1, characterized in that By utilizing the PostConstruct annotation feature in Java, all entries in the IK database table are automatically queried and loaded into the cache when the project is started.

7. An IK word segmentation library updating system, characterized in that: include: The server is configured to automatically query all entries in the IK database table and load the entries into a cache in response to any project startup; An IK word segmenter is configured to load the entries in the cache into the IK word library in the form of data stream through a remote call interface; The IK database table, the cache and the IK vocabulary are deployed on the server side.

8. The system according to claim 7, characterized in that Also includes the client: The server is further configured to, in response to receiving a change query request sent by a client, detect whether the client has a term change according to a status identifier in the header information of the change query request; in response to determining that the client has a term change, send an upload instruction to the client; as well as, Taking the changed terms to update the IK database table to ensure that the IK word segmenter can dynamically load the latest terms; The client is configured to upload the changed entry to the server via a remote call interface in response to receiving an upload instruction sent by the server.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute any one of claims 1 to 7.

Images