Asynchronous service information retrieval and index management method and system

Abstract

The invention discloses an asynchronous service information retrieval and index management system and method. The system comprises an access management module which is connected with at least one application system and is used for acquiring identification information corresponding to unstructured data in the application system, storing the identification information in a message queue, acquiring the unstructured data of the application system according to the message queue and sending the unstructured data to a data processing module; the data processing module is connected with the access management module, and is used for performing text extraction on the unstructured data to obtain to-be-written data and writing the to-be-written data into the data storage module; and the data storage module is connected with the data processing module and is used for performing word segmentation processing on the to-be-written data according to the basic word bank and the professional word bank to obtain to-be-stored data, storing the to-be-stored data and constructing a target index corresponding to the application system. According to the technical scheme provided by the embodiment of the invention, the retrieval service is flexible and extensible.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and in particular to an asynchronous service-based information retrieval and index management method and system. Background Art

[0002] With the development of the big data era and text processing technology, the requirements for text retrieval are getting higher and higher. Simply relying on databases for fuzzy matching can no longer meet business needs. Full-text retrieval technology uses word segmentation technology to implement its functions, but the standard word library cannot cover the professional word libraries of various industries, which greatly reduces the retrieval effect. At the same time, due to historical reasons, the large amount of unstructured data in each application system cannot be efficiently retrieved.

[0003] Therefore, how to process the unstructured data of each application system and provide efficient, flexible and scalable retrieval services for the application system is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides an asynchronous service-oriented information retrieval and index management method and system to improve the flexibility and scalability of information retrieval.

[0005] According to one aspect of the present invention, an asynchronous service-oriented information retrieval and index management system is provided, comprising an access management module, a data processing module and a data storage module; wherein:

[0006] The access management module is connected to at least one application system, and is used to obtain identification information corresponding to unstructured data in the application system, store the identification information in a message queue, obtain the unstructured data of the application system according to the message queue, and send the unstructured data to the data processing module;

[0007] The data processing module is connected to the access management module, and is used to receive the unstructured data sent by the access management module, perform text extraction on the unstructured data to obtain data to be written, and write the data to be written into the data storage module;

[0008] The data storage module is connected to the data processing module, and is used to receive the data to be written written by the data processing module, perform word segmentation processing on the data to be written according to the basic vocabulary and the professional vocabulary to obtain the data to be stored, and store the data to be stored and construct a target index corresponding to the application system.

[0009] According to another aspect of the present invention, an asynchronous service-based information retrieval and index management method is provided, which is applied to an asynchronous service-based information retrieval and index management system, including an access management module, a data processing module and a data storage module, wherein the access management module is connected to at least one application system, the data processing module is connected to the access management module, and the data storage module is connected to the data processing module; wherein the asynchronous service-based information retrieval and index management method includes:

[0010] Acquire identification information corresponding to the unstructured data in the application system, store the identification information in a message queue, acquire the unstructured data of the application system according to the message queue, and send the unstructured data to the data processing module;

[0011] Receiving the unstructured data sent by the access management module, performing text extraction on the unstructured data to obtain data to be written, and writing the data to be written into the data storage module;

[0012] The data to be written written by the data processing module is received, the data to be written is segmented according to the basic vocabulary and the professional vocabulary to obtain the data to be stored, and the data to be stored is stored and a target index corresponding to the application system is constructed.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor;

[0015] and a memory communicatively coupled to the at least one processor;

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the asynchronous service-oriented information retrieval and index management method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the asynchronous service-based information retrieval and index management method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention is to obtain the identification information corresponding to the unstructured data in the application system, store the identification information in the message queue, obtain the unstructured data of the application system according to the message queue, and send the unstructured data to the data processing module; receive the unstructured data sent by the access management module, extract the text of the unstructured data to obtain the data to be written, and write the data to be written to the data storage module; receive the data to be written written by the data processing module, perform word segmentation processing on the data to be written according to the basic vocabulary and the professional vocabulary to obtain the data to be stored, store the data to be stored and construct the target index corresponding to the application system. The problem of low flexibility and scalability of the existing retrieval service is solved, the unstructured data of each application system is unified into one platform, and the incremental unstructured data generated by each application system is transmitted regularly. For new application systems, expansion access can be carried out at any time, and integrated full-text retrieval is realized. Every user can search, locate matching files, and locate the application system where they are located, which is convenient for users and maximizes the value of unstructured data. At the same time, a service interface is provided to each application system, so that the application system can integrate the platform and use the full-text search capability to retrieve all structured data in its own system and locate specific business scenarios, greatly improving the convenience of the system.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 This is a structural diagram of an asynchronous service-oriented information retrieval and index management system provided according to Embodiment 1 of the present invention;

[0022] Figure 2 This is an architecture diagram of an asynchronous service-oriented information retrieval and index management system provided according to Embodiment 2 of the present invention;

[0023] Figure 3 This is a diagram illustrating the system access method;

[0024] Figure 4 It is a data processing flow chart;

[0025] Figure 5 Design graphs for unstructured data processing;

[0026] Figure 6 A hierarchical diagram for asynchronous service-based information retrieval and index management systems;

[0027] Figure 7 To process the logic diagram;

[0028] Figure 8 Execute logic diagrams for message queues and consumer agents;

[0029] Fig. 9 This is a flow chart of an asynchronous service-based information retrieval and index management method provided by Embodiment 3 of the present invention;

[0030] Fig.10 It is a schematic diagram of the structure of an electronic device provided according to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 is a structural diagram of an asynchronous service-oriented information retrieval and index management system provided according to the first embodiment of the present invention. This embodiment can be applied to information retrieval in a business system, such as Figure 1As shown, the asynchronous service-oriented information retrieval and index management system includes an access management module 110, a data processing module 120 and a data storage module 130, wherein:

[0035] The access management module 110 is connected to at least one application system, and is used to obtain identification information corresponding to unstructured data in the application system, store the identification information in a message queue, obtain the unstructured data of the application system according to the message queue, and send the unstructured data to the data processing module;

[0036] The data processing module 120 is connected to the access management module, and is used to receive the unstructured data sent by the access management module, extract text from the unstructured data to obtain data to be written, and write the data to be written into the data storage module;

[0037] The data storage module 130 is connected to the data processing module, and is used to receive the data to be written written by the data processing module, perform word segmentation processing on the data to be written according to the basic vocabulary and the professional vocabulary to obtain the data to be stored, and store the data to be stored and construct a target index corresponding to the application system.

[0038] In this embodiment, the application system refers to the business system. There is a large amount of unstructured data in the business system. In order to manage and retrieve the data of the business system, the application system is connected to the access management module; there are multiple types and numbers of application systems, and the access management module can be connected to multiple application systems. The asynchronous service-oriented information retrieval and index management system in this embodiment can provide information storage and retrieval services for multiple application systems at the same time. The identification information of the unstructured data in the application system is obtained through the access management module. The identification information is used to distinguish different unstructured data. The identification information of the obtained unstructured data is stored in the message queue. The unstructured data of the application system is obtained based on the message queue, and the unstructured data is sent to the data processing module. The data processing module is a module for processing unstructured data, which mainly converts unstructured data into structured data, such as the data obtained after text extraction processing of unstructured data as the data to be written, and then the data to be written is written to the data storage module. The data storage module is mainly a module for storing data to be written and establishing indexes. The data storage module can be understood as a server that provides data storage and retrieval, such as an ES server (full-text retrieval server); the data storage module includes a basic vocabulary and a professional vocabulary, and the data to be written is segmented based on the basic vocabulary and the professional vocabulary. The data obtained after processing is the data to be stored. The data to be stored is stored in the server and an index corresponding to the data to be stored, i.e., the target index, is established.

[0039] On the basis of the above technical solution, the access management module includes a log submodule, an interface submodule, an intermediate library submodule and a queue consumption submodule; the log submodule is connected to the application system, and is used to collect log information in the application system and store the log information in a message queue; wherein the log information includes identification information of the unstructured data; the interface submodule is connected to the application system, and is used to call a preset interface to obtain identification information of the unstructured data in the application system when receiving a call request from the application system, and store the identification information in the message queue; the intermediate library submodule is connected to the application system, and is used to receive identification information of the unstructured data synchronized by the application system, and store the identification information in the message queue; the queue consumption submodule is connected to the data processing module, and is used to obtain unstructured data corresponding to the log information or identification information from the application system in sequence according to the storage order of the log information or identification information in the message queue, and send the unstructured data to the data processing module.

[0040] In this embodiment, a connection with the application system is established through three access methods. The first is that the application system records the identification information of the unstructured data in the log of the application system, and the log submodule in the access management module can obtain the log information in the application system and store the log information in the message queue; the second is that the application system pushes the identification information of the unstructured data to the access management module through the preset interface in the interface submodule, and stores the identification information in the message queue; the third is that the application system regularly updates synchronously, updates the identification information of the unstructured data to the intermediate library submodule, and the intermediate library submodule further stores the identification information of the unstructured data in the message queue. In this embodiment, the queue consumption submodule is used to pull data from the application system. The queue consumption submodule obtains the unstructured data corresponding to the log information or identification information from the application system in sequence according to the storage order of the log information or identification information in the message queue, and sends the unstructured data to the data processing module. It should also be noted that the access management module in this embodiment supports a variety of different access methods, and the application system can choose different methods for access according to the actual situation, so as to minimize the transformation and have no intrusion on the application system.

[0041] On the basis of the above embodiment, the data processing module includes an extraction submodule and a writing submodule; wherein the extraction submodule is connected to the access management module, used to receive the unstructured data, and based on the text extraction method corresponding to the unstructured data, performs text extraction on the unstructured data to obtain the data to be written, and sends it to the writing submodule; the writing submodule is connected to the extraction module and the data storage module, used to receive the data to be written, and write the data to be written into the data storage module.

[0042] In this embodiment, the extraction submodule refers to a module used to extract text data from unstructured data. In actual applications, different unstructured data correspond to different text extraction methods. The extraction submodule can perform text extraction on the unstructured data according to the text extraction method that matches the unstructured data to obtain the data to be written and send the data to be written to the writing submodule; the writing submodule is a module that provides data writing function. The data to be written is written to the data storage module through the writing submodule, and the data storage module establishes a corresponding index based on the received data to be written.

[0043] On the basis of the above embodiment, the extraction submodule includes a text extraction unit, an identification extraction unit and a multimedia extraction unit; wherein the text extraction unit is used to receive the document-type unstructured data sent by the access management module, perform text extraction processing on the document-type unstructured data, and obtain the data to be written corresponding to the document-type unstructured data; the identification extraction unit is used to receive the image-type unstructured data sent by the access management module, perform optical character recognition processing on the image-type unstructured data, and obtain the data to be written corresponding to the image-type unstructured data; the multimedia extraction unit is used to receive the multimedia-type unstructured data sent by the access management module, perform text conversion processing on the multimedia-type unstructured data, and obtain the data to be written corresponding to the multimedia-type unstructured data.

[0044] In a preferred embodiment, in order to adapt to different forms of unstructured data, the extraction submodule includes a text extraction unit, an identification extraction unit and a multimedia extraction unit. In the application scenario of this embodiment, when the received unstructured data belongs to the unstructured data of the document class, such as Word, PPT, Excel, html web pages, such unstructured data is the unstructured data of the document class, and the text extraction unit directly extracts the above-mentioned document class unstructured data. For another example, when the unstructured data is a PDF file or a picture, it belongs to the image class unstructured data, and the image class unstructured data can be processed by the identification extraction unit, such as performing unstructured data OCR (Optical Character Recognition, optical character recognition) extraction processing on the image class; for another example, if the unstructured data is the unstructured data of the multimedia class, such as audio and video, by inputting such unstructured data into the multimedia extraction unit, the multimedia extraction unit converts the audio and video therein into text, and the text extraction processing of the multimedia class unstructured data can be realized.

[0045] It should also be noted that when the extraction submodule extracts and processes unstructured data, if the application system allows the data to be stored locally, the unstructured data can be stored for subsequent search and display. For application systems that do not allow it, the unstructured data is only extracted for text and then stored for word segmentation. The source files of the unstructured data are only cached during processing and are not stored after processing.

[0046] In a preferred embodiment, the extraction submodule further includes an extraction configuration unit, wherein the extraction configuration unit is used to configure an extraction unit corresponding to the unstructured data based on the type of the unstructured data. The extraction unit corresponding to the unstructured data is configured based on the extraction configuration unit. For example, when a certain type of unstructured data is newly added to the current application system, but the current extraction unit cannot realize the extraction processing of this type of structured data, an extraction unit matching the type of the unstructured data can be newly added through the extraction configuration unit. That is, the extraction capability can be expanded according to the increase in the type of unstructured data that needs to be processed, and the extraction of various types of unstructured data can be completed.

[0047] On the basis of the above technical solution, the data storage module includes a word segmentation submodule, an index establishment submodule and an index update submodule; wherein the word segmentation submodule is used to, after receiving the data to be written, perform word segmentation processing on the data to be written according to the basic vocabulary and the professional vocabulary to obtain the data to be stored, and store the data to be stored in the data cluster; the index establishment submodule is used to establish a target index corresponding to the application system based on the data to be stored; the index update submodule is used to update the target index based on the newly added data to be stored when the newly added data to be stored is retrieved.

[0048] In this embodiment, the word segmentation module can be understood as a word segmenter, which performs word segmentation processing on the data to be written to obtain the corresponding data to be stored. The data cluster can be an ES data cluster, and the data to be stored is stored through the pre-built ES data cluster. In this implementation, the word segmentation processing of the data to be written is based on the basic vocabulary and the professional vocabulary. The basic vocabulary contains basic vocabulary and sentences, etc., and the professional vocabulary contains professional vocabulary, such as professional vocabulary related to the business processed by the application system. The data to be written is segmented according to the basic vocabulary and the professional vocabulary, that is, the sentences therein are divided into multiple words as the data to be stored; further, the target index corresponding to the data to be stored is established through the index establishment submodule in this embodiment. In another preferred embodiment scenario, when newly added data to be stored is detected, the target index is updated in time based on the newly added data to be stored to improve the accuracy of subsequent retrieval.

[0049] Based on the above embodiment, the asynchronous service-oriented information retrieval and index management system also includes a data retrieval module; the data retrieval module is connected to the data storage module and is used to determine the target retrieval result based on the information to be retrieved, the target index and the data to be stored stored in the data storage module when receiving the information to be retrieved; wherein the target retrieval result includes the retrieval result data corresponding to the information to be retrieved and the application system information to which the retrieval result data belongs.

[0050] In this embodiment, the data retrieval module is used to search for content that matches the information to be retrieved in the data to be stored. For example, through the information to be retrieved, the category or storage address to which the information to be retrieved belongs is first searched in the target index, and then the specific content corresponding to the information to be retrieved is searched from this category as the target retrieval result. Preferably, the information to be retrieved includes the content to be retrieved and the range filtering information. After obtaining the target retrieval result, the retrieval result data and the application system to which the retrieval result data belongs can be determined according to the target retrieval result. For example, if the retrieval result is data a and the application system information is system A, the application system information can be the identification of the application system, thereby tracing the target retrieval result.

[0051] On the basis of the above embodiment, the data retrieval module includes a first retrieval sub-module and a second retrieval sub-module, wherein the first retrieval sub-module is connected to the data storage module and is used to determine the retrieval authority of the information to be retrieved when the information to be retrieved is received, and determine the target index of at least one of the application systems based on the retrieval authority, and determine the target retrieval result based on at least one of the target indexes; the second retrieval sub-module is connected to the application system and is used to determine the target retrieval result based on the target index of the application system corresponding to the information to be retrieved when the information to be retrieved is received, and send the target retrieval result to the application system.

[0052] In a preferred embodiment, the first search submodule refers to a unified search entry. The user inputs the corresponding information to be searched through the interface, and can find the matching information to be searched in the data to be stored in each application system, provided that the user has the corresponding search authority, for example, the search authority of the information to be searched is the search authority of all systems. After the first search module receives the information to be searched input by the user, the information to be searched can be sent to the data storage module, and then the content matching the information to be searched can be searched in the data to be stored corresponding to all application systems in the data storage module, and the search results can be fed back to the user after the search results are obtained. For another example, if the search authority is the search authority of application systems A and B, the information to be searched can be searched through the target index of systems A and B.

[0053] In another embodiment, the second search submodule is a search service interface for application systems, providing unified standard query services for each application system, and the parameters are search content and range filtering information, and the service returns the filtered matching search results to the application system. That is, only the data to be stored corresponding to the current application system can be retrieved through the second search submodule, thereby improving data security.

[0054] Based on the above technical solution, the data retrieval module also includes a synonym determination submodule, which is used to determine the synonym search result corresponding to the target search result after determining the target search result, and update the target search result based on the synonym search result. In this embodiment, by continuously maintaining and improving synonyms, the search results required by the user can be returned to the greatest extent during the search.

[0055] The technical solution of the embodiment of the present invention is to obtain the identification information corresponding to the unstructured data in the application system, store the identification information in the message queue, obtain the unstructured data of the application system according to the message queue, and send the unstructured data to the data processing module; receive the unstructured data sent by the access management module, extract the text of the unstructured data to obtain the data to be written, and write the data to be written to the data storage module; receive the data to be written written by the data processing module, perform word segmentation processing on the data to be written according to the basic vocabulary and the professional vocabulary to obtain the data to be stored, store the data to be stored and construct the target index corresponding to the application system. The problem of low flexibility and scalability of the existing retrieval service is solved, the unstructured data of each application system is unified into one platform, and the incremental unstructured data generated by each application system is transmitted regularly. For new application systems, expansion access can be carried out at any time, and integrated full-text retrieval is realized. Every user can search, locate matching files, and locate the application system where they are located, which is convenient for users and maximizes the value of unstructured data. At the same time, a service interface is provided to each application system, so that the application system can integrate the platform and use the full-text search capability to retrieve all structured data in its own system and locate specific business scenarios, greatly improving the convenience of the system.

[0056] Embodiment 2

[0057] Figure 2 This is an architecture diagram of an asynchronous service-oriented information retrieval and index management system provided according to Embodiment 2 of the present invention. This embodiment is a preferred embodiment among the above embodiments, and its specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiments are not repeated here.

[0058] like Figure 2 As shown, the system includes: application system access management, message queue and consumer agent management, unstructured data processing center, ES cluster construction and vocabulary and word segmentation management, and retrieval service management.

[0059] The technical solution disclosed in the embodiment of the present invention has been applied to a large energy enterprise, unifying the unstructured data of various application systems into one platform, and regularly transmitting the incremental unstructured data generated by each business system. New systems can be expanded and accessed at any time, realizing integrated full-text retrieval. Every employee can search, locate matching files, and locate the system where they are located, which is convenient for users and maximizes the value of unstructured data. At the same time, a service interface is provided to each business system, so that the business system integrates the platform, uses the full-text retrieval capability to retrieve all structured data in its own system, locates specific business scenarios, and greatly improves the convenience of the system.

[0060] In order to implement the above method, the present invention designs the retrieval platform system including application system access management, message queue and consumer agent management, unstructured data processing center, ES cluster construction and word library and word segmentation management, and retrieval service management. Through the collaboration of various modules, the full-text retrieval service platform can be accessed with minimal or even no changes to the application system, so that the application system can be flexibly and quickly accessed and services can be provided without intruding. The specific scheme design is now described.

[0061] 1. Application system access management

[0062] Application system access is to establish data interaction between the search platform and the application system, including three optional access methods, refer to Figure 3 , which is a diagram illustrating the system access method. 1) Log collection method: the application only needs to record the identification information of the data that needs full-text retrieval in the log, and the platform pushes the information to the message queue through real-time collection. 2) The platform provides an API service interface. The application pushes the agreed identifier to the platform by calling the interface, and the platform pushes it into the message queue. 3) The platform provides an intermediate library. The application can directly synchronize incremental updates regularly through the database, and the platform regularly pushes incremental data into the message queue. The application system can choose different methods and platforms for access according to the actual situation, so as to minimize the transformation and non-intrusion to the application system.

[0063] 2. Message queue and consumer agent management

[0064] Message queue and consumer agent management is used to manage the pulling of data from the application system. When the message queue receives a task, it will consume the queue information through the consumer agent. The proxy service will obtain the corresponding unstructured data and other corresponding unified standard fields (including fields used for search range filtering) from the application system according to the identifier. At the same time, if there is personalized information that needs to be associated, it can be returned to the platform together, and the platform will use it in the unified retrieval service.

[0065] 3. Unstructured data processing center

[0066] See also Figure 4 , is a data processing flow chart. The unstructured data processing center is used to extract text from the unstructured data obtained from the application system, store it in ES to create an index, and finally realize the full-text retrieval function. Unstructured data processing is divided into the following categories: 1) Word, PPT, Excel, html web pages, etc. can directly extract text data from them, and then process them and put them into the warehouse. 2) PDF pictures, etc. need to perform OCR recognition to extract text content and then process them into the warehouse. 3) Multimedia content such as audio and video, convert the audio into text and then store it into the warehouse. The processing center can expand the extraction capability according to the increase in the type of unstructured data that needs to be processed, and complete the extraction of various types of unstructured data. For applications that allow data to be stored locally, it can be stored for subsequent search and display. For systems that do not allow it, the platform only extracts the text and stores it after word segmentation. The source file is only cached during processing, and the source file is not stored after processing. See Figure 5 ,Design graph for unstructured data processing.

[0067] 4. ES cluster construction and vocabulary and word segmentation management

[0068] As a data storage center, the ES cluster is used to segment data and create indexes to achieve full-text search capabilities. The ES cluster needs to use a Chinese word segmenter plug-in and maintain a set of basic vocabulary and extended professional vocabulary. ES uses this vocabulary as the basis for segmentation and indexing, so that it can meet the accuracy of full-text search using professional terms in different professional fields. At the same time, it is also necessary to establish a set of synonym vocabulary, and through continuous maintenance and improvement of synonyms, the maximum amount of search results required by users can be returned during search.

[0069] The storage of indexes supports multi-tenancy. The data of each access system is stored independently. As the vocabulary continues to improve, the index will be rebuilt on a rolling basis to continuously improve the accuracy of search results.

[0070] 5. Retrieval service management

[0071] The search service is divided into a unified search entry and a search service interface for application systems. The unified search entry is a full search within the platform, which is only available to users who have the authority to search across systems. Users can use this to conduct a full-range search, locate files and thus locate the data of the application system, complete the search and trace the source. The search service interface for application systems provides unified standard query services for each application system. The parameters are the search content and range filtering information. The service will return the filtered matching search results to the application system.

[0072] Based on the current situation of the dramatic increase in data volume in various application systems, it has become difficult for systems in various business fields to search for unstructured data. If each system independently builds a full-text search platform, the cost is very high and the data cannot be connected. This design can achieve an integrated full-text search service platform with low cost and high efficiency, connect the unstructured data of various systems, and is flexible and scalable. It has low cost, short construction period and good effect for accessing new systems.

[0073] The management of ES cluster and word library fully guarantees the stability and high availability of the system foundation. The results of user searches can be returned within 1 second. As the amount of data increases and the number of access systems increases, the cluster can be dynamically expanded to carry higher data volumes and search requirements. Word segmentation and synonym management enable users' searches to be matched more accurately, greatly improving the user experience. The search service provides a standard service interface, which is very friendly to developers and can quickly meet the full-text search requirements of the application system. Message queue and consumer agent management enable various operations to be executed asynchronously, without blocking the business or putting pressure on the system, ensuring the stable operation of the service. The access management of the application system minimizes or even eliminates the intrusion into the application system, and has demonstrated good application effects in all aspects.

[0074] The embodiment of the present invention achieves the effect of once-built access at any time, without intrusion to the accessed system, and can provide full-text search services. Each access system has an independent storage space to achieve multi-tenant management, and can also provide unified full-volume search. With the increase of access systems and processing pressure, imperceptible horizontal expansion can be carried out to continuously provide stable and efficient services.

[0075] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0076] like Figure 2 As shown in the figure, we need to build a cluster environment as the bottom-level support. The word segmenter needs to be a Chinese word segmenter, and then the management and maintenance of the basic word library and the professional word library can be coordinated to achieve the accuracy of word segmentation. By managing and maintaining a set of synonyms for a business, the matching degree of the search can be improved. At the same time, we need to establish an unstructured data processing center, which is composed of a variety of tools to process different data types. At the same time, for new file types, it can realize plug-and-play functional expansion. Figure 6 , which is a hierarchical diagram of an asynchronous service-based information retrieval and index management system. Figure 7 To process the logic diagram, Figure 8 Execute the logic diagram for the message queue and consumer agent.

[0077] The embodiment of the present invention realizes the capability of once-built and anytime access, and can quickly access a new system to realize full-text search service for it.

[0078] The specific implementation is as follows:

[0079] 1. Application system access, select the most suitable method from the three provided methods: log, interface and intermediate library; the data processing unit in this step only needs to provide the primary key ID of the file and the system identifier, and the data enters the message processing queue for the next step of processing.

[0080] 2. Determine the data standards, including basic data fields, which are fixed and required data, including file name, file type, and file storage; data used to filter the search range is customized by the application system according to actual conditions; extended data is other required data provided by the application system on demand, such as file classification with business significance, etc. Establish data indexes in the ES cluster according to the above data standards.

[0081] 3. The application system provides a data service interface to provide the corresponding data in the data standard. The application system needs to provide a service interface for obtaining unstructured data. The platform will write it into the configuration file. The message queue and consumer agent management will call the corresponding service according to the application system identifier, and use the file's primary key ID to obtain the corresponding unstructured and structured data, and the obtained data will enter the next step for processing.

[0082] 4. Unstructured data enters the unstructured data processing center, and different file types are forwarded to the processing unit of the corresponding file type through scheduling processing. The full-text content is extracted, assembled with the acquired structured data, and written into the ES cluster.

[0083] 5. Provide search services. The ES cluster's vocabulary is divided into basic vocabulary, professional vocabulary and synonym vocabulary. Text data is built based on the vocabulary and enters the ES cluster. The search service provides the following search services on the ES cluster:

[0084] a) Provide a unified full-text search service to retrieve the content within the authority of all application systems, and trace the source to the location where the files of the application system are generated.

[0085] b) Specify the customized full-text search service of the system and directly integrate the service with the application system. The application system can directly call the service to obtain the full-text search results.

[0086] Embodiment 3

[0087] Fig. 9This is a flowchart of an asynchronous service-oriented information retrieval and index management method provided by Embodiment 3 of the present invention. The method is applied to an asynchronous service-oriented information retrieval and index management system, the system includes an access management module, a data processing module and a data storage module, the access management module is connected to at least one application system, the data processing module is connected to the access management module, and the data storage module is connected to the data processing module. Fig. 9 As shown, the method includes:

[0088] S310, obtaining identification information corresponding to the unstructured data in the application system, storing the identification information in a message queue, obtaining the unstructured data of the application system according to the message queue, and sending the unstructured data to the data processing module.

[0089] In this embodiment, the application system refers to the business system. There is a large amount of unstructured data in the business system. In order to manage and retrieve the data of the business system, the application system is connected to the access management module. There are multiple types and numbers of application systems. The access management module can be connected to multiple application systems. The asynchronous service-oriented information retrieval and index management system in this embodiment can provide information storage and retrieval services for multiple application systems at the same time. The identification information of the unstructured data in the application system is obtained through the access management module. The identification information is used to distinguish different unstructured data. The identification information of the obtained unstructured data is stored in the message queue. The unstructured data of the application system is obtained based on the message queue, and the unstructured data is sent to the data processing module.

[0090] S320, receiving the unstructured data sent by the access management module, performing text extraction on the unstructured data to obtain data to be written, and writing the data to be written into the data storage module.

[0091] Specifically, the data processing module is a module for processing unstructured data, mainly converting unstructured data into structured data, such as using the data obtained after text extraction processing on the unstructured data as the data to be written, and then writing the data to be written into the data storage module.

[0092] S330, receiving the data to be written written by the data processing module, performing word segmentation processing on the data to be written according to the basic vocabulary and the professional vocabulary to obtain data to be stored, storing the data to be stored and constructing a target index corresponding to the application system.

[0093] Specifically, the data storage module is mainly a module for storing data to be written and establishing indexes. The data storage module can be understood as a server that provides data storage and retrieval, such as an ES server; the data storage module includes a basic vocabulary and a professional vocabulary, and the data to be written is segmented according to the basic vocabulary and the professional vocabulary. The data obtained after processing is the data to be stored. The data to be stored is stored in the server and an index corresponding to the data to be stored, i.e., the target index, is established.

[0094] The technical solution of the embodiment of the present invention is to obtain the identification information corresponding to the unstructured data in the application system, store the identification information in the message queue, obtain the unstructured data of the application system according to the message queue, and send the unstructured data to the data processing module; receive the unstructured data sent by the access management module, extract the text of the unstructured data to obtain the data to be written, and write the data to be written to the data storage module; receive the data to be written written by the data processing module, perform word segmentation processing on the data to be written according to the basic vocabulary and the professional vocabulary to obtain the data to be stored, store the data to be stored and construct the target index corresponding to the application system. The problem of low flexibility and scalability of the existing retrieval service is solved, the unstructured data of each application system is unified into one platform, and the incremental unstructured data generated by each application system is transmitted regularly. For new application systems, expansion access can be carried out at any time, and integrated full-text retrieval is realized. Every user can search, locate matching files, and locate the application system where they are located, which is convenient for users and maximizes the value of unstructured data. At the same time, a service interface is provided to each application system, so that the application system can integrate the platform and use the full-text search capability to retrieve all structured data in its own system and locate specific business scenarios, greatly improving the convenience of the system.

[0095] Embodiment 4

[0096] Fig.10 is a schematic diagram of the structure of an electronic device provided according to Embodiment 4 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0097] like Fig.10As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0098] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0099] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as an asynchronous service-based information retrieval and index management method.

[0100] In some embodiments, an asynchronous service-based information retrieval and index management method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the asynchronous service-based information retrieval and index management method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform an asynchronous service-based information retrieval and index management method in any other appropriate manner (e.g., by means of firmware).

[0101] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0103] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0105] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0106] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0107] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0108] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An asynchronous service-oriented information retrieval and index management system, characterized in that: It includes an access management module, a data processing module and a data storage module; wherein, The access management module is connected to at least one application system, and is used to obtain identification information corresponding to unstructured data in the application system, store the identification information in a message queue, obtain the unstructured data of the application system according to the message queue, and send the unstructured data to the data processing module; The data processing module is connected to the access management module, and is used to receive the unstructured data sent by the access management module, perform text extraction on the unstructured data to obtain data to be written, and write the data to be written into the data storage module; The data storage module is connected to the data processing module, and is used to receive the data to be written written by the data processing module, perform word segmentation processing on the data to be written according to the basic vocabulary and the professional vocabulary to obtain the data to be stored, and store the data to be stored and construct a target index corresponding to the application system.

2. The system according to claim 1, characterized in that The access management module includes a log submodule, an interface submodule, an intermediate library submodule and a queue consumption submodule; wherein, The log submodule is connected to the application system and is used to collect log information in the application system and store the log information in a message queue; wherein the log information includes identification information of the unstructured data; The interface submodule is connected to the application system and is used to call a preset interface to obtain identification information of unstructured data in the application system upon receiving a call request from the application system, and store the identification information in the message queue. The intermediate library submodule is connected to the application system and is used to receive the identification information of the unstructured data synchronized by the application system and store the identification information in the message queue; The queue consumption submodule is connected to the data processing module, and is used to obtain unstructured data corresponding to the log information or identification information from the application system in sequence according to the storage order of the log information or identification information in the message queue, and send the unstructured data to the data processing module.

3. The system according to claim 1, characterized in that The data processing module includes an extraction submodule and a writing submodule; wherein, The extraction submodule is connected to the access management module, and is used to receive the unstructured data, and based on the text extraction method corresponding to the unstructured data, perform text extraction on the unstructured data to obtain the data to be written, and send it to the writing submodule; The writing submodule is connected to the extraction module and the data storage module, and is used for receiving the data to be written and writing the data to be written into the data storage module.

4. The system according to claim 3, characterized in that The extraction submodule includes a text extraction unit, a recognition extraction unit and a multimedia extraction unit; wherein, The text extraction unit is used to receive the document-type unstructured data sent by the access management module, perform text extraction processing on the document-type unstructured data, and obtain the data to be written corresponding to the document-type unstructured data; The recognition and extraction unit is used to receive the image-type unstructured data sent by the access management module, perform optical character recognition processing on the image-type unstructured data, and obtain the data to be written corresponding to the image-type unstructured data; The multimedia extraction unit is used to receive the multimedia unstructured data sent by the access management module, and to perform text conversion on the multimedia unstructured data to obtain the data to be written corresponding to the multimedia unstructured data.

5. The system according to claim 3, characterized in that The extraction submodule also includes an extraction configuration unit, wherein: The extraction configuration unit is used to configure an extraction unit corresponding to the unstructured data based on the type of the unstructured data.

6. The system according to claim 1, characterized in that The data storage module includes a word segmentation submodule, an index creation submodule and an index update submodule; wherein, The word segmentation submodule is used for, after receiving the data to be written, performing word segmentation processing on the data to be written according to the basic word library and the professional word library to obtain data to be stored, and storing the data to be stored in the data cluster; The index establishment submodule is used to establish a target index corresponding to the application system based on the data to be stored; The index updating submodule is used to update the target index based on the newly added data to be stored when the newly added data to be stored is retrieved.

7. The system according to claim 1, characterized in that The asynchronous service-oriented information retrieval and index management system also includes a data retrieval module, wherein: The data retrieval module is connected to the data storage module and is used to determine the target retrieval result based on the information to be retrieved, the target index and the data to be stored stored in the data storage module when receiving the information to be retrieved; The target search result includes search result data corresponding to the information to be searched and application system information to which the search result data belongs.

8. The system according to claim 7, characterized in that The data retrieval module includes a first retrieval submodule and a second retrieval submodule, wherein: The first retrieval submodule is connected to the data storage module and is used to determine the retrieval authority of the information to be retrieved when receiving the information to be retrieved, and determine the target index of at least one of the application systems based on the retrieval authority, and determine the target retrieval result based on at least one of the target indexes; The second search submodule is connected to the application system, and is used to determine a target search result based on a target index of the application system corresponding to the information to be searched when receiving the information to be searched sent by the application system, and send the target search result to the application system.

9. The system according to claim 7, characterized in that The data retrieval module also includes a synonym determination submodule, wherein: The synonym determination submodule is used to determine the synonym search result corresponding to the target search result after determining the target search result, and update the target search result based on the synonym search result.

10. An asynchronous service-based information retrieval and index management method, characterized in that: The method is applied to an asynchronous service-oriented information retrieval and index management system, comprising an access management module, a data processing module and a data storage module, wherein the access management module is connected to at least one application system, the data processing module is connected to the access management module, and the data storage module is connected to the data processing module; wherein the asynchronous service-oriented information retrieval and index management method comprises: Acquire identification information corresponding to the unstructured data in the application system, store the identification information in a message queue, acquire the unstructured data of the application system according to the message queue, and send the unstructured data to the data processing module; Receiving the unstructured data sent by the access management module, performing text extraction on the unstructured data to obtain data to be written, and writing the data to be written into the data storage module; The data to be written written by the data processing module is received, the data to be written is segmented according to the basic vocabulary and the professional vocabulary to obtain the data to be stored, and the data to be stored is stored and a target index corresponding to the application system is constructed.

Images