Data collection and retrieval distributed system and method and computer equipment

Abstract

The invention relates to a data collection and retrieval distributed system and method and computer equipment. Comprising a front-end application used for collecting initial data through a multi-coroutine concurrent execution database; the ES cluster is used for performing index recovery according to the operation conditions of different cluster nodes and retrieving the target data through an inverted index; the back-end application performs service registration and discovery based on the micro-service framework, and provides information calling associated with the initial data or the target data for the front-end application; and the message middleware is used for dynamically adjusting the process of data collection and data retrieval. According to the method and the device, quick retrieval can be realized, and data consistency under a distributed condition can be ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data collection and retrieval distributed system, method and computer device. Background Art

[0002] Most of the existing data retrieval systems are built using traditional technologies, such as .Net, Delphi, and Mysql. A small number of them are cloud computing-based systems, but they lack the cloud-native microservice system architecture, cannot use virtualization technologies such as containers for elastic scaling and node expansion, and the system can only withstand limited concurrency pressure. In addition, there is a lack of a complete system for core businesses such as information retrieval and collection in a big data environment, and cannot guarantee fast data retrieval and data consistency under distributed conditions. Summary of the invention

[0003] Based on this, the purpose of this application is to provide a data collection and retrieval distributed system, method and computer device that can achieve fast retrieval and data consistency under distributed conditions to solve the technical problems mentioned in the above background technology.

[0004] In the first aspect, the present application provides a data collection and retrieval distributed system. The system includes a front-end application, a back-end application, a database, an ES cluster, a microservice framework, and a message middleware, wherein:

[0005] A front-end application, used for concurrently executing the database to collect initial data through multiple coroutines;

[0006] ES cluster, used to restore indexes based on the operating conditions of different cluster nodes and retrieve target data through inverted indexes;

[0007] The back-end application performs service registration and discovery based on the microservice framework, and provides the front-end application with information calls associated with the initial data or the target data;

[0008] Message middleware is used to dynamically adjust the process of data collection and data retrieval.

[0009] In one embodiment, the front-end application includes a crawler framework; the collection of initial data from the database through concurrent execution of multiple coroutines includes: creating multiple coroutines based on a coroutine pool, and concurrently crawling the initial data of the web page through the crawler framework; the coroutine represents a user-mode lightweight thread; when any coroutine performs a read or write operation and is blocked, other coroutines are suspended through a target scheduling mode; the read and write operations include operations of writing initial data in the database.

[0010] In one embodiment, the ES cluster is also used to: divide the index library corresponding to the database into multiple pieces; distribute and back up each piece in each cluster node; and store the backup piece and the primary piece corresponding to any piece on different cluster nodes.

[0011] In one embodiment, index recovery according to the operation status of different cluster nodes includes: if the operation status of the current cluster node among multiple cluster nodes is down, data recovery is performed; multiple current slices in the current cluster node are determined; and backup slices corresponding to the multiple current slices are obtained from other cluster nodes to achieve index recovery of the index library.

[0012] In one embodiment, the index recovery according to the operation status of different cluster nodes includes: if the operation status of the current cluster node among multiple cluster nodes is down, determining multiple current slices in the current cluster node; obtaining backup slices corresponding to the multiple current slices from other cluster nodes to achieve index recovery of the index library.

[0013] In one embodiment, the system also includes a transaction log; the cache system of the ES cluster includes multiple data segments; the process of performing data recovery includes: writing the target data to the data segment through a refresh operation; when the log volume of the transaction log reaches a target threshold, performing data recovery of the target data through a flush operation.

[0014] In one embodiment, retrieving target data through an inverted index includes: creating a word segmentation dictionary through a target word segmentation method; indexing and expanding the target data through the word segmentation dictionary to obtain extended data; confirming category information of the extended data when performing inverted indexing in an index library corresponding to a database; the category information includes at least a document position, word frequency, and offset corresponding to the extended data.

[0015] In one embodiment, the back-end application includes a Gin framework; the Gin framework includes policy control and policy service; the policy control provides a Web Api microservice for the front-end application to call, and the policy service provides microservices for data collection and data retrieval processes; the microservice framework includes a service module and a configuration center; the service registration and discovery based on the microservice framework includes: the policy control and policy service are both registered through the service module; the policy service is monitored through the configuration center, and the microservice changes in the Gin framework are made according to the monitored situation.

[0016] In one embodiment, providing a front-end application with an information call associated with the initial data or the target data includes: determining a target microservice associated with the initial data or the target data from the policy service; discovering the address information of the target microservice through the configuration center and providing it to the policy control for calling; and completing the data transmission between the back-end application and the front-end application through the target protocol.

[0017] In one embodiment, the process of data collection and data retrieval is dynamically adjusted, including: creating a transaction message with consumption logic through a target semantic method; the transaction message includes an upstream message corresponding to the data collection process and a downstream message corresponding to the data retrieval process; the execution result of the upstream message is asynchronously confirmed multiple times through the message middleware; when the execution result is successful, the execution of the downstream message is triggered; when the execution result is a failure, the transaction rollback or message review of the upstream message is triggered.

[0018] In one embodiment, the system also includes a data warehouse, which includes a management tool and an offline tool; the management tool integrates multiple big data processing middleware, which is used to collect and analyze log information, and visualize and analyze monitoring indicators of each cluster node; the offline tool is used to complete the full synchronization process of heterogeneous data, and realize the incremental data synchronization process in combination with the message middleware.

[0019] In a second aspect, the present application also provides a data collection and retrieval method, comprising:

[0020] Collect initial data from the database through concurrent execution of multiple coroutines;

[0021] Restore the index based on the operation status of different cluster nodes, and retrieve the target data through the inverted index;

[0022] Perform service registration and discovery based on the microservice framework, and provide front-end applications with information calls associated with initial data or target data;

[0023] Dynamically adjust the process of initial data collection and targeted data retrieval.

[0024] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above data collection and retrieval method when executing the computer program.

[0025] The above-mentioned data collection and retrieval distributed system, method and computer device, the front-end application collects the initial data of the database through multiple concurrent executions, which can significantly improve the concurrent processing capability of data collection and shorten the time of data collection. The ES cluster performs index recovery and establishes an inverted index according to the operation status of different nodes, which helps to enhance the performance of data retrieval and improve the ability to quickly access and query target data. The back-end application registers and discovers services based on the microservice framework, which helps to achieve the independence and scalability of services, and provides information calls associated with the initial data or target data to the front-end application, which can realize data persistence support. The message middleware dynamically adjusts the data collection and retrieval process, solves the problem of real-time incremental synchronization of heterogeneous data sources and asynchronous decoupling of microservices, ensures the stability and data consistency of the system under high load, and improves the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the structure of a data collection and retrieval distributed system in one embodiment;

[0027] Figure 2 is an application flow chart of a data collection and retrieval method in one embodiment;

[0028] Figure 3 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] This embodiment provides a data collection and retrieval distributed system, including a front-end application, a back-end application, a database, an ES cluster, a microservice framework and a message middleware, wherein: the front-end application is used to collect initial data from the database through multi-process concurrent execution; the ES cluster is used to perform index recovery according to the operation status of different cluster nodes, and retrieve target data through an inverted index; the back-end application performs service registration and discovery based on the microservice framework, and provides the front-end application with information calls associated with initial data or target data; the message middleware is used to dynamically adjust the data collection and data retrieval process.

[0031] Specifically, this system is based on the popular microservice system and cloud native technology in the industry, and uses a web application framework with separated front-end and back-end applications. Through the crawler framework in the front-end application, the efficient collection process of initial data is regularly processed through multi-coroutine distributed concurrent processing, and various initial data are recorded in a MySQL database, etc., realizing the whole process of data collection, cleaning, conversion and integration. Considering that the data in the database will reach tens of millions or hundreds of millions of data records in the future, there are problems such as unstable performance and slow retrieval speed when crawling data regularly, so it is necessary to establish a distributed database cluster and database sub-library and sub-table.

[0032] ES cluster, pre-shards the index library corresponding to the database, and the backup shards are not stored on the same cluster node as the primary shards, so that the complete index library information can be restored according to the operation status of different cluster nodes, that is, the process of sharding the initial data and establishing an inverted index. The front-end application provides an interactive interface containing an input box, and the user enters the target data to be queried, and then the target data is retrieved through the inverted index established in the ES cluster. Among them, the inverted index representation maps each word in the document set to a list of documents containing the word, thereby realizing a method for quickly retrieving words that appear in the document.

[0033] Since the microservice framework includes a configuration center and a service module, a microservice decoupling system is established through Nacos+Consul, so that data transmission between services can be called through the target protocol. After the policy module in the back-end application registers the service through service discovery, it can provide Web Api microservices called by the web front-end application, initial data writing, and microservices for querying target data based on different conditions. The policy module also acts as a service provider, providing the address information of the microservice to the configuration center for service discovery, so as to realize the information call associated with the initial data or target data, and realize the persistence support of data. The message middleware, such as RocketMQ, divides the process of data collection and data retrieval into upstream main business and downstream branch business, and determines whether the transaction can be delivered to the downstream branches according to the execution of the upstream main business, thereby realizing the dynamic adjustment of the initial data collection and target data retrieval process.

[0034] Among them, cloud native is a distributed cloud based on distributed deployment and unified operation and management, and is a set of cloud technology product systems based on technologies such as containers, microservices, and DevOps. Cloud native technology gives full play to the elasticity and distributed advantages of cloud platforms to achieve rapid deployment, on-demand scaling, and non-stop delivery. Distributed computing is the study of how to divide a problem that requires a very huge computing power to solve into many small parts, and then distribute these parts to multiple computers for processing, and finally combine these calculation results to get the final result. Microservices is a cloud native system approach that contains many small components or services that are loosely coupled and can be deployed separately in a single application. These services usually have their own technology stack, including databases and data management models; communicate with each other through a combination of REST APIs, event streams, and message brokers; and are organized according to business capabilities, with service separators called bounded contexts.

[0035] In the above-mentioned distributed system for data collection and retrieval, the front-end application can significantly improve the concurrent processing capability of data collection and shorten the time of data collection by executing the database for initial data concurrently through multiple coroutines. The ES cluster performs index recovery and establishes an inverted index according to the operation status of different nodes, which helps to enhance the performance of data retrieval and improve the ability to quickly access and query target data. The back-end application registers and discovers services based on the microservice framework, which helps to achieve the independence and scalability of services, and provides information calls associated with initial data or target data to the front-end application, realizing data persistence support. The message middleware dynamically adjusts the data collection and retrieval process, solves the problem of real-time incremental synchronization of heterogeneous data sources and asynchronous decoupling of microservices, ensures the stability and data consistency of the system under high load, and improves the overall performance of the system.

[0036] In one embodiment, the initial data is collected by executing the database concurrently through multiple coroutines, including: creating multiple coroutines based on the coroutine pool, and concurrently crawling the initial data of the web page through the crawler framework; when any coroutine performs read and write operations and is blocked, other coroutines are suspended through the target scheduling mode.

[0037] Among them, front-end applications include crawler frameworks, such as the golang+colly framework. Golang is a statically typed, compiled open source programming language. Colly is a fast and scalable crawler framework widely used in the Go language, providing a higher level of abstraction, making it easy to crawl web pages, process HTML content, and manage request queues. A coroutine represents a user-mode lightweight thread that can be created based on a coroutine pool and implement switching and scheduling of multiple execution threads within a single thread. Read and write operations include operations that write initial data to the database.

[0038] Specifically, through the crawler framework, and taking full advantage of the Go language itself, we create multi-coroutine distributed concurrent crawling of the initial data of each area, and then perform data parsing, extraction and integration. When crawling different types of websites in a certain area, the CRUD of the MySQL database to the disk, that is, io read and write operations, will frequently be involved, and other jobs need to wait, resulting in a waste of system CPU resources. In addition, setting a time interval for crawling will also cause the crawling thread to be idle. Therefore, we create multiple coroutines corresponding to different data areas for concurrent processing. When any coroutine performs a read and write operation and is blocked, other coroutines in the same context executable queue are suspended to other threads according to the target scheduling mode, that is, switched to a new thread for execution until other coroutines are scheduled for execution by the new crawling task. Among them, the blocking of io read and write operations may be due to network IO / disk IO blocking.

[0039] Among them, the target scheduling mode includes the MPG scheduling mode, which is a scheduling model used to manage concurrent execution within the Go language. M, P, and G represent different components, which work together to achieve efficient concurrent execution; M is the physical entity that executes the Go code and is responsible for executing tasks in the Go program. P represents the scheduling context and can be regarded as a local scheduler that is responsible for connecting the G waiting to be executed with M; G is the concurrent execution body in the Go language.

[0040] In this embodiment, by creating multiple coroutines to crawl the initial data of different data areas in a distributed manner, the extra waiting time caused by interface blocking is reduced, system resources are fully utilized, CPU utilization is improved, the data crawling rate is greatly optimized, and efficient writing of initial data and efficient retrieval of subsequent target data are ensured.

[0041] In one embodiment, the ES cluster is also used to: divide the index library corresponding to the database into multiple pieces; distribute and back up each piece in each cluster node; and store the backup piece and the primary piece corresponding to any piece on different cluster nodes.

[0042] Specifically, Figure 1 As shown, Figure 1 The structure diagram of the data collection and retrieval distributed system in one embodiment. The use of ES cluster further enhances the system reliability. ES cluster logically divides an index library into many pieces, distributes them to different cluster nodes, and backs up each piece. The backup piece corresponding to any piece is not stored on the same node as the main piece. Therefore, a read-write separated distributed storage and retrieval system is constructed through the database Mysql+ES ​​cluster, which can be applied to full-text retrieval and near real-time search scenarios.

[0043] In one embodiment, index recovery is performed according to the operating conditions of different cluster nodes, including: if the operating condition of the current cluster node among multiple cluster nodes is down, data recovery is performed; and / or, multiple current slices in the current cluster node are determined; and backup slices corresponding to the multiple current slices are obtained from other cluster nodes to achieve index recovery of the index library.

[0044] Specifically, if the current cluster node among multiple cluster nodes is down, data recovery is required through the transaction log in the system. And / or, when the current cluster node is down, multiple current slices in the current cluster node are determined, and since other nodes store backup slices corresponding to the current slices, the complete index library information can be restored through the backup information obtained from other nodes.

[0045] In the above embodiment, the ES cluster shard storage method not only bears the read request load to achieve load balancing, but also enables system fault tolerance and recovery when abnormal operation of cluster nodes occurs. It also realizes dynamic expansion of cluster nodes, ensuring high availability and high reliability of the system in the data collection and retrieval process.

[0046] In one embodiment, the process of performing data recovery includes: writing target data into a data segment through a refresh operation; and when the log volume of the transaction log reaches a target threshold, performing data recovery of the target data through a flush operation.

[0047] Among them, Figure 1 As shown, the system also includes a transaction log, such as Translog; the cache system of the ES cluster includes multiple data segments, such as segment; refresh operations such as refresh operations, and flush operations such as flush operations.

[0048] Specifically, when a data segment in the cache system of the ES cluster is not written to disk due to a system crash, the system can recover the data through the transaction log. For full-text retrieval and near real-time search scenarios, a lightweight method is used to make a document searchable. When storing the initial data, the ES cluster will first write the data to a memory cache and the data written to the transaction log at the same time. At this time, the data is invisible. The data becomes visible only after the data segment is written to the file system cache through a refresh operation. When the log volume of the transaction log reaches a certain target threshold, the data is refreshed to the disk by a flush operation, and the corresponding log in the transaction log is deleted.

[0049] In this embodiment, the time that the ES cluster can be searched is optimized from several minutes to seconds, and the storage performance is improved. In addition, when a data segment in the system cache is not written to the disk due to a system abnormality, the data can still be accurately restored through the transaction log, ensuring the high reliability of the data writing process.

[0050] In one embodiment, retrieving target data through an inverted index includes: creating a segmentation dictionary through a target segmentation method; indexing and expanding the target data through the segmentation dictionary to obtain extended data; and confirming category information when performing inverted indexing on the extended data in an index library corresponding to the database.

[0051] The category information at least includes the document position, word frequency, and offset corresponding to the extended data.

[0052] Among them, the target word segmentation method, such as ik Chinese word segmentation, adopts the coarse-grained ik_smart rule that does not split to the greatest extent, but there are still problems. It will split a phrase consisting of four characters into two words. For example, digital economy will be split into digital and economy, and the whole cannot be retained. In other words, if it is a newer real-time hot word / popular word, the ik word segmentation dictionary cannot recognize it, and these hot words will be segmented, that is, split, causing us to not be able to search for these keywords, because its inverted index does not index these hot words as a whole, but splits them into multiple words to establish an index. For example, "digital economy" will be split into "digital" and "economy", and an inverted index will be established and stored. Therefore, we expand the original target data through a custom word segmentation dictionary, a custom word segmentation vocabulary, and a custom stop word library.

[0053] Specifically, when facing TB or even PB-level data volumes, it will take a long time to retrieve the target data when querying based on conditions. Since the ES cluster uses inverted index and target word segmentation to store documents, the word segmentation dictionary created by the target word segmentation can achieve index expansion of the target data, and the category information during inverted indexing records the documents, document locations, word frequencies, offsets, etc. corresponding to each word, which makes the query of the expanded target data extremely fast.

[0054] In this embodiment, the way ES builds an inverted index greatly optimizes the time consumption of system retrieval compared to traditional solutions, and users can search for target data in various regions very quickly. In addition, the ES custom word segmentation dictionary also optimizes data retrieval and enhances the user's search experience.

[0055] In one embodiment, the ES cluster is deployed on K8s. The K8s deployment container service facilitates container contraction and expansion, building a high-performance, highly available cloud-native distributed architecture.

[0056] In one embodiment, service registration and discovery are performed based on a microservice framework, including: both policy control and policy service are registered through a service module; policy service is monitored through a configuration center, and microservice changes in the Gin framework are made according to the monitored situation.

[0057] Among them, the back-end application includes the Gin framework; the Gin framework includes policy control and policy services; policy control provides Web Api microservices for front-end applications to call, and policy services provide microservices for data collection and data retrieval processes; the microservice framework includes service modules and configuration centers.

[0058] Specifically, Figure 1 As shown, this system uses Golang+Gin+Vue to develop a web application framework with separated front-end and back-end applications. The front-end application is based on the Golang framework, and the back-end application is based on the Gin framework. Vue is a progressive JavaScript framework for building user interfaces, which aims to achieve responsive data binding and combined view components through concise APIs. The back-end application based on the Gin framework includes policy controls such as Policy-Web-Api and policy services such as Policy-Service, and both are registered through the service module Consul. Both are initialized in advance according to the specified configuration file of the Nacos Configuration Center. The Nacos Configuration Center will perform MD5 monitoring on the policy service Policy-Service and push changes to each microservice based on the detected MD5 changes.

[0059] Among them, the Gin framework is a lightweight HTTP web framework written in Go that can handle a large number of concurrent requests. Policy-Web-Api refers to the policy control implemented in the Web API. In Web API development, policies are usually used to implement various functions such as authorization, authentication, logging, and current limiting. Policy-Service is a service responsible for implementing or managing policies. In a microservice system or distributed system, such a service may be responsible for centrally managing various policies (such as access control policies, data protection policies, business rules, etc.) and controlling access to resources, data, or services based on these policies. MD5-Message Digest Algorithm is a widely used cryptographic hash function that can generate a 128-bit hash value to ensure complete and consistent information transmission.

[0060] In this embodiment, the service registration and discovery mechanism is usually used in conjunction with load balancing, which can automatically distribute requests to different service instances, improve the throughput and availability of the system, and achieve decoupling between different services in the data collection and data retrieval process.

[0061] In one embodiment, information calls associated with initial data or target data are provided to the front-end application, including: determining a target microservice associated with the initial data or target data from a policy service; discovering address information of the target microservice through a configuration center and providing it to the policy control for calling; and completing data transmission between the back-end application and the front-end application through a target protocol.

[0062] Specifically, the policy service Policy-Service acts as a service provider, determines the target microservice associated with the initial data or target data, and exposes the IP and port, i.e. the address information of the target microservice, to the configuration center Nacos, for the policy control Policy-Web-Api to make subscriber calls, and then the back-end application and the front-end application perform RPC calls through, for example, the gRPC protocol to complete data transmission. Among them, gRPC is widely used in microservice systems and is a remote procedure call protocol that is efficient, intuitive, and convenient for data transmission and processing. RPC (Remote Procedure Call) is a computer communication protocol that allows a computer program to call a subroutine in another computer program over the network and obtain a return value.

[0063] In one embodiment, the system uses a cloud-native, fast, scalable, and distributed Kong as a microservice API gateway, processes customer requests by building a Kong gateway, and finally proxies to the final API.

[0064] In this embodiment, the target microservice is dynamically discovered by the configuration center, and the system can flexibly respond to changes without hard-coding the service address, which improves the maintainability and scalability of the system. The adoption of the target protocol ensures efficient and consistent data transmission between the back-end application and the front-end application, which helps to improve the performance of the overall system.

[0065] In one embodiment, the process of data collection and data retrieval is dynamically adjusted, including: creating a transaction message with consumption logic through a target semantic method; asynchronously confirming the execution result of the upstream message multiple times through the message middleware; when the execution result is successful, triggering the execution of the downstream message; when the execution result is a failure, triggering a transaction rollback or message review of the upstream message.

[0066] Among them, transaction messages include upstream messages corresponding to the data collection process and downstream messages corresponding to the data retrieval process. The target semantic method is, for example, the EXACTLY-ONCE semantic method. In the context of data processing and message delivery, it means ensuring that each piece of data or message is processed only once, and the processing results are accurate and non-duplicate. This semantics is crucial to ensuring data consistency and accuracy, especially in distributed systems and high-concurrency environments.

[0067] Specifically, this system creates transaction messages with consumption logic through the target semantic method, and on top of the processing logic for writing initial data, the process of initial data crawling and writing local transactions to the MySQL database is used as the upstream main business, and the process of data integration and retrieval, as well as logging in the ES cluster is used as the downstream branch business, so as to process transactions in a distributed manner. When entering and executing the upstream main business, the front-end application will asynchronously send the upstream message corresponding to the upstream main business to the server of the message middleware. At this time, the upstream message is a semi-transactional message, and these semi-transactional messages will not be consumed immediately, for example, they will be marked as "temporarily undeliverable".

[0068] When the upstream main business is executed, it will be submitted to the message middleware multiple times according to its execution results. If the execution is successful, that is, the result is in the commit state (Commit), the upstream message will be delivered to the downstream branches to execute the downstream branch business. Otherwise, if it fails, the transaction will be rolled back (Rollback). The server of the message middleware will roll back the transaction and will not deliver the semi-transactional message to the downstream branch; if the result is not received within the timeout, a message check will be initiated for the upstream message to check the execution result.

[0069] In one embodiment, in the implementation of EXACTLY-ONCE semantics, a consumption record table of a new message is created in the database, and the primary key ID can be set as a unique ID on the business / transaction. When a transaction is started on the consumer side, data is first inserted into the consumption record table, and then the original consumption logic is followed. After the consumption logic is completed, the transaction is submitted. In this way, as long as the consumption is successful, the transaction will be submitted, and an error will be reported when inserting into the consumption record table next time the consumption is consumed, as long as the primary key conflict of inserting into the consumption record table is captured.

[0070] In the above embodiment, the distributed transaction message processing method ensures the consistency of heterogeneous data between the local business of system data changes such as upstream data crawling and database writing and the downstream branch business, and fully utilizes the ability of message asynchronization to shorten the link and improve concurrency. Through the specific implementation of the target semantic method, it is ensured that the message sent to the message system can only be processed once by the consumer end. Even if the producer end retries the message sending and causes a message to be delivered repeatedly, the message will not be consumed repeatedly.

[0071] In one embodiment, the system also includes a data warehouse, which includes a management tool and an offline tool; the management tool integrates multiple big data processing middleware to collect and analyze log information, and visualize and analyze monitoring indicators of each cluster node; the offline tool is used to complete the full synchronization process of heterogeneous data, and realize the incremental data synchronization process in combination with the message middleware.

[0072] Specifically, Figure 1 As shown in the figure, the management tool adopts the ELKB+Metricbeat solution, integrating big data processing middleware such as Elasticsearch, Logstash, Kibana, FileBeat and Metricbeat, collecting and analyzing log information, and visualizing and analyzing the monitoring indicators of each cluster node, supporting data visualization and analysis. Among them, ELKB is an open source distributed log management solution, which has a closed-loop log processing process, efficient retrieval performance, linear scalability and low operation and maintenance costs. Elasticsearch is used to store and search log data and provide full-text search capabilities. Logstash (or Beats) is a log collection and processing tool. Logstash has complex functions and is suitable for deployment in server clusters; Beats is more lightweight and suitable for deployment on business machines, collecting data in real time and passing it to Elasticsearch. Kibana provides a web-based graphical interface for searching, analyzing and visualizing log data in Elasticsearch. Metricbeat is a lightweight open source metric data collector.

[0073] Offline tools such as DataX are open source versions of Alibaba Cloud DataWorks data integration. This offline data synchronization tool / platform is used to solve data integration and efficient full data synchronization between various heterogeneous data sources, and is combined with the message middleware RocketMQ to achieve asynchronous distribution and incremental synchronization, maximizing support for data integration and ensuring data consistency. Among them, DataX is an open source, efficient, and easy-to-use data synchronization tool that is used in a variety of scenarios such as data migration, data backup, and data synchronization in the big data field; DataWorks is a one-stop big data development and governance platform based on a big data engine.

[0074] In one embodiment, ETL technology is also used in the data warehouse. ETL is the process of extracting, cleaning, transforming and loading the data of the business system into the data warehouse. The purpose is to integrate the scattered, messy and non-uniform data in the enterprise to provide an analytical basis for the enterprise's decision-making.

[0075] In the above embodiment, the system adopts an open source distributed log management solution, integrates a variety of big data processing middleware, realizes the collection and analysis of log information, and tracks and studies each cluster node, supporting data visualization and analysis. Subsequently, based on the ETL data warehouse technology DataX, these data can be integrated and combined with machine learning and other methods for business intelligence analysis, to achieve system intelligent analysis and consulting opinions, and provide users with key information for decision-making and maximizing the use of policy data.

[0076] To summarize, this application provides a cloud-native distributed system for policy collection and full-text real-time retrieval, a distributed log management solution based on ELKB+Metricbeat, and a distributed storage and retrieval system for database read-write separation solutions. It uses ES distributed clusters and custom word segmentation libraries to optimize the system architecture and solve the problem of fast retrieval and real-time search of full-text policy data.

[0077] In one embodiment, Figure 3 As shown, a data collection and retrieval method is provided, which is applied to a data collection and retrieval distributed system, and includes the following steps: collecting initial data from a database through concurrent execution of multiple coroutines; recovering indexes according to the operating conditions of different cluster nodes, and retrieving target data through an inverted index; registering and discovering services based on a microservice framework, and providing front-end applications with information calls associated with initial data or target data; and dynamically adjusting the process of initial data collection and target data retrieval.

[0078] Specifically, Figure 2 As shown, Figure 2 The following is the application principle diagram of the data collection and retrieval method. The two core parts of the system business process are data collection and data retrieval. The initial data collected includes but is not limited to the policy titles, release time and policy document URL addresses of the two dimensions of the regional government and government office. For example, the latest policy information of different regions is crawled once a day. There are different data characteristics between different regions, such as administrative districts, functional areas and specific areas.

[0079] For example, when it is an administrative district, users can search for administrative districts through the interactive interface: such as national, provincial, municipal, and district levels. When it is a functional area, users can search for functional areas through the interactive interface, such as limited to economic development zones, high-tech zones, self-innovation zones, free trade zones, national new areas, etc. When it is a specific area, users can search for specific areas through the interactive interface, such as limited to national central cities, trillion-dollar city clubs, provincial capitals, separately planned cities, sub-provincial cities, first-tier cities, new first-tier cities, etc.

[0080] At a preset time every day, such as 0:00 a.m., the background will regularly collect data from each region through multi-coroutine distributed concurrent crawling technology, call the backend application service collector to update the data in real time, and synchronize the data to the database and ES cluster. The specific implementation process of the data collection and retrieval distributed method can refer to the application process of the above-mentioned data collection and retrieval distributed system, and this embodiment will not be repeated here.

[0081] Therefore, through products that integrate data collection and retrieval methods, based on the microservice system and cloud native technology that are popular in the industry, it is possible to collect information such as government consulting cases, policy document information, original links to policy news, etc. from official websites, providing a large amount of authoritative data information for industrial policy research and consulting work. It is suitable for full-text retrieval and near real-time search scenarios, and supports data visualization and analysis.

[0082] Based on the same inventive concept, the embodiment of the present application also provides a data collection and retrieval device for implementing the data collection and retrieval method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data collection and retrieval device embodiments provided below can refer to the limitations on the data collection and retrieval method above, and will not be repeated here.

[0083] In one embodiment, a data collection and retrieval device is provided, comprising: a data processing module, an information calling module and a process adjustment module, wherein;

[0084] The data processing module is used to collect initial data from the database through multi-process concurrent execution; restore indexes according to the operating conditions of different cluster nodes, and retrieve target data through inverted indexes.

[0085] The information call module is used to register and discover services based on the microservice framework, and provide information calls associated with initial data or target data for front-end applications.

[0086] The process adjustment module is used to dynamically adjust the initial data collection and target data retrieval processes.

[0087] Each module in the above data collection and retrieval can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0088] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O), a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data collection and retrieval method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external touchpad or mouse, etc.

[0089] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0090] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0091] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0092] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0093] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0094] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data collection and retrieval distributed system, characterized in that: The system includes front-end applications, back-end applications, databases, ES clusters, microservice frameworks, and message middleware, among which: A front-end application, used for concurrently executing the database to collect initial data through multiple coroutines; ES cluster, used to restore indexes based on the operating conditions of different cluster nodes and retrieve target data through inverted indexes; The back-end application performs service registration and discovery based on the microservice framework, and provides the front-end application with information calls associated with the initial data or the target data; Message middleware is used to dynamically adjust the process of data collection and data retrieval.

2. The system according to claim 1, characterized in that The front-end application includes a crawler framework; the collection of initial data by the database through concurrent execution of multiple coroutines includes: Multiple coroutines are created based on the coroutine pool, and initial data of the webpage is crawled concurrently through the crawler framework; the coroutine represents a user-mode lightweight thread; When any coroutine performs a read / write operation and is blocked, other coroutines are suspended through the target scheduling mode; the read / write operation includes an operation of writing initial data into the database.

3. The system according to claim 1, characterized in that The ES cluster is also used to: Dividing the index library corresponding to the database into multiple pieces; Distribute and back up each of the slices in each cluster node; the backup slice and the primary slice corresponding to any slice are stored on different cluster nodes; The index recovery according to the operation status of different cluster nodes includes: If the current cluster node among multiple cluster nodes is down, perform data recovery; and / or, determining a plurality of current slices in the current cluster node; The backup pieces corresponding to the multiple current pieces are obtained from other cluster nodes to restore the index of the index library.

4. The system according to claim 3, characterized in that The system also includes a transaction log; the cache system of the ES cluster includes multiple data segments; The process of data recovery includes: Writing the target data into the data segment through a refresh operation; When the log volume of the transaction log reaches a target threshold, data recovery of the target data is performed through a flushing operation.

5. The system according to claim 1, characterized in that The retrieving the target data through the inverted index includes: Create a word segmentation dictionary through the target word segmentation method; Indexing and expanding the target data through the word segmentation dictionary to obtain extended data; Confirm the category information of the extended data when performing inverted indexing in the index library corresponding to the database; the category information at least includes the document position, word frequency, and offset corresponding to the extended data.

6. The system according to claim 1, characterized in that The backend application includes a Gin framework; the Gin framework includes policy control and policy services; the policy control provides a Web Api microservice for the frontend application to call, and the policy service provides microservices for data collection and data retrieval processes; The microservice framework includes a service module and a configuration center; The service registration and discovery based on the microservice framework includes: The policy control and policy service are both registered through the service module; The policy service is monitored through the configuration center, and the microservices in the Gin framework are changed according to the monitored situation.

7. The system according to claim 6, characterized in that The providing the front-end application with information call associated with the initial data or the target data includes: Determining a target microservice associated with the initial data or the target data from the policy service; Discovering the address information of the target microservice through the configuration center and providing it to the policy control for calling; The data transmission between the back-end application and the front-end application is completed through the target protocol.

8. The system according to claim 1, characterized in that The process of data collection and data retrieval is dynamically adjusted, including: Creating a transaction message with consumption logic through a target semantic method; the transaction message includes an upstream message corresponding to a data collection process and a downstream message corresponding to a data retrieval process; Asynchronously confirming the execution result of the upstream message multiple times through the message middleware; When the execution result is successful, trigger the execution of the downstream message; When the execution result is failure, a transaction rollback or message review of the upstream message is triggered.

9. The system according to any one of claims 1 to 8, characterized in that: The system also includes a data warehouse, which includes management tools and offline tools; The management tool integrates multiple big data processing middleware to collect and analyze log information and visualize and analyze monitoring indicators for each cluster node; The offline tool is used to complete the full synchronization process of heterogeneous data, and realize the incremental synchronization process of data in combination with the message middleware.

10. A data collection and retrieval method, characterized in that: The method comprises: Collect initial data from the database through concurrent execution of multiple coroutines; Restore the index based on the operation status of different cluster nodes, and retrieve the target data through the inverted index; Perform service registration and discovery based on the microservice framework, and provide front-end applications with information calls associated with initial data or target data; Dynamically adjust the process of initial data collection and targeted data retrieval.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to claim 10 are implemented.

Images