Data synchronization and query method and device, equipment and medium

Abstract

The invention relates to the technical field of big data and the field of financial science and technology, and discloses a data synchronization and query method, which comprises the following steps of: receiving a business operation request, executing data updating, generating updated data, encapsulating the updated data into a message, transmitting the message to a search service module through a message queue, analyzing the message and generating real-time data; real-time data is stored to a real-time index through a search engine, off-line data is obtained from a big data warehouse and stored to an off-line index with a date identifier, whether the off-line index is started to serve as a main index or not is judged, and the real-time data and the off-line data are written in the index switching process at the same time; after index switching is completed, query operation based on the main index is executed, and a result is returned. According to the method and the device, through the unified processing of the offline data and the real-time data, the inconsistency caused by data separation is avoided, the high efficiency and the timeliness of data processing are ensured, and the synchronous updating and query operation of the total data in a service scene is realized.

Description

Technical Field

[0001] The present invention relates to the fields of big data technology and financial technology, and in particular to a data synchronization and query method, device, equipment and storage medium. Background Art

[0002] In the context of big data processing in the financial sector, enterprises often need to process offline batch data and real-time data at the same time to meet different business needs. For example, in business scenarios such as customer transaction data analysis, risk control, and anti-fraud monitoring, financial institutions need to process a large amount of historical data (such as transaction records, account information, etc.), and at the same time, they also need to quickly respond to real-time transactions, market conditions, and risk events. To this end, two data processing methods are usually used: offline data processing and real-time data processing.

[0003] In practice in the financial field, offline data and real-time data processing are usually separated, which leads to a series of problems:

[0004] Offline data and real-time data are processed using different systems or architectures, which can easily lead to data inconsistency. For example, after the real-time data is updated, the offline data is not updated in time, which may affect the accuracy of subsequent data analysis.

[0005] Offline data processing has a long cycle and usually requires data loading and calculation in batches, which makes it difficult to support the rapid response requirements in real-time business scenarios. For financial services that require immediate feedback, such as risk control, offline processing is insufficient.

[0006] Since offline and real-time data processing systems are independent of each other, the system architecture is relatively complex and the maintenance and management costs are high. Financial institutions need to invest a lot of resources to maintain the operation of these two processing architectures and solve the coordination problems between different systems.

[0007] Due to the separation of offline and real-time data processing processes, it is difficult to achieve unified analysis of all data. In the financial field, decision-making and risk assessment often require comprehensive consideration of historical data and real-time data, and the separation processing method of existing technologies is difficult to meet this demand.

[0008] Therefore, when facing the need to integrate offline and real-time data, existing financial big data processing solutions have problems such as data inconsistency, poor timeliness, and high system complexity. These deficiencies have seriously affected the ability of financial institutions to respond quickly and efficiently to business scenarios. How to integrate offline and real-time data through a more unified and efficient data processing framework is a key issue that the financial industry urgently needs to solve. Summary of the invention

[0009] The main purpose of the present invention is to provide a data synchronization and query method, device, equipment and storage medium, aiming to solve the technical problems in the prior art that offline data and real-time data processing are separated, resulting in data inconsistency and poor processing timeliness, and cannot meet the needs of unified and efficient processing of full data in the business.

[0010] To achieve the above object, the present invention provides a data synchronization and query method, comprising:

[0011] Receive business operation requests;

[0012] Execute data update according to the business operation request, and generate update data including primary key ID, associated fields, update time, field information to be indexed and / or operation type;

[0013] Encapsulating the update data into a message, and sending the message to the search service module through a message queue;

[0014] The search service module receives and parses the message, extracts update data from the message, and generates real-time data according to the update data;

[0015] The real-time data is stored in the current real-time index through a search engine;

[0016] Obtain offline data from a big data warehouse, and store the offline data in an offline data index with a date identifier;

[0017] Determine whether to enable the offline data index as the primary index, and if the offline data index is enabled as the primary index, perform an index switching operation;

[0018] During the index switching process, the real-time data is written into the current real-time index and the offline data index;

[0019] Copy the real-time data that has been stored in the current real-time index before the index switching operation to the offline data index, so that the offline data index becomes the primary index;

[0020] Based on the primary index, a query operation is performed to obtain and return the query result.

[0021] Furthermore, to achieve the above object, the present invention provides a data synchronization and query device, comprising:

[0022] A business request receiving module, used for receiving business operation requests;

[0023] A business processing module, used to execute data update according to the business operation request, and generate update data including primary key ID, associated fields, update time, field information to be indexed and / or operation type;

[0024] A message queue module, used for encapsulating the update data into a message, and sending the message to the search service module through the message queue;

[0025] A search service module, configured to receive and parse the message, extract update data from the message, and generate real-time data according to the update data;

[0026] A real-time index module, used for storing the real-time data into the current real-time index through a search engine;

[0027] An offline index module, used to obtain offline data from a big data warehouse and store the offline data in an offline data index with a date mark;

[0028] A primary index module, used to determine whether to enable the offline data index as the primary index, and if the offline data index is enabled as the primary index, perform an index switching operation;

[0029] An index synchronization module, used to write the real-time data into the current real-time index and the offline data index during the index switching process;

[0030] A data migration module, used to copy the real-time data that has been stored in the current real-time index before the index switching operation to the offline data index, so that the offline data index becomes the primary index;

[0031] The query module is used to perform a query operation based on the primary index, and obtain and return the query result.

[0032] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a data synchronization and query program stored in the memory and executable on the processor, and the data synchronization and query program, when executed by the processor, implements the steps of the data synchronization and query method described above.

[0033] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, on which a data synchronization and query program is stored, and when the data synchronization and query program is executed by a processor, the steps of the data synchronization and query method described above are implemented.

[0034] Beneficial effects: The present invention relates to the fields of big data technology and financial technology, and discloses a data synchronization and query method, which generates update data including primary key ID, associated fields and update time by receiving business operation requests and executing data updates, encapsulates the update data into messages, transmits them to the search service module through the message queue, parses the messages and generates real-time data; the real-time data is stored in the real-time index through the search engine, the offline data is obtained from the big data warehouse, and stored in the offline index with a date mark, determines whether to enable the offline index as the main index, and writes real-time and offline data simultaneously during the index switching process; after completing the index switching, executes the query operation based on the main index, and returns the result. The present invention avoids the inconsistency caused by data separation by combining the unified processing of offline data and real-time data, ensures the efficiency and timeliness of data processing, realizes efficient support for synchronous update, index switching and query operations of the full amount of data in business scenarios, improves the system response speed, reduces the system complexity, and ensures the real-time and integrity of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0036] Figure 1 A schematic diagram of an application environment of a data synchronization and query method in an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a flow chart of an embodiment of a data synchronization and query method of the present invention;

[0038] Figure 3 A schematic diagram of functional modules of a preferred embodiment of the data synchronization and query device of the present invention;

[0039] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0040] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0042] The data synchronization and query method provided by the embodiment of the present invention can be applied in Figure 1In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can receive business operation requests through the user terminal and execute data updates to generate updated data. The updated data is encapsulated into a message, transmitted to the search service module through a message queue, the message is parsed and real-time data is generated; the real-time data is stored in the real-time index through the search engine, the offline data is obtained from the big data warehouse, and stored in the offline index with a date mark, and it is determined whether to enable the offline index as the main index, and the real-time and offline data are written simultaneously during the index switching process; after the index switching is completed, the query operation based on the main index is executed, and the result is returned. The present invention avoids the inconsistency caused by data separation by combining the unified processing of offline data and real-time data, ensures the efficiency and timeliness of data processing, realizes the synchronous update of the full amount of data in the business scenario, the index switching and the efficient support of the query operation, improves the system response speed, reduces the system complexity, and ensures the real-time and integrity of the data. Among them, the user terminal can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0043] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a data synchronization and query method provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0044] like Figure 2 As shown, the data synchronization and query method proposed by the present invention includes the following steps:

[0045] S10, receiving a business operation request;

[0046] In this embodiment, the system receives external operation requests, which are usually initiated by users or systems through API interfaces and contain specific business operation instructions. Business operation requests are the starting point of the entire data processing flow. These requests represent the user or system's desire to operate on data (add, modify or delete).

[0047] The business system receives these operation requests by listening to API requests, message queues, or RPC calls. For example, the POST, PUT, and DELETE requests of the REST API correspond to the operations of adding, modifying, and deleting data, respectively. In financial scenarios, operation requests may include customers submitting transactions, modifying account information, or deleting redundant data.

[0048] The specific content of a business operation request may include instructions for adding (Insert), modifying (Update) or deleting (Delete). These instructions determine the subsequent data operation behaviors, such as inserting a new record, updating existing data, or deleting it from the system.

[0049] When the system receives a business operation request, it will parse the instruction type in the request body. The instruction type is usually identified by the HTTP method (such as POST, PUT, DELETE) or the operation field in the message (such as "operation_type"). Depending on the instruction type, the system will call the corresponding logic module to perform data operations. For example, POST requests are used to add data, PUT requests are used to modify data, and DELETE requests are used to delete data. After receiving the instruction, the system will first verify the legitimacy of the request content to ensure that the operation is valid, and then convert it into a database operation or index operation.

[0050] Example description: In the application scenario of the financial industry, business operation requests can cover multiple scenarios:

[0051] New data: When a customer opens an account or conducts a new transaction, the system receives a new record containing transaction information by requesting the user.

[0052] Modify data: Update customer information or account status. The system receives the modification request and updates the customer's personal information.

[0053] Deleting data: When the system detects transaction records or expired data that are no longer needed, it receives a deletion request and cleans up the data.

[0054] Through this process, the financial system can flexibly manage data according to the instructions of users or back-end systems to ensure real-time updating and maintenance of data.

[0055] By receiving and parsing business operation requests, the system can flexibly respond to different types of data operation requirements and achieve efficient data processing and management. Whether it is adding, modifying or deleting operations, it can ensure the accuracy, completeness and timeliness of data, providing reliable data source support for subsequent data indexing and querying.

[0056] S20, executing data update according to the business operation request, generating update data including primary key ID, associated fields, update time, field information to be indexed and / or operation type;

[0057] In this embodiment, the actual database operation is performed based on the business operation request received previously. Whether it is a new addition, modification or deletion operation, the system updates the database accordingly according to the instructions in the business request. This process involves changing existing records or creating new records to ensure that the data is synchronized with the business logic.

[0058] The system will first verify and parse the requested data, and confirm whether it is a new, modified, or deleted operation based on the request content. For a new operation, the system will generate a new record and write the data to the database; for a modification operation, the system will find the corresponding record and update the corresponding field; for a deletion operation, the system will delete the specific record from the database. These operations are usually completed using SQL queries (such as INSERT INTO, UPDATE, DELETE) or NoSQL update methods.

[0059] After executing the data operation, the system needs to generate an updated data object containing key information. This object mainly includes:

[0060] Primary key ID: A unique identifier used to accurately locate a data record.

[0061] Related fields: Fields that are associated with other tables or data to facilitate establishing relationships between data.

[0062] Update time: The timestamp that identifies the current data operation, used to track the last update time of the data.

[0063] Field information that needs to be indexed: key fields to be indexed in the search engine, which are used for subsequent fast retrieval.

[0064] Operation Type: Indicates whether the operation is add, modify, or delete.

[0065] When implemented, the system will automatically generate an update object containing the above fields after completing the data operation. This object is used for subsequent index updates or synchronization operations. For example, in a customer information system, when a user modifies a customer's contact information, the system will generate an update data containing the customer ID (primary key ID), the associated account ID (associated field), the modification time (update time), and the contact field information that needs to be indexed. The operation type will mark this operation as "modification".

[0066] Example: In financial business scenarios, data updates may involve the management of customer information, transaction records, and other data. For example, when a customer submits a new transaction, the system will write the transaction information (such as amount, account ID, etc.) into the database through a business operation request. After the system completes the new operation, it will generate an update data, including the transaction's unique ID (primary key ID), account ID (associated field), transaction time (update time), transaction amount (field information that needs to be indexed), and operation type (new).

[0067] By executing data updates according to business operation requests, the system can generate updated data objects containing key fields and operation types to achieve effective tracking and management of data. This process provides an accurate and complete data source for subsequent data indexing and query operations, ensuring efficient operation of the system and data consistency.

[0068] S30, encapsulating the update data into a message, and sending the message to the search service module through a message queue;

[0069] In this embodiment, after executing the data update and generating the updated data, the system encapsulates the data into a message. This means that the updated data will be structured and packaged into a message format that can be transmitted. The content of the message contains key information such as the primary key ID, associated fields, update time, and operation type, ensuring that the data carried in the message can accurately describe the update operation. The process of encapsulating into a message usually follows a message transmission protocol, such as JSON or XML format, to ensure that the message can be correctly parsed between different systems.

[0070] The system will package the updated data into a predefined message format. For example, using the JSON format, the fields in the updated data (such as primary key ID, associated fields, operation type, etc.) will be encapsulated as JSON objects. The message will be structured according to the standardized message protocol to facilitate the subsequent system to correctly receive and parse the data. The system can use existing serialization libraries (such as Gson, Jackson, etc.) for data serialization.

[0071] After the message is encapsulated, the system sends the message to the search service module through the message queue. The message queue is used to transmit data asynchronously to ensure that business operations are not blocked after the data is updated. As a middleware, the message queue is responsible for passing messages from the producer (data update module) to the consumer (search service module) so that the search service can obtain the updated data in time and update the index.

[0072] Message queue systems usually adopt a publish-subscribe model or a point-to-point model. In the publish-subscribe model, message producers send messages to queues, and multiple consumers can receive messages from the queues. In the point-to-point model, producers and consumers correspond one to one, and consumers receive and process messages from queues. Typical message queue systems include Kafka, RabbitMQ, ActiveMQ, etc. The system publishes encapsulated messages to the specified queue or topic through the message queue API. Consumers (search service modules) receive and process messages by listening to the queue or topic.

[0073] Example description: In financial scenarios, message queues can be used to transmit updated data in real time to ensure data synchronization and consistency in financial systems. For example, when a customer submits a new transaction or modifies account information, the system will encapsulate this data into a message and send it to the search service module through the message queue. The content of the message may include important information such as transaction ID, account ID, transaction amount, etc. By using message queues, the system can achieve asynchronous transmission, ensure the real-time and efficient processing of financial transactions, and avoid blocking the business system due to synchronous operations.

[0074] By encapsulating the updated data into messages and sending them through message queues, the system achieves asynchronous data transmission, ensuring the efficiency and real-time performance of the data update process. This not only reduces the load on the business system, but also ensures that the search service module can obtain updated data in a timely manner and update the index, thereby improving the response speed of the entire system and data consistency.

[0075] S40, the search service module receives and parses the message, extracts update data from the message, and generates real-time data according to the update data;

[0076] In this embodiment, the search service module acts as a consumer of the message queue and is responsible for receiving data update messages from the message queue. In this step, the system needs to parse the message and obtain the updated data therein. The message is usually a structured data object containing information such as the primary key ID, associated fields, and operation type. The task of the search service module is to correctly parse these fields and identify the content and type of the message.

[0077] When implemented, the search service module listens to the specified message queue (such as Kafka Topic or RabbitMQQueue). When a new message is received, the module automatically triggers the message processing mechanism. Message parsing usually depends on the message format, such as JSON, XML, etc. The system can use the corresponding parsing library (such as Jackson, Gson, etc.) to deserialize the message, convert the message into a processable object, and extract key information such as primary key ID, associated fields, and operation type.

[0078] After parsing the message, the system needs to extract key update data from the message. The update data includes the content previously encapsulated in the message, such as the primary key ID (unique identifier), associated fields (other associated data or records), update time, operation type (add, modify or delete), and field information that needs to be indexed. This data will be used to update the index in the search engine to maintain the real-time and accuracy of the data.

[0079] After parsing the message, the system extracts the fields in the message one by one and stores them in the system's memory or temporary database. The extracted fields include primary key ID (used to identify data records), associated fields (used to maintain data association), operation type (determine how to process the index), etc. The extracted updated data can be saved in the memory structure for subsequent index operations.

[0080] The system uses the extracted updated data to generate real-time data. Real-time data refers to the latest data that will be updated in the search engine index. The process of generating real-time data includes creating new data records, updating existing data, or deleting data based on the type of operation. The system will handle different operations accordingly.

[0081] When implementing, the system will first determine the operation type (add, modify, or delete), and then decide whether to create a new data record, modify existing data, or delete the corresponding record from the index based on the primary key ID. The real-time data generation process relies on search engines (such as Elasticsearch) and interacts through its API interface to ensure that the data index is updated in a timely manner.

[0082] Example: In the financial field, when a customer submits a new transaction, the system sends the transaction data to the search service module through the message queue. The search service module receives and parses the message, extracting data such as the transaction ID, customer ID, transaction amount, and transaction time. Based on the transaction type (such as a new transaction), the search service module generates a new transaction record and stores it as real-time data in the search engine's index. This process ensures that transaction data in the financial system can be quickly retrieved and reflected in customer account information in a timely manner.

[0083] By receiving and parsing messages through the search service module, extracting updated data from the messages and generating real-time data, the system can ensure that the data in the search engine is updated in real time. This not only improves the efficiency of data retrieval, but also ensures the overall consistency and real-time performance of the system, especially when processing high-frequency financial data, it can effectively ensure the accuracy and timeliness of transaction and account information.

[0084] S50, storing the real-time data into the current real-time index through a search engine;

[0085] In this embodiment, the generated real-time data is stored in the current real-time index through a search engine (such as Elasticsearch, etc.). The search engine provides efficient data storage and retrieval capabilities, and organizes data in the form of indexes to facilitate subsequent query operations. Real-time data is data that has been processed by the system and needs to be retrieved quickly. Storing it through a search engine can ensure that users or systems can quickly obtain this data.

[0086] The system will call the API interface provided by the search engine (such as Elasticsearch's IndexAPI), send the extracted real-time data to the search engine, and specify the index for storage. After the data is stored in the index, it can be sharded and replicated, which enhances the concurrency and fault tolerance of queries. The fields of real-time data (such as primary key ID, update time, operation type, etc.) will be mapped to specific fields in the index for subsequent query and retrieval.

[0087] The current real-time index is where the latest data is stored in the search engine. Unlike other indexes (such as offline data indexes), the real-time index always contains the latest data generated by the system during real-time processing. When the system writes to the real-time index, it ensures the uniqueness and consistency of the data. For example, if the data already exists in the index, the system may perform an update operation instead of a new operation.

[0088] When storing real-time data, the system will check the primary key ID to ensure the uniqueness of the data. If the record already exists, the search engine will perform an update operation (through Elasticsearch's _update operation), otherwise, it will perform an insert operation. Real-time indexes are usually defined as high-priority indexes, and the system will ensure their efficiency when querying.

[0089] Example: In the financial industry, when a new transaction occurs, the transaction data is immediately processed and converted into real-time data. For example, information including transaction amount, timestamp, and account ID is processed by the system and passed to the search engine. Through search engines such as Elasticsearch, these real-time data are stored in real-time indexes, ensuring that subsequent queries and data analysis can quickly obtain the latest transaction records. In some real-time monitoring scenarios, financial institutions can immediately capture anomalies or risk signals in high-frequency transactions by querying data in real-time indexes.

[0090] By storing real-time data in real-time indexes through search engines, the system can achieve efficient retrieval and timely update of data. Real-time indexes ensure the rapid availability of data after business operations, especially in high-frequency data processing scenarios (such as financial transactions), which can ensure the real-time and consistency of data.

[0091] S60, acquiring offline data from a big data warehouse, and storing the offline data in an offline data index with a date identifier;

[0092] In this embodiment, batches of offline data are extracted from a big data warehouse (such as Hadoop, Hive, etc.). These data are usually historical data or data generated in a scheduled batch manner, and may include historical transaction records of users, long-term customer information, etc. In business scenarios, offline data is periodic batch data that needs to be processed on a large scale.

[0093] The system extracts data from the big data warehouse through scheduled tasks or API interfaces. Common implementations include using big data frameworks (such as Spark or MapReduce) to run batch tasks and periodically read table data in the data warehouse. These tasks are executed at predefined time intervals, such as daily, weekly, or monthly, to load data from the data warehouse for a specific time period, such as a data set within a specified date range. This operation can be implemented through Hive SQL queries, Hadoop's HDFS interface, or other APIs.

[0094] The data extracted from the big data warehouse will be stored in an offline data index with a date tag. By adding a date tag to the index (such as index_20231015), it can be ensured that each batch of offline data can be stored and distinguished independently to avoid data overwriting. At the same time, the index with a date tag facilitates the management and query of historical data.

[0095] The system stores the extracted offline data in the index of the search engine (such as Elasticsearch). The index name is usually dynamically generated according to the import date of the data to ensure that the index name is unique after each data load. The bulk API of the search engine can be used to write data in batches. When storing, each document (record) contains its timestamp or other relevant identification fields to ensure that the data can be sharded and stored according to the date. This process is implemented through the index management function of Elasticsearch to ensure safe storage and fast retrieval of data.

[0096] Example: In the financial field, offline data usually involves historical transaction records, account change information, etc. After the end of each working day, financial institutions will extract all transaction data for the day from the big data warehouse and import it into the offline data index of the search engine. This index will have a date mark for the day, such as "transaction record_20231015", which is used to record the transaction data of the day. By adding a date mark to each offline index, the system can trace back historical data at any time, analyze the transaction situation of each working day, and avoid data coverage or confusion. At the same time, financial regulators can also quickly find transaction information that needs to be reviewed through date marks.

[0097] By extracting offline data from the big data warehouse and storing it in an index with a date mark, the system can effectively manage a large amount of historical data, ensure the clear separation and independent storage of different batches of data, and avoid data overwriting. It not only enhances the traceability of historical data, but also improves the efficiency of data query, which is helpful for processing large-scale historical transaction data and compliance review in the financial industry.

[0098] S70, determining whether to enable the offline data index as the primary index, and if the offline data index is enabled as the primary index, performing an index switching operation;

[0099] In this embodiment, by analyzing the status of the current offline data index, it is decided whether to replace it with the primary index. The primary index is the core index used for data query and business processing, and usually contains the latest and complete data. In this step, the system determines whether to enable the offline data index based on various conditions (such as data integrity, update timestamp, index status, etc.). The premise for enabling the offline data index is that the data in the index is already newer and more complete than the data in the current real-time index.

[0100] The system compares the data status of the offline data index with the current main index to determine whether the offline data has higher timeliness and integrity. Common judgment criteria include: whether the number of offline data records meets expectations, whether the data update in the index is complete, and whether the system resource load can support index switching. If these conditions are met, the system will make a decision to enable the offline data index and prepare for the switch.

[0101] The index switching operation refers to replacing the current primary index used for querying with the offline data index. The index switching operation is dynamic to ensure that data processing is not interrupted. The system needs to ensure the synchronization of real-time data and offline data during the switch to avoid data loss or query inconsistency. After the switch operation is completed, the offline data index will become the new primary index, and subsequent query requests will be based on this index.

[0102] During implementation, the system uses a dual-write strategy to ensure that during the index switching process, real-time data is written to both the new offline data index and the old real-time index. After the index switching is completed, the system will point all new data operations to the offline data index, making it the new primary index. The switched offline data index begins to take over all query operations, and the original real-time index may be archived or continue to be used for historical queries.

[0103] Example: In financial data processing, the index switching operation can be used to process large amounts of historical data. For example, a bank extracts customer transaction records from a big data warehouse every day and stores them in an offline data index. After confirming that the offline data index contains the latest transaction data and meets the integrity requirements, the system will switch the offline index to the primary index to ensure that customers can obtain the latest transaction records when querying transactions. This switching operation can be performed without interrupting transaction services, ensuring the real-time and accuracy of data.

[0104] By determining whether to enable the offline data index as the primary index, the system can flexibly switch indexes while ensuring data integrity and real-time performance. This not only improves data query efficiency, but also ensures that the system can dynamically respond to data changes, especially in financial business scenarios involving large-scale data processing, effectively avoiding data loss or query inconsistency.

[0105] S80, during the index switching process, writing the real-time data into the current real-time index and the offline data index;

[0106] In this embodiment, during the index switching process, the system needs to ensure data consistency and synchronization, so the real-time data will be written to two indexes at the same time: the current real-time index and the new offline data index. This mechanism is called a double-write strategy, and its purpose is to ensure that real-time data can exist in both indexes before the index switching is completed to avoid data loss or query inconsistency. When the new offline data index is successfully switched to the primary index, new data operations will continue to point to the new primary index.

[0107] Every time the system receives real-time data, it calls two write operations, one to write the current real-time index and one to write the new offline data index. This can be achieved through the search engine's API interface (such as Elasticsearch's IndexAPI or BulkAPI). The double write operation of real-time data during index switching can ensure the synchronization of data in the two indexes. The double write strategy will continue to be executed during the index switching process until the new index completely replaces the current real-time index.

[0108] The purpose of the dual write operation is to ensure data synchronization and consistency. Since new real-time data is still being generated during the index switching process, it is necessary to ensure the consistency of the data in the two indexes. Before the index switching is completed, all real-time data will be written to both indexes at the same time to ensure that the offline data index can contain all the latest real-time data when it is switched to the primary index.

[0109] When the search engine performs an index switch, it checks the status of each write operation to confirm whether the data has been successfully written to both indexes. If a write fails, the system will enable a retry mechanism to ensure that the data is fully written. A common implementation method is to use transactions or idempotent write operations to ensure the security and consistency of multiple writes. After the search engine completes the index switch, it will stop writing to the old real-time index, and new real-time data will only be written to the new primary index.

[0110] Example: In financial services, such as real-time transaction processing or account update operations, when the financial system switches from one index to another new offline data index, the dual-write strategy ensures that all real-time transaction data can be stored in both the old index and the new index. For example, when a customer initiates a transfer or modifies account information, the system will write these real-time update data to the current real-time index and the new offline index at the same time. Through this dual-write operation, the system can ensure that the new index contains all the latest transaction information after the index switch is completed, avoiding any omissions or data inconsistencies.

[0111] By writing real-time data into the current real-time index and offline data index simultaneously during the index switching process, the system can ensure data consistency and integrity, and avoid query errors caused by data loss or asynchrony during the index switching process. At the same time, the dual-write strategy can ensure a smooth transition of index switching and ensure the continuity of business operations, which is particularly suitable for scenarios such as financial services that have extremely high requirements for data consistency.

[0112] S90, copying the real-time data that has been stored in the current real-time index before the index switching operation to the offline data index, so that the offline data index becomes the primary index;

[0113] In this embodiment, during the index switching process, the system needs to ensure that all data stored in the current real-time index before the switch can be synchronously copied to the new offline data index. This operation is intended to ensure that the new index contains all the real-time data that has been stored before the switch operation, thereby maintaining data integrity.

[0114] During the implementation process, the system will perform a batch data copy operation before executing the index switch. Through the API provided by the search engine (such as Elasticsearch's reindexAPI), the data stored in the current real-time index (especially the data before the switch operation) is copied in batches to the new offline data index. This process ensures that all historical data is completely migrated to the new index to avoid data loss or inconsistency.

[0115] After the index switch is completed, the offline data index officially replaces the current real-time index and becomes the main index of the system. At this point, all query requests and data write operations will be performed on the new main index.

[0116] After the index switch is completed, the system will adjust the data query and write logic so that all new real-time data and query operations point to the offline data index. The system will enable write permissions for the new offline data index and stop writing to the old real-time index. This implementation is usually controlled through the search engine's index management API to ensure a smooth transition during the switch and subsequent data consistency.

[0117] Example description: In financial data processing scenarios, such as day-end clearing or large-scale historical transaction data processing, the system may need to copy all transaction data stored in the current real-time index to a new offline data index. After the copy is completed, the offline data index will become the new primary index. When users query historical data or conduct risk assessments, the system will automatically query the data in the primary index. This ensures data consistency during the switching process, especially providing important guarantees for time-sensitive data in financial transactions, avoiding data loss or delays.

[0118] By copying the real-time data in the current real-time index to the new offline data index and making it the primary index, the system can ensure the integrity and consistency of the data. It not only ensures the correct migration of historical data, but also supports smooth transition after index switching, avoiding inaccurate query results due to inconsistent data, and is particularly suitable for scenarios in financial services that require high data reliability.

[0119] S100, executing a query operation based on the primary index, obtaining and returning a query result.

[0120] In this embodiment, the system performs data retrieval operations through the primary index (enabled index) according to the user's query request. The primary index is the index currently used to store the latest and complete data, and all data queries are performed based on the primary index. This step ensures that users can quickly obtain the latest and processed results.

[0121] The system executes queries through the search engine's query API (such as Elasticsearch's Search API). The query conditions usually include the values ​​of specific fields (such as primary key ID, date range, keywords, etc.), and the search engine retrieves relevant data in the primary index based on these conditions. The search engine optimizes the query to the best performance through the index structure, supporting complex query operations such as full-text search and filtering. The efficient query performance of the primary index ensures that users can obtain results in a short time, especially in large-scale data query scenarios.

[0122] After the query is executed, the system will extract the data records that meet the query conditions from the primary index and return these records to the user or system as query results. The query results are usually returned in structured data (such as JSON format) for subsequent processing or display.

[0123] The query results are returned to the caller through the search engine API. The results may contain multiple records or pages. The system can further process the query results, such as sorting, aggregation, or filtering, to ensure that the results meet the user's expectations. The returned result set will be passed to the requesting client or user interface through the API, or further flow into the business processing logic.

[0124] Example description: In a financial scenario, when a user queries the historical transaction records of a specific account, the system will perform a query operation through the current primary index based on the account ID, date range and other conditions entered by the user. For example, if a customer wants to query transactions within the past month, the system will search for relevant transaction records in the enabled primary index and return the results quickly. Financial systems can use this query operation to meet complex scenarios such as risk assessment and customer analysis, ensuring that users can efficiently and accurately obtain the latest transaction data and analysis results.

[0125] By executing query operations based on the primary index and returning query results, the system can effectively provide fast and accurate data retrieval services. Whether in historical data retrieval or real-time data query, the efficient query mechanism of the primary index ensures the accuracy and consistency of the data, which helps the real-time decision-making and data analysis of the business, especially in financial business, which can greatly improve the system's response speed and user experience.

[0126] The present invention relates to the fields of big data technology and financial technology, and discloses a data synchronization and query method. By receiving a business operation request and executing data update, update data including a primary key ID, associated fields and update time is generated, and the update data is encapsulated into a message, which is passed to a search service module through a message queue, and the message is parsed and real-time data is generated. Real-time data is stored in a real-time index through a search engine, and offline data is obtained from a big data warehouse and stored in an offline index with a date mark. It is determined whether to enable the offline index as the primary index, and real-time and offline data are written simultaneously during the index switching process. After the index switching is completed, a query operation based on the primary index is executed and the result is returned. The present invention avoids the inconsistency caused by data separation by combining the unified processing of offline data and real-time data, ensures the efficiency and timeliness of data processing, realizes efficient support for synchronous update, index switching and query operations of the full amount of data in business scenarios, improves the system response speed, reduces the system complexity, and ensures the real-time and integrity of the data.

[0127] In one embodiment, the above S40 includes:

[0128] S401, the search service module receives a message body containing update data by monitoring a message queue;

[0129] S402, the search service module parses the message body and extracts update data from the message body;

[0130] S403: The search service module executes a generation, modification or deletion operation of real-time data according to the operation type in the update data.

[0131] In this embodiment, the search service module acts as a consumer of the message queue and receives update data by monitoring a specific channel or topic in the message queue. These message bodies contain detailed information of the update operation, such as key information such as primary key ID, associated fields, and update time.

[0132] The search service module can be implemented through the API or library provided by the message queue system (such as Kafka, RabbitMQ, etc.). During initialization, the search service module will subscribe to the specified message channel. When a new message enters the queue, the module will automatically receive and process the message. The monitoring mechanism provided by the message queue system ensures asynchronous reception of messages, reducing system congestion and delay.

[0133] After receiving the message body, the search service module needs to parse the message content to extract key information. This key information usually includes data related to the update operation, such as the primary key ID (used to identify data records), associated fields (used to record data relationships), operation type (add, modify or delete), and other fields.

[0134] The search service module will parse the message body in the format of JSON, XML, etc. Parsing tools (such as Jackson, Gson, etc.) can convert messages into data objects within the system. The extracted update data will be stored in memory or a temporary structure for subsequent operations.

[0135] The search service module performs specific operations based on the operation type provided in the message. The operation type can be insert, update, or delete. Depending on the operation type, the system will generate new real-time data in the search engine index, modify existing data, or delete specified data records.

[0136] The search service module performs corresponding operations based on the operation type in the message (such as adding, modifying or deleting) through the IndexAPI, UpdateAPI or DeleteAPI interfaces of the search engine (such as Elasticsearch). For example, the adding operation will insert a new document record into the index, the modifying operation will update the existing document content, and the deleting operation will remove the relevant documents in the index.

[0137] This embodiment uses the search service module to monitor the message queue, parse the message and perform data operations according to the operation type. The system can achieve rapid response and processing after the data is updated, ensuring the synchronization, real-time and consistency of the data in the search engine index. It not only improves the efficiency of data retrieval, but also ensures the data integrity and reliability of the entire system in high-concurrency business scenarios, and is particularly suitable for scenarios in financial business that require high data accuracy.

[0138] In one embodiment, after the above S40, the method further includes:

[0139] S404, identifying the missing necessary fields in the real-time data through the search service module according to the predefined field rules;

[0140] S405, querying an external data source to obtain a corresponding value of the missing necessary field, wherein the external data source includes a database or a large data warehouse;

[0141] S406, supplement the corresponding values ​​of the missing necessary fields into the real-time data.

[0142] In this embodiment, during the data processing, the system checks whether necessary fields are missing in the real-time data according to the pre-defined field rules. These necessary fields are the key part of the data integrity requirements and are usually used to ensure the accuracy of indexes and data queries. If some fields are missing, the system will further process to supplement the missing fields.

[0143] When implemented, the system checks the field integrity of each piece of real-time data through defined rules. The rules can include which fields must exist and which fields are optional. For example, primary key IDs, associated fields, etc. must be present. If some required fields are detected to be empty or missing, the system will record it and mark the data for further processing. Rules can be implemented using configuration files, database field constraints, or through programming logic.

[0144] If there are missing fields in the real-time data, the system needs to supplement these field values ​​by querying external data sources. External data sources can be relational databases, big data warehouses, or other systems that store complete historical data or other related data.

[0145] The search service module accesses external data sources (such as databases, Hadoop, etc.) through APIs, SQL queries, or data interfaces to obtain specific values ​​of missing fields. For example, if a piece of data is missing the detailed address of a user, the system can find and complete the field in the external data source by associating the user ID. The query logic can be implemented through predefined mapping rules, such as querying data in external systems based on primary keys or other associated fields.

[0146] After successfully querying the corresponding field values ​​of the external data source, the system adds these values ​​to the real-time data to ensure the integrity of each piece of data. This operation ensures that the storage and use of real-time data in the search index is complete and can meet subsequent query requirements.

[0147] During implementation, the system writes the field values ​​queried externally into the records of real-time data through in-memory data structures or intermediate storage. After data supplementation is completed, the real-time data will be submitted to the index again through the index update interface (such as Elasticsearch's UpdateAPI) to ensure that the data finally stored in the index is complete and meets business requirements.

[0148] This embodiment ensures the integrity and consistency of data by identifying necessary fields missing in real-time data and automatically querying external data sources to complete them. Especially in complex financial business scenarios, data integrity is a key factor that directly affects subsequent queries, analysis, and decision-making. This mechanism effectively improves the system's automated processing capabilities, reduces errors caused by manual operations, and significantly improves the accuracy and timeliness of data processing.

[0149] In one embodiment, the above S60 includes:

[0150] S601, setting a timed scheduling task, wherein the timed scheduling task is used to trigger an operation of extracting data from a big data warehouse according to a preset time interval, and generate a corresponding scheduling time each time scheduling;

[0151] S602, generating a corresponding query condition based on the scheduling time, wherein the query condition is used to determine the time period data to be extracted according to the timestamp or date field;

[0152] S603, based on the query condition, obtaining offline data of a corresponding time period from the big data warehouse;

[0153] S604, storing the offline data in an offline data index with a date identifier, where the date identifier is generated according to a corresponding scheduling time;

[0154] S605: Setting an index management strategy for the offline data index.

[0155] In this embodiment, the system manages the operation of regularly extracting data from the big data warehouse through scheduled scheduling tasks. The scheduling task is automatically triggered according to a preset time interval (such as daily, weekly or monthly), thereby ensuring the regular extraction of batches of historical data. During implementation, the system will use a scheduling management tool (such as Cron, Quartz scheduler) to configure the scheduled task to ensure that the query task of the big data warehouse is triggered at the set time point. Each time a task is executed, the system will generate a scheduling time and record the current scheduling time for subsequent processing. The timestamp will be used to generate query conditions.

[0156] The system generates query conditions based on the time of each scheduled task to determine the time period data that needs to be extracted from the big data warehouse. The query conditions mainly rely on timestamp or date fields to ensure that the data range extracted each time is accurate. The system generates conditions for querying the big data warehouse by combining the scheduling time with the preset query template. Usually the query conditions specify the start and end time periods. For example, if the scheduling time is executed every day, the query condition may be the data time period of the previous day (such as WHERE date>='2023-10-15'AND date<'2023-10-16').

[0157] After generating the query conditions, the system will extract offline data from the big data warehouse based on the conditions. Big data warehouses (such as Hadoop, Hive, etc.) store a large amount of historical data. By querying and extracting data for a specific time period, it is ensured that the latest historical data is obtained each time a task is scheduled. When implemented, the system executes queries through big data query interfaces (such as Hive SQL, HDFS API) to obtain offline data that meets the query conditions. This data is usually extracted in batches and temporarily stored in the system memory or buffer, waiting for subsequent storage operations.

[0158] The extracted offline data needs to be stored in an offline data index with a date tag. The date tag is generated according to the scheduling time and is used to identify the time point of data storage to avoid overwriting previous data records. The system writes the extracted offline data to a specific index through the index management function of a search engine (such as Elasticsearch). For example, each time a scheduling task is executed, the system creates a new index (such as offline_data_20231015) and writes the data of the day to the index. The index name usually has a timestamp to facilitate subsequent data management and query.

[0159] In order to effectively manage historical data, the system needs to set corresponding index management strategies for offline data indexes, including automatically deleting historical data indexes or merging multiple indexes. The purpose of these strategies is to avoid taking up too much storage space and ensure the retrieval performance of the system. The system can set index management strategies through the lifecycle management function of the search engine (such as Elasticsearch's lifecycle management ILM). For example, automatically delete historical indexes that are older than a certain period of time (such as data from 30 days ago), or merge multiple small indexes into a larger index to reduce query overhead. Index management strategies are customized according to business needs in order to control storage costs and optimize system performance.

[0160] This embodiment can efficiently manage large-scale historical data by setting up scheduled tasks, generating query conditions, and automatically extracting offline data. By storing these data in indexes with date marks and controlling them using index management policies, the system achieves efficient data storage, flexible management, and fast query. It reduces the complexity of manual intervention, ensures the integrity and real-time nature of the data, and can provide strong support for end-of-day processing, compliance checks, etc., especially in financial services.

[0161] In one embodiment, the above S80 includes:

[0162] S801, each time real-time data is received, the real-time data is written into the current real-time index and the offline data index through a double-write strategy;

[0163] S802, confirming whether the real-time data has been successfully written into the current real-time index and the offline data index;

[0164] S803, when any index write fails, start the retry mechanism in the dual write strategy to rewrite the real-time data into the failed index;

[0165] S804, after completing the index switching operation and confirming through comparison that all real-time data is completely synchronized in the current real-time index and the offline data index, stop executing the dual-write strategy;

[0166] S805: After the index switching operation is completed, the received real-time data will only be written into the offline data index.

[0167] In this embodiment, during the index switching process, the double write strategy is used to ensure that the real-time data is synchronized in the two indexes. Whenever the system receives real-time data, the system will write both the current real-time index and the offline data index that will become the primary index to ensure data consistency. By configuring the double write strategy, the system calls two write operations when receiving new real-time data, one to the current real-time index and the other to the offline data index. The double write operation can be completed through the search engine's BulkAPI or IndexAPI to ensure data synchronization.

[0168] The system needs to verify the success status of each write operation to ensure that the real-time data has been written to both indexes at the same time. The system confirms the success of each write operation through the search engine's response status code or log. If the write is successful, the system will continue to perform subsequent operations; if an error occurs, the system will start the retry mechanism.

[0169] When any index fails during the write process, the system starts a retry mechanism. The retry mechanism ensures that all data is written to both indexes to avoid data loss. When implementing the retry mechanism, the system captures the write failure and retries the write operation according to the configured retry logic until the operation succeeds or the preset retry limit is reached. You can set idempotent operations to ensure that data is not written repeatedly during retries.

[0170] When the index switching operation is completed, the system will perform a comparison to ensure that the data in the two indexes are completely synchronized. Once synchronization is confirmed, the system will stop the dual write operation so that subsequent real-time data is only written to the new primary index. The system can confirm data synchronization by comparing the amount of data or checksum in the two indexes. When the data in the two indexes is completely consistent, the system will update the index configuration, terminate the dual write strategy, and switch the new offline data index to the primary index.

[0171] When the index switching operation is successful, the new primary index will receive all real-time data, and the old real-time index will no longer be used. After the index switching is completed, the system updates the write logic to write all real-time data to the new offline data index. The new index becomes the new primary index, and subsequent data operations will be based on this index.

[0172] This embodiment adopts a dual-write strategy during the index switching process, so that the system can ensure the synchronization and consistency of real-time data and avoid the risk of data loss or asynchrony during the switching process. The retry mechanism further enhances the reliability of data writing and is particularly suitable for use in high-concurrency data processing scenarios. After the index switching is completed, the system's storage and query performance is also optimized.

[0173] In one embodiment, in the above S70, determining whether to enable the offline data index as the primary index includes:

[0174] S701, generating an integrity verification result by verifying whether the number of records in the offline data index matches the expected number, so as to verify the integrity of the offline data index;

[0175] S702, comparing the last update time of the offline data index and the current real-time index, and generating a timestamp comparison result, wherein the timestamp comparison result is used to analyze whether the update time of the data in the offline data index is later than the update time of the data in the real-time index;

[0176] S703: Based on the integrity verification result and the timestamp comparison result, combined with preset business rules and scheduling conditions, determine whether to enable the offline data index as the primary index.

[0177] In this embodiment, before enabling the offline data index, the system needs to verify the integrity of the index. By comparing the number of records in the index with the expected number, an integrity verification result is generated to ensure the accuracy and integrity of the data in the offline data index. The system uses the record statistics function to check the actual number of records in the offline data index and compare it with the expected number. If the two match, the index is considered complete; otherwise, a warning message will be generated, indicating that the index data is incomplete. The expected number is usually estimated based on the amount of data in the scheduling time period.

[0178] The system needs to compare the last update time of the offline data index and the current real-time index to generate a timestamp comparison result. This result is used to determine whether the data in the offline data index is newer than the data in the real-time index. When implementing, the system extracts the last modification time (timestamp) of the two indexes and compares them. If the timestamp of the offline data index is later than that of the current real-time index, it means that the offline data is relatively fresh and can be switched as the primary index. Otherwise, the system will not perform the switching operation.

[0179] After integrity verification and timestamp comparison, the system determines whether to enable the offline data index as the primary index based on these results combined with business rules and scheduling conditions. The system makes decisions based on scheduling conditions, business rules, and verification and comparison results. If the data in the offline data index is complete and updated later than the real-time index, the system will automatically enable the offline data index as the primary index. Business rules may include business processing priorities or certain specific scheduling strategies.

[0180] This embodiment verifies the integrity and timestamp of offline data indexes, so that the system can ensure that complete and up-to-date data is used when switching to a new index. This not only improves the accuracy of data, but also ensures the stability and efficiency of the system, which greatly improves the reliability of the system, especially in the financial field that requires high data consistency and timeliness.

[0181] In one embodiment, the above S100 includes:

[0182] S1001, receiving a query request, and extracting a query condition from the query request;

[0183] S1002, based on the query condition, performing a data query operation in the primary index to obtain a query result;

[0184] S1003: Return the query result to the initiator of the query request.

[0185] In this embodiment, the system receives a query request from a user or application. The query request usually contains conditions such as time range, field name, keywords, etc. to specify the content to be retrieved. The system receives the query request through an API interface or a query module of a search engine (such as a RESTful interface or GraphQL). The content in the request body is parsed to extract the query conditions, which may include a start date, an end date, a specific field, or a keyword. These conditions will be used for data retrieval in subsequent steps.

[0186] The system performs data retrieval operations on the primary index based on the extracted query conditions. The primary index is the currently enabled index that contains the latest and complete data, and the query results are returned based on this index. When implemented, the system uses the search engine's query interface (such as Elasticsearch's SearchAPI) to perform query operations. The query conditions are passed to the index and the index's data structure is used for retrieval. The search engine can support multiple query methods such as full-text search, keyword matching, time range filtering, etc. to ensure that data records that meet the conditions are accurately extracted.

[0187] After the data retrieval is completed, the system packages the query results and returns them to the initiator of the query request. The query results may contain one or more records that meet the conditions, and the specific format can be JSON, XML, or other formats suitable for machine processing. The query results are usually returned to the caller through the API interface, and the system will encapsulate the data according to the requested format or client requirements. The returned data can be displayed directly to the user or further processed by the application. Through paging or batch processing, the system can efficiently return large-scale data sets to ensure performance.

[0188] Example: In a financial data processing scenario, a user may initiate a query request to retrieve customer transaction records within a specific date range. After receiving the query request, the system extracts the query conditions, such as the time range (such as January 1, 2023 to January 31, 2023), field name (such as customer ID, transaction amount), or keyword (such as "risk"). Based on these conditions, the system retrieves the transaction records that meet the conditions in the current primary index and returns the results to the query request initiator for subsequent data analysis or auditing.

[0189] By performing query operations based on the primary index, this embodiment enables the system to efficiently and accurately retrieve data that meets user needs and promptly return query results to the initiator. This improves the speed and accuracy of data queries, especially in financial business scenarios that require processing large amounts of data, ensuring the system's responsiveness and supporting real-time analysis and decision-making.

[0190] In one embodiment, a data synchronization and query device is provided, and the data synchronization and query device corresponds one-to-one to the data synchronization and query method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the data synchronization and query device of the present invention. Business request receiving module 10, business processing module 20, message queue module 30, search service module 40, real-time index module 50, offline index module 60, main index module 70, index synchronization module 80, data migration module 90 and query module 100. The functional modules are described in detail as follows:

[0191] A service request receiving module 10, used to receive a service operation request;

[0192] The business processing module 20 is used to execute data update according to the business operation request and generate update data including primary key ID, associated fields, update time, field information to be indexed and / or operation type;

[0193] A message queue module 30, used to encapsulate the update data into a message, and send the message to the search service module through the message queue;

[0194] A search service module 40, configured to receive and parse the message, extract update data from the message, and generate real-time data according to the update data;

[0195] A real-time indexing module 50, used to store the real-time data into a current real-time index through a search engine;

[0196] An offline index module 60 is used to obtain offline data from a big data warehouse and store the offline data in an offline data index with a date mark;

[0197] The main index module 70 is used to determine whether to enable the offline data index as the main index, and if the offline data index is enabled as the main index, perform an index switching operation;

[0198] An index synchronization module 80, used to write the real-time data into the current real-time index and the offline data index during the index switching process;

[0199] A data migration module 90, used to copy the real-time data that has been stored in the current real-time index before the index switching operation to the offline data index, so that the offline data index becomes the primary index;

[0200] The query module 100 is used to perform a query operation based on the primary index, and obtain and return a query result.

[0201] In one embodiment, the search service module 40 is specifically used for:

[0202] The search service module receives the message body containing the updated data by monitoring the message queue;

[0203] The search service module parses the message body and extracts update data from the message body;

[0204] The search service module executes the operation of generating, modifying or deleting the real-time data according to the operation type in the update data.

[0205] In one embodiment, the search service module 40 is specifically used for:

[0206] According to predefined field rules, identifying necessary fields missing in real-time data through the search service module;

[0207] Querying an external data source to obtain corresponding values ​​of the missing necessary fields, the external data source including a database or a big data warehouse;

[0208] The corresponding values ​​of the missing necessary fields are supplemented into the real-time data.

[0209] In one embodiment, the offline indexing module 60 is specifically configured to:

[0210] Setting a scheduled scheduling task, which is used to trigger the operation of extracting data from the big data warehouse according to a preset time interval and generate a corresponding scheduling time each time scheduling;

[0211] Generate a corresponding query condition based on the scheduling time, the query condition is used to determine the time period data to be extracted according to the timestamp or date field;

[0212] Based on the query condition, obtaining offline data of a corresponding time period from the big data warehouse;

[0213] Storing the offline data in an offline data index with a date identifier, wherein the date identifier is generated according to a corresponding scheduling time;

[0214] An index management strategy is set for the offline data index.

[0215] In one embodiment, the index synchronization module 80 is specifically configured to:

[0216] Every time real-time data is received, the real-time data is written to the current real-time index and offline data index through the dual-write strategy;

[0217] Confirm whether the real-time data has been successfully written into the current real-time index and offline data index;

[0218] When any index fails to be written, the retry mechanism in the dual-write strategy is started to rewrite the real-time data into the failed index;

[0219] After the index switching operation is completed and all real-time data is confirmed to be fully synchronized in the current real-time index and the offline data index through comparison, the dual-write strategy is stopped;

[0220] After the index switching operation is completed, the received real-time data will only be written into the offline data index.

[0221] In one embodiment, the main index module 70 is specifically configured to:

[0222] By verifying whether the number of records in the offline data index matches the expected number, an integrity verification result is generated to verify the integrity of the offline data index;

[0223] Compare the last update time of the offline data index and the current real-time index to generate a timestamp comparison result, wherein the timestamp comparison result is used to analyze whether the update time of the data in the offline data index is later than the update time of the data in the real-time index;

[0224] Based on the integrity verification result and the timestamp comparison result, combined with preset business rules and scheduling conditions, it is determined whether to enable the offline data index as the primary index.

[0225] In one embodiment, the query module 100 is specifically configured to:

[0226] receiving a query request, and extracting a query condition from the query request;

[0227] Based on the query condition, perform a data query operation in the primary index to obtain a query result;

[0228] The query result is returned to the initiator of the query request.

[0229] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes a function or step on the server side of a data synchronization and query method.

[0230] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a user side of a data synchronization and query method

[0231] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0232] Receive business operation requests;

[0233] Execute data update according to the business operation request, and generate update data including primary key ID, associated fields, update time, field information to be indexed and / or operation type;

[0234] Encapsulating the update data into a message, and sending the message to the search service module through a message queue;

[0235] The search service module receives and parses the message, extracts update data from the message, and generates real-time data according to the update data;

[0236] The real-time data is stored in the current real-time index through a search engine;

[0237] Obtain offline data from a big data warehouse, and store the offline data in an offline data index with a date identifier;

[0238] Determine whether to enable the offline data index as the primary index, and if the offline data index is enabled as the primary index, perform an index switching operation;

[0239] During the index switching process, the real-time data is written into the current real-time index and the offline data index;

[0240] Copy the real-time data that has been stored in the current real-time index before the index switching operation to the offline data index, so that the offline data index becomes the primary index;

[0241] Based on the primary index, a query operation is performed to obtain and return the query result.

[0242] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0243] Receive business operation requests;

[0244] Execute data update according to the business operation request, and generate update data including primary key ID, associated fields, update time, field information to be indexed and / or operation type;

[0245] Encapsulating the update data into a message, and sending the message to the search service module through a message queue;

[0246] The search service module receives and parses the message, extracts update data from the message, and generates real-time data according to the update data;

[0247] The real-time data is stored in the current real-time index through a search engine;

[0248] Obtain offline data from a big data warehouse, and store the offline data in an offline data index with a date identifier;

[0249] Determine whether to enable the offline data index as the primary index, and if the offline data index is enabled as the primary index, perform an index switching operation;

[0250] During the index switching process, the real-time data is written into the current real-time index and the offline data index;

[0251] Copy the real-time data that has been stored in the current real-time index before the index switching operation to the offline data index, so that the offline data index becomes the primary index;

[0252] Based on the primary index, a query operation is performed to obtain and return the query result.

[0253] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0254] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0255] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0256] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data synchronization and query method, characterized in that: The following steps are involved: Receive business operation requests; Execute data update according to the business operation request, and generate update data including primary key ID, associated fields, update time, field information to be indexed and / or operation type; Encapsulating the update data into a message, and sending the message to the search service module through a message queue; The search service module receives and parses the message, extracts update data from the message, and generates real-time data according to the update data; The real-time data is stored in the current real-time index through a search engine; Obtain offline data from a big data warehouse, and store the offline data in an offline data index with a date identifier; Determine whether to enable the offline data index as the primary index, and if the offline data index is enabled as the primary index, perform an index switching operation; During the index switching process, the real-time data is written into the current real-time index and the offline data index; Copy the real-time data that has been stored in the current real-time index before the index switching operation to the offline data index, so that the offline data index becomes the primary index; Based on the primary index, a query operation is performed to obtain and return the query result.

2. The data synchronization and query method according to claim 1, characterized in that: The search service module receives and parses the message, extracts update data from the message, and generates real-time data according to the update data, including: The search service module receives the message body containing the updated data by monitoring the message queue; The search service module parses the message body and extracts update data from the message body; The search service module executes the operation of generating, modifying or deleting the real-time data according to the operation type in the update data.

3. The data synchronization and query method according to claim 1, characterized in that: After the search service module receives and parses the message, extracts the update data from the message, and generates real-time data according to the update data, it further includes: According to predefined field rules, identifying necessary fields missing in real-time data through the search service module; Querying an external data source to obtain corresponding values ​​of the missing necessary fields, the external data source including a database or a big data warehouse; The corresponding values ​​of the missing necessary fields are supplemented into the real-time data.

4. The data synchronization and query method as claimed in claim 1, characterized in that: Obtaining offline data from a big data warehouse and storing the offline data in an offline data index with a date identifier includes: Setting a scheduled scheduling task, which is used to trigger the operation of extracting data from the big data warehouse according to a preset time interval and generate a corresponding scheduling time each time scheduling; Generate a corresponding query condition based on the scheduling time, the query condition is used to determine the time period data to be extracted according to the timestamp or date field; Based on the query condition, obtaining offline data of a corresponding time period from the big data warehouse; Storing the offline data in an offline data index with a date identifier, wherein the date identifier is generated according to a corresponding scheduling time; An index management strategy is set for the offline data index.

5. The data synchronization and query method according to claim 1, characterized in that: During the index switching process, writing the real-time data into the current real-time index and the offline data index includes: Every time real-time data is received, the real-time data is written to the current real-time index and offline data index through the dual-write strategy; Confirm whether the real-time data has been successfully written into the current real-time index and offline data index; When any index fails to be written, the retry mechanism in the dual-write strategy is started to rewrite the real-time data into the failed index; After the index switching operation is completed and all real-time data is confirmed to be fully synchronized in the current real-time index and the offline data index through comparison, the dual-write strategy is stopped; After the index switching operation is completed, the received real-time data will only be written into the offline data index.

6. The data synchronization and query method according to claim 1, characterized in that: Determining whether to enable the offline data index as the primary index includes: By verifying whether the number of records in the offline data index matches the expected number, an integrity verification result is generated to verify the integrity of the offline data index; Compare the last update time of the offline data index and the current real-time index to generate a timestamp comparison result, wherein the timestamp comparison result is used to analyze whether the update time of the data in the offline data index is later than the update time of the data in the real-time index; Based on the integrity verification result and the timestamp comparison result, combined with preset business rules and scheduling conditions, it is determined whether to enable the offline data index as the primary index.

7. The data synchronization and query method according to claim 1, characterized in that: Based on the primary index, a query operation is performed to obtain and return query results, including: receiving a query request, and extracting a query condition from the query request; Based on the query condition, perform a data query operation in the primary index to obtain a query result; The query result is returned to the initiator of the query request.

8. A data synchronization and query device, characterized in that: The data synchronization and query device comprises: A business request receiving module, used for receiving business operation requests; A business processing module, used to execute data update according to the business operation request, and generate update data including primary key ID, associated fields, update time, field information to be indexed and / or operation type; A message queue module, used for encapsulating the update data into a message, and sending the message to the search service module through the message queue; A search service module, configured to receive and parse the message, extract update data from the message, and generate real-time data according to the update data; A real-time index module, used for storing the real-time data into the current real-time index through a search engine; An offline index module, used to obtain offline data from a big data warehouse and store the offline data in an offline data index with a date mark; A primary index module, used to determine whether to enable the offline data index as the primary index, and if the offline data index is enabled as the primary index, perform an index switching operation; An index synchronization module, used to write the real-time data into the current real-time index and the offline data index during the index switching process; A data migration module, used to copy the real-time data that has been stored in the current real-time index before the index switching operation to the offline data index, so that the offline data index becomes the primary index; The query module is used to perform a query operation based on the primary index, and obtain and return the query result.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a data synchronization and query program stored in the memory and executable on the processor. When the data synchronization and query program is executed by the processor, the steps of the data synchronization and query method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a data synchronization and query program, which, when executed by a processor, implements the steps of the data synchronization and query method according to any one of claims 1 to 7.

Images