Data updating method and device, computer equipment, storage medium and program product

Abstract

The invention discloses a data updating method and device, computer equipment, a storage medium and a program product, and belongs to the technical field of databases. The method comprises the steps that new data are written into a first index, a random document identifier is generated for a document where each piece of new data is located, and the new data comprise timestamp fields recording data writing time; querying all the data in the first index, grouping all the data, sorting the data in each group according to timestamp fields of all the data, and screening the data in each group to obtain screened data; and performing an initialization operation on the first index every a first time period. According to the method, data are screened through timestamp sorting and business primary key grouping, and the problems of data updating efficiency and accuracy of Elasticsearch in an online synchronous service scene are solved in combination with periodic index initialization.

Description

Technical Field

[0001] This invention relates to the field of data processing, and specifically to a data updating method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of information technology, search engine technology is increasingly widely used in various business systems. Elasticsearch, as a distributed, RESTful search and analytics engine, has been widely applied in big data processing and retrieval due to its high performance, real-time capabilities, and scalability. In online synchronous service scenarios, real-time data updates and efficient querying have become key challenges in system design.

[0003] Currently, Elasticsearch primarily employs a document ID-based update mechanism for data updates. For example, by configuring the data source to automatically create ElasticSearch indexes and scheduling data update tasks according to a set update frequency, timely updates of index data are achieved. Alternatively, a wide table structure can be built based on join query fields, and Elasticsearch's document storage and indexing capabilities can be used to construct wide table documents, creating inverted indexes and sharding storage strategies for these documents. Summary of the Invention

[0004] The inventors discovered that existing technologies still have several issues with data updates in Elasticsearch. First, when performing an UPDATE operation, Elasticsearch requires specifying an ID to write data. This writing process necessitates a full index scan, regardless of the data size. The entire index to be updated must be read into memory for ID identification. In a cluster where read / write operations are not separated, this consumes memory and I / O resources typically used for normal business queries, especially in scenarios with high-frequency updates of large indexes, where performance overhead is particularly pronounced. Second, Elasticsearch's overwrite update operation, at its underlying implementation, marks each data entry and deletes these marked entries during periodic automatic segment merging. However, in Elasticsearch, deleting a large number of marked entries during segment merging increases system resource consumption and impacts overall system performance. Furthermore, existing Elasticsearch data update methods often lack effective mechanisms for managing data timeliness, failing to intelligently filter and update data based on its time attributes, resulting in low efficiency when processing time-sensitive business data.

[0005] To address at least one of the aforementioned technical problems, this invention proposes a data update method, apparatus, and computer device. In this method, a random document identifier is generated for the document containing the data, and the data is sorted and filtered according to the timestamp field. This avoids the performance overhead of full index scanning, reduces the occupation of traditional resources by deleting markers in the overwrite update operation, and initializes the first index after a first time interval, thereby improving the overall performance and data processing efficiency of the data system.

[0006] In one aspect, the present invention provides a data update method, the data update method comprising:

[0007] New data is written to the first index, and a random document identifier is generated for each document containing the new data, wherein the new data includes a timestamp field that records the time the data was written;

[0008] Query all data in the first index, group all data, sort the data in each group according to the timestamp field of all data, and filter the data in each group to obtain the filtered data;

[0009] At each first time interval, the first index is initialized.

[0010] Optionally, when writing new data to the first index, the existing data in the first index is retained (i.e., not deleted).

[0011] Optionally, in the first index, the random document identifier is automatically generated by the Elasticsearch system, and each document identifier is unique.

[0012] Optionally, all data in the first index are grouped according to a preset business primary key, and the data in each group is sorted in order from newest to oldest according to the timestamp field. The data that is first in the sorted order in each group is then extracted to obtain the filtered data.

[0013] Optionally, the preset business primary key is a combination of at least one core business field, and the preset business primary key has been pre-configured and stored in the system configuration file before new data is written.

[0014] Optionally, the initialization operation includes:

[0015] Create a second index, which is a completely new empty index;

[0016] Write the new data to the second index;

[0017] Add an index alias to the second index;

[0018] Delete the first index.

[0019] Optionally, a second index can be created based on a predefined Elasticsearch index template.

[0020] Optionally, deleting the first index includes: deleting the alias of the first index and physically deleting the first index.

[0021] In another aspect, the present invention provides a data updating apparatus, the data updating apparatus comprising:

[0022] The data writing unit writes new data to the first index and generates a random document identifier for each document containing the new data, wherein the new data includes a timestamp field that records the time the data was written.

[0023] The data filtering unit queries all data in the first index, groups all data, sorts all data according to the timestamp field of the data in each group, and filters the data in each group to obtain the filtered data.

[0024] An initialization unit performs an initialization operation on the first index at each first time interval.

[0025] To achieve the above objectives, according to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data update method.

[0026] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program / instructions are stored, which, when executed by a processor, implement the steps of the above-described data update method.

[0027] To achieve the above objectives, according to another aspect of the present invention, a computer program product is also provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the above-described data update method.

[0028] The beneficial effects of this invention are as follows: a random document identifier is generated for the document containing the data, and the data is grouped. The data in each group is sorted and filtered according to the timestamp field of the data. Furthermore, the first index is initialized after a first time interval. Thus, the validity and value of the data can be guaranteed while removing invalid and expired data, thereby improving the overall performance and data processing efficiency of the data system. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0030] Figure 1 This is a schematic diagram of a data update method according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of data filtering in an embodiment of this application;

[0032] Figure 3 This is a structural block diagram of the data update device according to an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the structure of the computer device for implementing the method of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0036] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0037] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0039] It should be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0040] It should be noted that the technical solution of this application provides users with corresponding operation entry points for users to choose to agree to or reject the automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0041] It should be noted that the data update method and apparatus of this invention can be used in the financial field, or in any field other than the financial field. This invention does not limit the application field of the data update method and apparatus.

[0042] First aspect of the embodiments

[0043] An embodiment of the first aspect of the present invention provides a data update method.

[0044] Figure 1 This is a schematic diagram of a data update method according to an embodiment of the present invention. Figure 1 As shown, in one embodiment of the present invention, the data update method of the present invention includes:

[0045] Step 101: Write new data to the first index and generate a random document identifier for each document containing the new data, wherein the new data includes a timestamp field that records the time the data was written;

[0046] Step 102: Query all data in the first index, group all data, sort the data in each group according to the timestamp field of all data, and filter the data in each group to obtain the filtered data;

[0047] Step 103: At each first time interval, initialize the first index.

[0048] As can be seen, through the synergistic effect of steps 101 to 103 above, the present invention generates random document identifiers for the documents containing the data, sorts and filters the data according to the timestamp field of the data, and initializes the first index after a first time interval. Thus, the performance overhead caused by full index scanning can be avoided, and the overall performance and data processing efficiency of the data system can be improved.

[0049] In the embodiments of this application, the first index is the target index for storing new data. The first index can store and associate data by establishing a mapping relationship between the values ​​of one or more key fields in the data table and the actual storage location (or primary key) of the data. The first index can be a B+ tree index, a hash index, a full-text index for text data, or a bitmap index suitable for low-radix numeric fields, etc. This application does not impose any restrictions.

[0050] In this embodiment, the data can be stored in the Elasticsearch system. The data in Elasticsearch exists in the form of a JSON format string. That is, the JSON format string is a container, and all fields of a data are encapsulated in the same JSON string. The relationship between fields is preserved through the key-value pair structure of JSON. Thus, Elasticsearch can quickly parse and query the data. The following describes this embodiment using the storage of data in the Elasticsearch system as an example.

[0051] In step 101, in some embodiments, when new data is written to the first index, existing data in the first index is retained (i.e., not deleted). That is, when new data is written to the first index, Elasticsearch uses "Insert" to write the data instead of "UPDATE" to update it. This avoids the step of deleting old data when using "UPDATE," eliminates the need to mark old data row by row, and avoids clearing existing historical data in the first index, allowing new and old data to temporarily coexist in the index.

[0052] According to the above embodiments, the resource consumption caused by data deletion operations can be avoided, and the efficiency of writing new data can be improved.

[0053] In some embodiments, in the first index, random document identifiers are automatically generated by the Elasticsearch system, and each document identifier (ID) is unique. Elasticsearch can automatically generate document identifiers based on its built-in algorithms, such as a random string generation mechanism, without requiring manual specification through business field combinations. For each document identifier, its ID is unique, meaning that no duplicate document identifiers will appear in the first index.

[0054] According to the above embodiments, no additional document identifier generation logic needs to be developed, which simplifies the technical implementation process and avoids errors that may occur when manually specifying IDs. In addition, making the document identifier in the first index unique can also prevent the data in the document from being accidentally overwritten.

[0055] In some embodiments, the new data written to the first index includes a timestamp field that records the time the data was written. The timestamp field in the new data records the time the new data was written to the first index, and this timestamp field is strongly bound to the data writing time. That is, the Elasticsearch system automatically encapsulates the current time as a field value and embeds it into the JSON structure of the new data the instant the data is written.

[0056] For example, the structure of the timestamp field can be YYYY-MM-DD HH-NN-XX, where YYYY represents the year, MM represents the month, DD represents the day, HH represents the hour, NN represents the minute, and XX represents the second. For example, the timestamp field can be 2024-01-01 00:00:00, which means that the new data corresponding to this timestamp field was written to the first index at 00:00:00 on January 1, 2024.

[0057] The above describes the writing of new data in the embodiments of this application. The following describes how to filter the data in the first index in the embodiments of this application.

[0058] In some embodiments, in step 102, all data in the first index are grouped according to a preset business primary key, the data in each group are sorted according to the timestamp field from newest to oldest, and the data that is first in the sorted order in each group is extracted to obtain the filtered data.

[0059] In the above embodiments, the preset business primary key can be a combination of at least one core business field, which has been pre-configured and stored in the system configuration file before new data is written. That is, the core business field identifies the data ownership or business attribute, and it directly corresponds to the smallest business unit that the user needs to obtain the latest data.

[0060] For example, if the business scenario is to display the latest status of a user's various types of orders, then the default business primary key is "user ID + order type". If the business scenario is to display the latest daily sales data of a store, then the default business primary key is "store ID + date". In other words, by using the default business primary key, business rules are used to define which data belong to the same group.

[0061] In the above embodiments, all data in the first index is grouped according to a preset business primary key. All data in the first index includes new data newly written to the first index according to the above embodiments, as well as other data originally contained in the first index. For the data in each group, the data can be sorted according to the timestamp fields from newest to oldest, and the first data point is extracted as the filtered data corresponding to the preset business component. That is, in each group of data, the newer the corresponding timestamp field, the earlier it is arranged in the order. Therefore, the latest data can be obtained in a business scenario; for example, the latest status of various types of orders for a user can be obtained, or the latest daily sales data for a store can be obtained.

[0062] Figure 2 This is a schematic diagram illustrating data filtering according to an embodiment of this application. It shows an example of filtering data in a first index according to the method described above.

[0063] like Figure 2As shown, the preset business primary key is a combination of field A and field B. In the first index, there are three preset business primary keys: those with the core business fields a1b1, a2b2, and a3b3. Each preset business primary key corresponds to a set of data. Taking the preset business primary key a1b1 as an example, this preset business primary key has two data entries with corresponding values ​​c0101 and c0201, each with a timestamp. These two data entries are arranged in ascending order of their timestamps, and the first data entry in this set is retrieved. That is, c0201 is used as the filtered data in the preset business primary key a1b1. Similarly, taking the preset business primary key a2b2 as an example, this preset business primary key has two data entries with corresponding values ​​c0102 and c0202, each with a timestamp. These two data entries are arranged in ascending order of their timestamps, and the first data entry in this set is retrieved. In other words, c0202 is used as the filtered data in the preset business primary key a2b2. Taking the preset business primary key a3b3 as an example, this preset business primary key has two data points, with corresponding data values ​​c0103 and c0203, and corresponding timestamps. These two data points are arranged in ascending order of their timestamps, and the first data point in this group is retrieved. That is, c0203 is used as the filtered data in the preset business primary key a2b3. Thus, data c0201, c0202, and c0203 are filtered as the data in the first index.

[0064] In the above embodiments, the present application was described using the selection of the latest data as an example. However, the present application is not limited to this. Other data besides the latest data can also be deleted to achieve the purpose of filtering data.

[0065] According to the above embodiments, it can be ensured that the latest data records under each business primary key combination are obtained, and invalid and expired data are removed, thus ensuring the validity and value of the data and avoiding data duplication and redundancy.

[0066] The above describes how data is filtered in the embodiments of this application. Considering that after writing and filtering data according to the above embodiments, there may be a problem of underlying data bloat after long-term data writing, in order to address this problem, in some possible implementations, index initialization operations can be performed to clean up the data. The index initialization is described below.

[0067] In some embodiments, the initialization operation of the first index includes:

[0068] Create a second index, which is a completely new empty index.

[0069] Write the new data to the second index;

[0070] Add an index alias to the second index;

[0071] Delete the first index.

[0072] In the above embodiments, for the second index, a completely new empty index can be created according to a preset Elasticsearch index template. For example, the first index is initialized starting in October 2025. When writing data for October 2025, a second index named "A_202510" is created. This second index has no data. The second index can have the same storage structure as the first index, thereby ensuring compatibility of the data writing format.

[0073] In some examples, the writing logic for new data into the second index remains consistent with that of the first index. That is, data is written to the second index by generating random document identifiers, and existing data in the second index is not deleted during the writing process. After writing new data to the second index, an index alias is added or bound to it. The system can access the data in the second index solely through the alias, without needing to access the physical index name. Therefore, no configuration modifications are required; the new second index can be accessed simply by switching the index associated with the alias, avoiding service termination.

[0074] In some embodiments, deleting the first index includes: deleting the alias of the first index and physically deleting the first index. That is, while deleting the alias of the first index, all physical files corresponding to the first index are completely removed from the Elasticsearch storage system, rather than just marked as "to be deleted". This ensures that the first index can no longer be accessed through the alias, while also releasing the disk, memory and other hardware resources occupied by the first index.

[0075] In the above embodiment, the first index can be initialized at regular intervals, that is, the initialization operation is performed periodically. This first interval can be set according to specific needs. For example, when the first interval is one month, the first index is initialized monthly. The first index name format is defined as A___YYYYMM. When data for October 1, 2025 is written, Elasticsearch will create a second index A___202510, write the data into the second index, add an alias B for the second index for user access, and remove the old index (A___202509) initialized last month. The second index will be initialized again at the one-month interval. In this way, the most efficient data cleaning can be achieved.

[0076] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0077] Second aspect of the embodiments

[0078] Based on the same inventive concept, embodiments of the present invention also provide a data updating apparatus, which can be used to implement the data updating method described in the embodiments of the first aspect above, as described in the following embodiments. Since the principle by which this data updating apparatus solves the problem is similar to the data updating method of the embodiments of the first aspect above, embodiments of this data updating apparatus can be found in the embodiments of the data updating method of the embodiments of the first aspect above, and repeated details will not be described again. The terms "unit" or "module" used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0079] Figure 3 This is a structural block diagram of the data update device according to an embodiment of the present invention, such as... Figure 3 As shown, in one embodiment of the present invention, the data update device 300 of the present invention includes:

[0080] The data writing unit 301 writes new data to the first index and generates a random document identifier for each document containing the new data. The new data includes a timestamp field recording the time the data was written.

[0081] The data filtering unit 302 queries all data in the first index, groups all data, sorts the data in each group according to the timestamp field of all data, and filters the data in each group to obtain the filtered data.

[0082] The initialization unit 303 performs an initialization operation on the first index at each first time interval.

[0083] Third aspect of the embodiments

[0084] To achieve the above objectives, according to another aspect of this application, a computer device is also provided. For example... Figure 4 As shown, the computer device 400 includes a memory 401, a processor 402, a communication interface 403, and a communication bus 404. The memory 401 stores a computer program that can run on the processor 402. When the processor 402 executes the computer program, it implements the steps in the method of the embodiment of the first aspect described above.

[0085] Processor 402 can be a central processing unit (CPU). Processor 402 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0086] The memory 401, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and units, such as the program units corresponding to those in the above method embodiments of the present invention. The processor 402 executes various functional applications and data processing of the processor 402 by running the non-transitory software programs, instructions, and modules stored in the memory 401, thereby implementing the methods in the above method embodiments.

[0087] Memory 401 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, memory 401 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 401 may optionally include memory remotely located relative to processor 402, and these remote memories may be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0088] The one or more units are stored in the memory 401 and, when executed by the processor 402, perform the methods described in the above embodiments.

[0089] The specific details of the aforementioned computer device 400 can be understood by referring to the relevant descriptions and effects in the above embodiments, and will not be repeated here.

[0090] Fourth aspect of the embodiment

[0091] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed in a computer processor, implements the steps in the above-described data update method. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0092] Fifth aspect of the embodiment

[0093] To achieve the above objectives, according to another aspect of this application, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the steps of the data update method of the embodiment of the first aspect described above.

[0094] Obviously, those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data update method, characterized in that, The data update method includes: New data is written to the first index, and a random document identifier is generated for each document containing the new data, wherein the new data includes a timestamp field that records the time the data was written; Query all data in the first index, group all data, sort the data in each group according to the timestamp field of all data, and filter the data in each group to obtain the filtered data; At each first time interval, the first index is initialized.

2. The data update method according to claim 1, characterized in that, When writing new data to the first index, the existing data in the first index is retained.

3. The data update method according to claim 1, characterized in that, In the first index, the random document identifier is automatically generated by the Elasticsearch system, and each document identifier is unique.

4. The data update method according to claim 1, characterized in that, The query process involves querying all data in the first index, grouping all data, sorting the data in each group based on the timestamp field, and filtering the data in each group to obtain the filtered data, including: All data in the first index are grouped according to the preset business primary key. The data in each group is sorted in order from newest to oldest according to the timestamp field. The data that is first in the sorted order in each group is extracted to obtain the filtered data.

5. The data update method according to claim 4, characterized in that, The preset business primary key is a combination of at least one core business field. The preset business primary key has been pre-configured and stored in the system configuration file before new data is written.

6. The data update method according to claim 1, characterized in that, The initialization operation includes: Create a second index, which is a completely new empty index; Write the new data to the second index; Add an index alias to the second index; Delete the first index.

7. The data update method according to claim 6, characterized in that, The creation of the second index includes: The second index is created based on a pre-defined Elasticsearch index template.

8. The data update method according to claim 6, characterized in that, Deleting the first index includes: Delete the alias of the first index and physically delete the first index.

9. A data update device, characterized in that, The data update device includes: The data writing unit writes new data to the first index and generates a random document identifier for each document containing the new data, wherein the new data includes a timestamp field that records the time the data was written. The data filtering unit queries all data in the first index, groups all data, sorts the data in each group according to the timestamp field of all data, and filters the data in each group to obtain the filtered data. An initialization unit performs an initialization operation on the first index at each first time interval.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 8.

Images