Data processing method, device, equipment, and storage medium

Abstract

An embodiment of the present application discloses a data processing method, apparatus, device, and storage medium; wherein the method includes: determining a first version identifier of a database corresponding to when reading data of a first page; querying a version snapshot corresponding to the first version identifier; wherein the version snapshot is used to record the data arrangement of the database corresponding to when reading data of the first page; and reading data of page k from the database according to the data identifier recorded in the version snapshot; wherein k is greater than 1.

Description

Technical Field

[0001] The embodiments of the present application relate to electronic technology, and relate to but are not limited to data processing methods and devices, equipment, and storage media. Background Art

[0002] Nowadays, users mostly browse the information they want through web pages. Web pages generally display data in pages. However, after users obtain the first n pages of data, when they obtain the n+1th page of data, it is often found that the data on the n+1th page is repeated with the data on the nth page. Summary of the Invention

[0003] In view of this, the data processing method, apparatus, device, and storage medium provided in the embodiments of the present application can avoid duplicate data on pages and ensure data consistency. The data processing method, apparatus, device, and storage medium provided in the embodiments of the present application are implemented as follows:

[0004] The data processing method provided by an embodiment of the present application includes: determining a first version identifier of a database corresponding to when reading data of a first page; querying a version snapshot corresponding to the first version identifier; wherein the version snapshot is used to record the data arrangement of the database corresponding to when reading data of the first page; and reading data of the kth page from the database according to the data identifier recorded in the version snapshot; wherein k is greater than 1.

[0005] The data processing method provided in an embodiment of the present application includes: determining the data arrangement of a database; generating a version snapshot and a version identifier corresponding to the database based on the data arrangement; and associating and storing the version snapshot with the version identifier of the corresponding database.

[0006] The data processing device provided in an embodiment of the present application includes: a determination module, used to determine a first version identifier of a database corresponding to when reading data of a first page; a query module, used to query a version snapshot corresponding to the first version identifier; wherein the version snapshot is used to record the data arrangement of the database corresponding to when reading the first page of data; and a reading module, used to read data of the kth page from the database according to the data identifier recorded in the version snapshot; wherein k is greater than 1.

[0007] The data processing device provided in an embodiment of the present application includes: a determination module for determining the data arrangement of a database; a generation module for generating a version snapshot and a version identifier corresponding to the database based on the data arrangement; and a storage module for storing the version snapshot in association with the version identifier of the corresponding database.

[0008] An electronic device provided in an embodiment of the present application includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0009] The computer-readable storage medium provided in the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0010] In an embodiment of the present application, after determining the version identifier of the database corresponding to when reading the first page of data, the data of page k is continued to be read from the database based on the data identifier recorded in the version snapshot corresponding to the version identifier. In this way, it can be ensured that when reading the kth page of data, it is read based on the same version snapshot as when reading the first page of data. Since the data arrangement in the version snapshot is orderly and unchanged, even if the data in the database changes during the data reading process, it will not affect the data reading, thereby avoiding duplicate data on the page, making the data read a complete set of data, and ensuring data consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0012] Figure 1 This is a schematic diagram of the implementation flow of the data processing method according to an embodiment of the present application;

[0013] Figure 2 This is a schematic diagram of the implementation flow of the data processing method according to an embodiment of the present application;

[0014] Figure 3 This is a schematic diagram of the implementation flow of the data processing method according to an embodiment of the present application;

[0015] Figure 4 This is a schematic diagram of the implementation flow of the data processing method according to an embodiment of the present application;

[0016] Figure 5 This is a schematic diagram of the implementation flow of the data processing method according to an embodiment of the present application;

[0017] Figure 6A This is a diagram of the structure of the table in the database;

[0018] Figure 6B This is the display effect diagram when there is no data update in the database;

[0019] Figure 6C If data is inserted, the database and user interface will be displayed after the data is inserted;

[0020] Figure 7 This is a snapshot version of the display effect diagram used in the embodiment of the present application;

[0021] Figure 8 This is a structural diagram of a data processing device according to an embodiment of the present application;

[0022] Figure 9 This is a structural diagram of a data processing device according to an embodiment of the present application;

[0023] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0026] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are used to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0028] An embodiment of the present application provides a data processing method, which is applied to a client. During implementation, the electronic device hosting the client can be any type of device with information processing capabilities, such as a mobile phone, tablet computer, computer, personal digital assistant, navigation system, digital phone, video phone, television, or sensor device. The functions implemented by this method can be implemented by a processor in the electronic device calling program code. Of course, the program code can be stored in a computer storage medium. Therefore, the electronic device includes at least a processor and a storage medium.

[0029] The present application provides a data processing method. Figure 1 This is a schematic diagram of the implementation flow of the data processing method of the embodiment of the present application, as shown in FIG. Figure 1 As shown, the method may include the following steps 101 to 103:

[0030] Step 101: The client determines the first version identifier of the database corresponding to the first page of data to be read.

[0031] It is understandable that after the client finishes reading the first page of data, when it continues to read the kth page of data, the database version corresponding to the first page of data may be the historical database version or the database version corresponding to the current display interface.

[0032] This is because: if the client reads the first page of data and then continues to read the kth page of data, the database changes (for example, the data in the database responds to operations such as addition, deletion, or movement), then the database version corresponding to the current display interface is the changed database version. Accordingly, the database version corresponding to when the client reads the first page of data becomes the historical database version; if the data in the database does not change after the client reads the first page of data, then when it continues to read the kth page of data, the database version corresponding to the previous reading of the first page of data is still the database version corresponding to the current display interface.

[0033] Every time a database is changed, the changed database has its corresponding version identifier; of course, the initial database also has its corresponding version identifier, and different databases correspond to different version identifiers.

[0034] When the client wants to read the kth page of data later, it is not clear whether the database version corresponding to the first page of data it read is the historical database version or the database version corresponding to the current display interface. Therefore, it is necessary to first determine the first version identifier of the database corresponding to the first page of data read by the client.

[0035] In some embodiments, the version identifier of the database may be the version number of the database, and each time a change occurs to the database, the corresponding version number is incremented by 1.

[0036] Step 102: The client queries the version snapshot corresponding to the first version identifier; wherein the version snapshot is used to record the data arrangement of the corresponding database when the first page of data is read.

[0037] A snapshot is a fully usable copy of a specified data set, representing the data as it appears at a specific point in time (when the copy begins). In other words, a version snapshot of a database represents the data arrangement of that database at a specific moment in time. The data arrangement recorded in a version snapshot is fixed, so changes to the database at a later point in time, resulting in a version snapshot, have no impact on the version snapshot of the database at the previous point in time.

[0038] The so-called data arrangement refers to the orderly arrangement of data in the database. In some embodiments, a cache zset structure is used to store the identifiers of data in the database. A zset is an ordered set that does not allow duplicate members and can sort data based on data identifiers, while occupying a small amount of storage space.

[0039] Step 103 : The client reads the data of page k from the database according to the data identifier of the version snapshot record; where k is greater than 1.

[0040] In some embodiments, after reading the data of page k from the database, the client further refreshes the current display interface according to the data of page k.

[0041] It is understandable that the version snapshot records the data arrangement of the corresponding database, and each data has its corresponding data identifier. When the database changes, the data identifier corresponding to each data is fixed. In some embodiments, the data identifier can be the data ID identifier.

[0042] In an embodiment of the present application, after determining the version identifier of the database corresponding to when reading the first page of data, the data of page k is continued to be read from the database based on the data identifier recorded in the version snapshot corresponding to the version identifier. In this way, it can be ensured that when reading the kth page of data, it is read based on the same version snapshot as when reading the first page of data. Since the data identifier recorded in the version snapshot is fixed, even if the data in the database changes after the client has read the first page of data, it will not affect the subsequent reading of the kth page of data, thereby avoiding duplicate data in the page, making the read data a complete set of data, and ensuring data consistency.

[0043] This embodiment of the application further provides a data processing method, Figure 2 A schematic diagram of the implementation flow of a data processing method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method may include the following steps 201 to 207:

[0044] In step 201 , the client queries whether the request list contains the first version identifier of the database corresponding to the first page of data to be read. If not, step 202 is executed; if yes, step 203 is executed.

[0045] In the embodiment of the present application, different users may initiate requests at different times, and the same user may also initiate multiple requests at different times.

[0046] For example, user 1 initiates the first request at time 1 and obtains the first page of data. This request corresponds to database version 1, and this version 1 is recorded in the request list. Subsequently, user 1 initiates the second request at time 2. The database version 1 corresponding to the first request at time 1 can be found in the request list, and then step 203 is executed. In this way, when user 1 requests to obtain the kth page of data at time 2, it is obtained based on the same version snapshot corresponding to the first page of data obtained at time 1, thus avoiding duplicate data on the page.

[0047] For example, user 2 initiates the first request at time 3, but has not initiated any requests before time 3. Therefore, the request list does not contain the first version identifier of the database corresponding to when user 2 obtained the first page of data. In this case, step 202 is executed, that is, version identifier 2 corresponding to the current database is used as the database version identifier corresponding to when user 2 obtained the first page of data; then, based on the version snapshot of the current database corresponding to version identifier 2, the kth page of data is obtained. In this way, when user 2 obtains the kth page of data, it is also based on the same version snapshot as when obtaining the first page of data (that is, the version snapshot corresponding to the current database with version identifier 2), thereby avoiding duplicate data on the page.

[0048] Step 202: The client obtains a second version identifier of the current database, and uses the second version identifier as the first version identifier.

[0049] In some embodiments, when the second version identifier of the current database is used as the first version identifier, the request list is refreshed, the invalid version identifier is deleted, and the second version identifier is added.

[0050] In step 203 , the client determines whether the version snapshot corresponding to the first version identifier is invalid; if invalid, execute step 204 ; if not invalid, execute step 205 .

[0051] It is understandable that the cache of the version snapshot is time-sensitive. To determine whether the version snapshot is invalid, it is necessary to determine whether the validity period of the snapshot identifier corresponding to the version identifier has been reached.

[0052] For example, when user 1 initiates a second request at time 2, the version snapshot corresponding to the first request initiated at time 1 may be valid or invalid. If the version snapshot is invalid, then in this case, user 1 cannot continue to obtain the kth page data based on the version snapshot at time 2, and step 204 needs to be executed. If the version snapshot is valid, step 205 is continued, that is, the version number corresponding to the version snapshot is used as the first version identifier to continue to obtain the kth page data. In this way, when user 1 obtains the kth page data at time 2, it is also based on the same version snapshot as when obtaining the first page data (that is, the version snapshot corresponding to the historical database with version identifier 1), thereby avoiding duplicate data on the page.

[0053] Step 204: The client obtains the second version identifier of the current database, and uses the second version identifier as the first version identifier.

[0054] In some embodiments, when the second version identifier of the current database is used as the first version identifier, the request list is refreshed, the invalid version identifier is deleted, and the second version identifier is added.

[0055] Step 205: The client determines that the version identifier is the first version identifier;

[0056] Step 206: The client queries the version snapshot corresponding to the first version identifier; wherein the version snapshot is used to record the data arrangement of the corresponding database when the first page of data is read;

[0057] Step 207 : The client reads the data of page k from the database according to the data identifier of the version snapshot record; where k is greater than 1.

[0058] In an embodiment of the present application, by querying whether the request list contains the first version identifier of the database corresponding to the time when the first page of data is read, it is determined whether the database version identifier corresponding to the time when the user obtains the first page of data is the version identifier of the current database or the version identifier of the historical database, so that when the user subsequently obtains the kth page of data, it is based on the same version snapshot as when the first page of data was obtained, thereby ensuring data consistency and avoiding duplicate data on the page.

[0059] The data processing methods provided in the following embodiments are applied to the server side. The electronic device acting as the server can be any type of device with information processing capabilities, such as a mobile terminal (e.g., a mobile phone, tablet computer, laptop computer), a personal computer, a single server, or a cluster of servers. The functions implemented by the method can be implemented by a processor in the electronic device calling program code. Of course, the program code can be stored in a computer storage medium. Therefore, the electronic device includes at least a processor and a storage medium.

[0060] Figure 3 A schematic diagram of the implementation flow of a data processing method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method may include the following steps 301 to 303:

[0061] Step 301: The server determines the data arrangement of the database.

[0062] In some embodiments, the configuration backend uses a cache zset structure to store data identifiers in the database, sorts the data in the database, and thereby determines the data arrangement of the database.

[0063] Step 302: The server generates a version snapshot and a version identifier corresponding to the database based on the data arrangement.

[0064] In an embodiment of the present application, after the data in the database is arranged in order, the configuration background generates a version snapshot and version identifier corresponding to the database based on the arrangement of the ordered data. In this way, the data identifier recorded in the generated version snapshot corresponding to the database is ordered and unchanged.

[0065] Step 303: The server associates and stores the version snapshot with the version identifier of the corresponding database.

[0066] In an embodiment of the present application, based on the arrangement of data in the database, a version snapshot and version identifier corresponding to the database are generated and stored in association, so that the client can read data from the database based on the version snapshot corresponding to the version identifier, thereby avoiding data duplication on the page.

[0067] This embodiment of the application further provides a data processing method, Figure 4 A schematic diagram of the implementation flow of a data processing method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method may include the following steps 401 to 406:

[0068] Step 401: The server determines the data arrangement of the database.

[0069] In some embodiments, determining the data arrangement of the database includes: when detecting that the database is updated, determining the data arrangement of the updated database.

[0070] In other embodiments, determining the data arrangement of the database includes: determining the data arrangement of the initial database.

[0071] It can be understood that the arrangement of the database includes not only the data arrangement of the updated database, but also the data arrangement of the initial database. Therefore, when determining the data arrangement of the database, it is necessary to determine the data arrangement of the database in both states.

[0072] Step 402: The server generates a version snapshot and a version identifier corresponding to the database based on the data arrangement.

[0073] Step 403: The server generates a snapshot identifier for the version snapshot; associates the snapshot identifier of the version snapshot with the version identifier of the corresponding database and stores them in a request list; and sets a validity period for the version snapshot corresponding to the snapshot identifier in the request list.

[0074] In some embodiments, the snapshot identifier of the version snapshot may be the name of the version snapshot, and the name of the version snapshot and the corresponding version number of the database are stored in the request list.

[0075] To reduce storage space, the validity period of the version snapshot corresponding to the snapshot ID in the request list is set. That is, the version snapshot has a storage time limit. For example, the validity period can be set to n hours. The validity period can be set based on the user's real-time requirements or the user's business needs, and there is no limitation here.

[0076] Step 404: The server queries the request list to determine whether the version snapshot corresponding to each snapshot identifier is within the corresponding validity period; and deletes the invalid version snapshot and the corresponding entry in the request list.

[0077] In some embodiments, the entry includes a snapshot identifier corresponding to the version snapshot and a version identifier of the database.

[0078] In an embodiment of the present application, when a version snapshot expires, the expired version snapshot is deleted; and the corresponding snapshot identifier and database version identifier are deleted from the request list. In this way, on the one hand, the storage space of the request list can be reduced, saving resources; on the other hand, by querying whether the version snapshot corresponding to the snapshot identifier stored in the request list is expired and deleting the expired version snapshot, the client can determine the first version identifier of the database corresponding to the first page of data without first determining whether the version snapshot is expired. Instead, the client can directly determine the version snapshot corresponding to the first version identifier included in the request list, that is, the first version identifier of the database corresponding to the first page of data when the client determines to read the first page of data. This is because: the expired version snapshot has been deleted. If the request list contains the first version number, then the version snapshot corresponding to the first version number must be valid. In this way, the data acquisition speed can be effectively improved and the real-time display of data can be guaranteed.

[0079] Step 405: The server responds to the data deletion operation and determines the data identifier of the target data to be deleted according to the data deletion operation;

[0080] Step 406: When the data identifier of the target data is not recorded in each valid version snapshot, the server deletes the target data from the database.

[0081] It is understandable that when the server responds to a delete operation on data in the database, the data cannot be deleted from the database immediately. This is because: the data identifier of the target data to be deleted may exist in the historical version snapshot. If the historical version snapshot containing the data identifier of the target data is valid at this time, then it is necessary to ensure that the user can still read the target data based on the historical version snapshot. Therefore, although the configuration background responds to the delete operation on the target data in the database, the target data will not be deleted directly from the database. Only when all version snapshots containing the data identifier of the target data are invalid, that is, when the data identifier of the target data cannot be found in any valid version snapshot, will the target data be deleted from the database.

[0082] In some embodiments, to determine whether the target data's data identifier is recorded in each valid version snapshot, a cache set structure can be used to store the valid version snapshots and the data identifiers of the current database. The cache set structure ensures that the stored data identifiers are unique. A set difference operation is then performed between each set containing the data identifiers of the valid version snapshots and the set containing the identifiers of all data in the current database to determine whether the target data's data identifier exists in the valid version snapshots. The cache set structure is similar to an array, but the values of its members are unique, with no duplicate values.

[0083] User pages retrieve data in pages. After retrieving the first n pages of data, the first n data are not updated when retrieving page n+1. If a new data entry is inserted into the first n pages before retrieving page n+1, the data entry originally on page n will be moved to page n+1. When retrieving page n+1, a data entry on page n+1 may duplicate data on page n.

[0084] In related technologies, the following methods are often used to solve the problem of duplicate data on pages:

[0085] (a) Using cache: Put the first n pages of data into the cache. When the first n pages of data are requested, directly read the cache to obtain the data;

[0086] (b) Use ID identification: The front-end request contains the ID of the last record requested, and subsequent requests obtain data from this ID;

[0087] (c) Client-side deduplication: Each request is compared with the currently acquired data to remove duplicate data. If there are too many duplicates in the current data, consider requesting again.

[0088] However, the above method has the following problems:

[0089] (a) Using cache: The original intention of the design is to ensure that the first n pages of data are not repeated. However, when the amount of data is large and is not within the range of the first n pages, it is still not possible to ensure that the data is not repeated.

[0090] (b) Using IDs: This solution has limitations. If the displayed data is sorted by ID, it can effectively solve the problem of duplicate data. However, if the data is sorted by other attributes, this solution will no longer meet the requirements.

[0091] (c) Client-side deduplication: Multiple requests may occur, and the number of useful records obtained by the request may not be equal to the number of records in the requested page size. For example, if the requested page size is 10 records and the available data is greater than 10, and there are two duplicate data in one request, only 8 data will be obtained in one request.

[0092] Based on this, the following describes an exemplary application of the embodiment of the present application in a practical application scenario.

[0093] In an embodiment of the present application, a latest version snapshot (recording all data ID information) is generated for each data change. When the user obtains the first page of data, the data is read through the latest version snapshot, and the n+1 (n>0 positive integer) page data is obtained through the version snapshot corresponding to the first page. In this way, the consistency of all requested page data is guaranteed, and data duplication problems can be avoided.

[0094] Compared with the methods in the related art, the improvements in the embodiments of the present application are as follows (1) to (3):

[0095] (1) Putting all data ID information into the sorting cache can ensure that all records are stored in a smaller cache and that all data is obtained without duplication;

[0096] (2) Information data is read from the cache, and the cached data is already sorted, and can be displayed in a specific order according to any attribute requirements;

[0097] (3) Since the data stored in the cache is deduplicated, there is no duplication of the read data and the original data, thus avoiding the problem of multiple requests and inconsistent requested data amounts.

[0098] In an embodiment of the present application, version control is used to ensure that user requested data is not repeated. A corresponding version will be generated each time the management end performs an operation (add, delete, move). When the user completes the first query, there will be corresponding version number information, and subsequent queries are based on this version number.

[0099] Figure 5 This is a flow chart of a data processing method provided in an embodiment of the present application. Figure 5 As shown, the data processing method is analyzed from both the configuration background logic and the client request logic.

[0100] First, let's explain the terms involved: D: database, recording information data; ZS: latest configuration record ID cache; S: record ID cache, recording record IDs in D; SV: snapshot version cache; SL: snapshot list cache; V: snapshot version number, the version number is initially 0 and increments.

[0101] (a) Analyze the configuration backend logic and perform steps 1 to 13 as follows:

[0102] Step 1: Determine whether to perform an add operation on the database; if yes, go to step 2; otherwise, go to step 11;

[0103] Step 2, add the record into database D;

[0104] Step 3: Put the record ID into cache ZS and S;

[0105] Step 4: Determine whether the snapshot cache list SL exists; if yes, go to step 5; otherwise, go to step 6;

[0106] Step 5: Set the maximum version number snapshot cache validity period to n hours; then proceed to step 6;

[0107] Step 6, generate version number V;

[0108] Step 7: Generate a record snapshot cache SV based on the cache ZS;

[0109] Step 8: Put the generated SV into the snapshot list cache SL;

[0110] Step 9: traverse the snapshot list cache SL and remove the invalid SV;

[0111] Step 10: Calculate the deleted records based on SV and S, and delete the redundant records in D and S;

[0112] Step 11, determine whether to perform a delete operation on the database; if yes, execute step 12, otherwise, execute step 13;

[0113] Step 12: Remove the record from the record ID cache ZS; then execute step 5;

[0114] Step 13: Update the record ID cache ZS; then execute step 5.

[0115] (b) Analyze the client request logic side and perform steps 1 to 6 as follows:

[0116] Step 1: Get the request list;

[0117] Step 2: Determine whether there is a version number SV corresponding to the user request in the request list; if so, go to step 3; otherwise, go to step 4;

[0118] Step 3: Determine whether the version number SV is invalid; if invalid, go to step 4; otherwise, go to step 5;

[0119] Step 4: Return null and cache invalidation flag;

[0120] Step 5: Obtain a record list according to the version number SV, and record the current version number SV in the list;

[0121] Step 6: Obtain a record list according to the latest version number SV, and record the latest version number SV in the list.

[0122] It should be noted that on the configuration backend, each time a database change occurs, a corresponding version snapshot SV is generated based on the version number V. V is part of the SV. Therefore, when a snapshot version SV is generated, a corresponding version number V exists for users to query.

[0123] Figures 6A to 6C Here is an example of displaying duplicate data to illustrate the effect of not using the optimization solution:

[0124] Figure 6AThe structure of the table in the database is given. Figure 6A It can be seen that the structure of the table includes: record id, name and sort item. Figure 6A The database includes 8 data, each of which corresponds to id 1, 2, 3, 4, 5, 6, 7 and 8; each id corresponds to the name record 1, record 2, record 3, record 4, record 5, record 6, record 7 and record 8; each id corresponds to the sorting items a, b, c, d, e, f, g and h, among which the sorting items are displayed to the front end in the lexicographical order of the values.

[0125] Figure 6B This is the display effect diagram when there is no data update in the database. Figure 6B As can be seen, when there are no data updates in the original database, the first page of data displayed in the user interface includes data with IDs 1, 2, 3, and 4, corresponding to sort items a, b, c, and d; the second page of data displayed in the user interface includes data with IDs 5, 6, 7, and 8, corresponding to sort items e, f, g, and h. Clearly, when there are no data updates in the database, the first and second pages of data displayed in the user interface do not overlap.

[0126] Figure 6C This is a diagram showing the display effects of the database and user interface after data is inserted.

[0127] in, Figure 6C The values ①, ②, and ③ in the table represent the execution order; Figure 6C Figure a is the original database, Figure b is the user interface for obtaining only the first page of data, Figure c is the changed database, and Figures d and e are the user interfaces for displaying the first and second pages of data.

[0128] Given below Figure 6C The execution steps are explained in .

[0129] Execute step ①: The user interface in Figure b only obtains the first page of data, which includes data with ids 1, 2, 3, and 4, and corresponding sort items a, b, c, and d;

[0130] Execute step ②: As shown in Figure c, insert a new data after the data with ID 8 in the original database. The new data ID is 9, the name is "Newly Added 1", and the sort item is aa, generating a changed database.

[0131] When executing step ③, the following two situations may occur: (1) Since the user interface has already obtained the first page of data based on the original database in Figure b, and the database has been updated in step ②, the user interface only obtains the updated second page of data. As shown in Figure e, the updated second page of data obtained includes data with ids of 4, 5, 6, and 7, and corresponding sorting items d, e, f, and g. It can be seen that the data on the second page is repeated with the data on the first page (i.e., sorting item d). The reason is that when the second page of data is obtained, the data in the database has changed, and the first page of data that has been obtained will not be automatically updated with the second page of data, resulting in the same data appearing on the second page and the first page.

[0132] (2) As shown in Figure d, after the database is updated, the user interface obtains the first and second page data based on the changed database. At this time, the first page data of the user interface includes data with ids of 1, 9, 2, and 3, and the corresponding sorting items are a, aa, b, and c; the second page data of the user interface includes data with ids of 4, 5, 6, and 7, and the corresponding sorting items are d, e, f, and g. There is no duplication of data on the first and second pages.

[0133] Figure 7 Here is an example using a snapshot version to illustrate the effect of the optimization solution:

[0134] like Figure 7 As shown, Figure 7 The values ①, ②, ③, ④, and ⑤ represent the execution order; icon a represents reading data from the database through the ID; icon b represents reading the ID through the snapshot; icon c represents automatically generating a version snapshot of the database table.

[0135] Given below Figure 7 The execution steps are explained in .

[0136] exist Figure 7 There are two situations in which data can be obtained:

[0137] (1) When users are acquiring data, the data in the database is updated: User 1 initiates request 1 based on the original database to acquire only the first page of data. Then the original database is updated. After the database is updated, User 1 initiates request 2 to acquire only the second page of data.

[0138] (2) When the user is acquiring data, the data in the database is not updated: User 2 initiates request 3 based on updating the database to acquire the first page of data and the second page of data.

[0139] Execute step 1: Generate corresponding snapshot 1 based on the original database. In snapshot 1, use the cache zset structure to store the ID list 1, 2, 3, 4, 5, 6, 7, and 8 of the original database;

[0140] Execute step 2: Get the display ID from snapshot 1. The user interface uses the ID to retrieve only the first page of data. The first page of data includes data with IDs 1, 2, 3, and 4, and the corresponding sort items are a, b, c, and d.

[0141] Execute step 3: Insert a new data after the data with ID 8 in the original database. The new data ID is 9, the name is "Newly Added 1", and the sort item is aa. This generates a changed database.

[0142] Execute step 4: Generate corresponding snapshot 2 based on the changed database. In snapshot 2, use the cache zset structure to store the ID list 1, 9, 2, 3, 4, 5, 6, and 7 of the changed database.

[0143] When executing step ⑤, the following two situations may occur:

[0144] The first case: before updating the database, user 1 initiates request 1, and the user interface only obtains the first page of data; after the database is updated, user 1 initiates request 2 again, and only obtains the second page of data; the second case: after the database is updated, user 2 initiates request 3, and obtains the first two pages of data from the change database.

[0145] The first scenario is explained in detail as follows: Before the database is updated, User Interface 1 has already retrieved the first page of data from the original database based on the IDs of Snapshot 1. This data includes data with IDs 1, 2, 3, and 4, and corresponding sort items a, b, c, and d. Now, after the database is updated, User Interface 1 retrieves only the second page of data. When retrieving this data, User Interface 1 retrieves it from the database based on the list of IDs stored in Snapshot 1. This data includes data with IDs 5, 6, 7, and 8, and corresponding sort items e, f, g, and h. This shows that even though the database has changed after User Interface 1 retrieved the first page of data, the second page of data retrieved by User Interface 1 is based on Snapshot 1, which corresponds to the original database, and therefore no duplicate data will occur.

[0146] The second scenario is explained in detail as follows: In step 4, Snapshot 2 is generated based on the change database. User Interface 2 directly retrieves the first two pages of data from the change database based on the ID list stored in Snapshot 2. The first page of data retrieved by User Interface 2 includes data with IDs 1, 9, 2, and 3, corresponding to sort items a, aa, b, and c. The second page of data retrieved includes data with IDs 4, 5, 6, and 7, corresponding to sort items d, e, f, and g. Clearly, when User Interface 2 retrieves the first two pages of data from the change database based on Snapshot 2, no duplicate data will occur.

[0147] pass Figure 7It can be seen that no matter how the data in the database changes, a corresponding version snapshot will be generated. The user interface only needs to use the corresponding version number in combination with the database to ensure data consistency and avoid duplicate data.

[0148] The following are the implementation details of the optimization scheme:

[0149] 1. Data is stored in the database (corresponding to Figure 5 Chinese: D);

[0150] 2. Use cache zset structure to store sorted data (corresponding to Figure 5 Chinese: ZS), set stores the current data record id of the database (corresponding to Figure 5 Chinese: S);

[0151] 3. Each operation (including adding, deleting and moving) generates a snapshot of the latest version based on the data in the zset structure (structure zset, corresponding to Figure 5 In: S_V), set the expiration time of the previous snapshot to n hours;

[0152] 4. Put the snapshot name into the snapshot list (list structure, corresponding process Figure 5 Middle: S_L);

[0153] 5. Traverse the snapshot list (S_L), remove past snapshots, and calculate the deleted record based on the current snapshot list and database record cache (S), and remove the record from the cache (S) and database (D);

[0154] 6. The client obtains the corresponding data information according to the corresponding version number.

[0155] For example, when a user initiates the first request, the latest version number is obtained. So when the user requests again 2 minutes later, the user still uses the version number corresponding to the previous 2 minutes. Therefore, the data requested by the user at this time is the version snapshot corresponding to the request 2 minutes ago, rather than the version snapshot corresponding to the latest version number generated by the database after the new data is added. If the user obtains the first page of data in the first request, then when he requests again 2 minutes later, the data of other pages will be obtained based on the version snapshot corresponding to the first page of data (so even if data is inserted in the newly generated version snapshot, it will not affect the version snapshot generated 2 minutes ago).

[0156] In the embodiments of this application, data snapshots (i.e., version snapshots) and version numbers (i.e., database version identifiers) are used to ensure data consistency and avoid information waterfall duplication. A data snapshot is a list of data IDs corresponding to each data update; a version number is a new version number generated for each data update.

[0157] In the embodiment of the present application, the orderliness, deduplication, and timed expiration characteristics of the cache are utilized to generate a corresponding data snapshot for each updated database; by reading a specific data snapshot, duplicate data is avoided. This can achieve the following effects:

[0158] (1) Save space, support sorting by any attribute, and read data quickly: Based on the space and sorting requirements, the zset structure of the Redis cache is adopted to store the IDs of the sorted data, achieving the purpose of storing sorted data in a smaller space; when querying data, the data ID is read through the cache, and the data is directly read from the database through the ID, which improves the speed of obtaining data.

[0159] (2) Data snapshot version number ensures the consistency of read data: a version number is generated for each modification. All page data of the same user are data snapshots under the same version number, which ensures the consistency of user displayed data and avoids the occurrence of duplicate data.

[0160] (3) Supports arbitrary adjustment of the existing data location: Since a corresponding data snapshot will be generated for each data change, it supports arbitrary adjustment of the current data location. The newly adjusted data can only be obtained through the newly adjusted data snapshot, and the data location is isolated from the existing data snapshot, thereby avoiding the occurrence of duplicate data. The so-called data location isolation means that the id sequence of the data in the database is changed without affecting the id sequence of the data in the database at the previous moment. For example, assuming that the data id sequence of the initial database is 1234, if the data id sequence in the database is changed to 1324, then when the subsequent request to obtain data with an id sequence of 1324 is made, it has nothing to do with the data id position 1234 in the initial database; of course, if there is another position adjustment, it has nothing to do with the previous data position.

[0161] In addition, when the amount of data is not large, a copy of the data put into the database can be stored in the cache synchronously, thereby improving the efficiency of data reading.

[0162] Based on the foregoing embodiments, an embodiment of the present application provides a data processing device, which includes the modules included and the units included in each module, and can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0163] Figure 8 This is a structural diagram of the data processing device according to an embodiment of the present application. Figure 8As shown, the device 80 includes a determination module 81, a query module 82 and a reading module 83, wherein: the determination module 81 is used to determine the first version identifier of the database corresponding to when reading the data of the first page; the query module 82 is used to query the version snapshot corresponding to the first version identifier; wherein the version snapshot is used to record the data arrangement of the database corresponding to when reading the first page of data; the reading module 83 is used to read the data of the kth page from the database according to the data identifier recorded in the version snapshot; wherein k is greater than 1.

[0164] In some embodiments, the data processing device 80 also includes a query module and an acquisition module, wherein the query module is used to query whether the first version identifier is included in the request list; the acquisition module is used to obtain the second version identifier of the current database if the first version identifier is not included in the request list; and use the second version identifier as the first version identifier.

[0165] In some embodiments, the determination module 81 is further used to determine whether the version snapshot corresponding to the first version identifier is invalid if the request list contains the first version identifier; the acquisition module is further used to obtain the second version identifier of the current database if invalid; and use the second version identifier as the first version identifier.

[0166] Figure 9 This is a structural diagram of the data processing device according to an embodiment of the present application. Figure 9 As shown, the device 90 includes a determination module 91, a generation module 92 and a storage module 93, wherein: the determination module 91 is used to determine the data arrangement of the database; the generation module 92 is used to generate a version snapshot and version identification corresponding to the database based on the data arrangement; the storage module 93 is used to associate and store the version snapshot with the version identification of the corresponding database.

[0167] In some embodiments, the determination module 91 is further configured to determine the data arrangement of the updated database when detecting that the database is updated.

[0168] In some embodiments, the determination module 91 is used to determine the data arrangement of the initial database.

[0169] In some embodiments, the data processing device 90 also includes a setting module, a generation module 92, for generating a snapshot identifier of the version snapshot; a storage module 93, for associating the snapshot identifier of the version snapshot with the version identifier of the corresponding database and storing them in a request list; the setting module is used to set the validity period of the version snapshot corresponding to the snapshot identifier in the request list.

[0170] In some embodiments, the data processing device 90 also includes a deletion module, wherein the determination module 91 is used to query the request list to determine whether the version snapshot corresponding to each snapshot identifier is within the corresponding validity period; the deletion module is used to delete the invalid version snapshot and the corresponding table entry in the request list.

[0171] In some embodiments, the data processing device 90 also includes a response module, which is used to respond to a data deletion operation; a determination module 91 is used to determine the data identifier of the target data to be deleted based on the data deletion operation; the deletion module is also used to delete the target data from the database when each valid version snapshot does not record the data identifier of the target data.

[0172] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0173] It should be noted that in the embodiments of this application Figure 8 and Figure 9 The division of modules in the data processing device shown is schematic and is only a logical functional division. In actual implementation, other division methods may be used. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. It can also be implemented in the form of a combination of software and hardware.

[0174] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0175] An embodiment of the present application provides an electronic device, Figure 10 This is a schematic diagram of the hardware entity of the electronic device according to the embodiment of the present application, such as Figure 10As shown, the electronic device 100 includes a memory 101 and a processor 102. The memory 101 stores a computer program that can be run on the processor 102. When the processor 102 executes the program, the steps in the method provided in the above embodiment are implemented.

[0176] It should be noted that the memory 101 is configured to store instructions and applications executable by the processor 102, and can also cache data to be processed or processed by the processor 102 and various modules in the electronic device 100 (for example, image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0177] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method provided in the above embodiment are implemented.

[0178] An embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0179] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0180] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0181] The term "and / or" in this article is only a description of the association relationship between associated objects, indicating that there can be three relationships. For example, object A and / or object B can mean: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0182] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0184] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed across multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of this embodiment.

[0185] In addition, all functional modules in the embodiments of the present application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0186] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0187] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0188] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0189] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0190] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0191] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Determining a first version identifier, where the first version identifier is used to identify a database version when reading first page data of a webpage from a database, where the database contains data of multiple pages of the webpage; Querying a version snapshot corresponding to the first version identifier; wherein the version snapshot is used to record the data arrangement of the database when the first page of webpage data is read from the database; According to the data identifier of the data on page k of the web page recorded in the version snapshot, the data on page k of the web page is read from the database; wherein k is greater than 1.

2. The method according to claim 1, characterized in that Determining the first version identifier includes: Query whether the request list contains the first version identifier; If the request list does not include the first version identifier, obtaining the second version identifier of the current database; The second version identifier is used as the first version identifier.

3. The method according to claim 2, characterized in that The method further comprises: If the request list includes the first version identifier, determining whether the version snapshot corresponding to the first version identifier is invalid; If invalid, obtain a second version identifier of the current database; and use the second version identifier as the first version identifier.

4. A data processing method, characterized in that: The method comprises: determining a data arrangement in a database, wherein the database includes data of a plurality of pages of a web page; Based on the data arrangement, generating a version snapshot and a version identifier corresponding to the database; The version snapshot corresponding to the database is associated with the version identifier corresponding to the database and stored, the version identifier is used to identify the version of the database, and the version snapshot is used to record the data arrangement of the database. The version snapshot at least includes the data identifier of the data of the kth page of the web page; where k is greater than 1.

5. The method according to claim 4, characterized in that Determining the data arrangement of the database includes: When it is detected that the database is updated, the data arrangement of the updated database is determined.

6. The method according to claim 5, characterized in that The step of determining the data arrangement of the database further includes: determining the data arrangement of the initial database.

7. The method according to claim 4, characterized in that The storing the version snapshot corresponding to the database in association with the version identifier corresponding to the database includes: Generate a snapshot identifier of the version snapshot; The snapshot identifier of the version snapshot and the version identifier corresponding to the database are associated and stored in a request list; and the validity period of the version snapshot corresponding to the snapshot identifier is set in the request list.

8. The method according to claim 7, characterized in that The method further comprises: Query the request list to determine whether the version snapshot corresponding to each snapshot identifier is within the corresponding validity period; Delete the invalid version snapshot and the corresponding entry in the request list.

9. The method according to claim 8, characterized in that The method further comprises: In response to a data deletion operation; Determining the data identifier of the target data to be deleted according to the data deletion operation; When the data identifier of the target data is not recorded in each valid version snapshot, the target data is deleted from the database.

10. A data processing device, characterized in that: include: a determination module, configured to determine a first version identifier, wherein the first version identifier is used to identify a database version when reading first page data of a webpage from a database, wherein the database contains data of multiple pages of the webpage; A query module, configured to query a version snapshot corresponding to the first version identifier; wherein the version snapshot is used to record the data arrangement of the database when the first page of webpage data is read from the database; The reading module is configured to read the data of the kth page of the web page from the database according to the data identifier of the data of the kth page of the web page recorded in the version snapshot; wherein k is greater than 1.

11. A data processing device, characterized in that: include: a determination module, configured to determine a data arrangement in a database, wherein the database includes data of a plurality of pages of a web page; A generation module, configured to generate a version snapshot and a version identifier corresponding to the database based on the data arrangement; A storage module is used to associate and store the version snapshot corresponding to the database with the version identifier corresponding to the database; the version identifier is used to identify the version of the database, and the version snapshot is used to record the data arrangement of the database, and the version snapshot at least includes the data identifier of the data of the kth page of the web page; where k is greater than 1.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images