Data processing method and device, electronic equipment and computer storage medium
By determining and updating historical data in the cache when receiving target service data in the application, the problem of data redundancy on third-party platforms is solved, the data structure is made simple and maintainable, and system storage resources are saved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING HONGTENG INTELLIGENT TECH CO LTD
- Filing Date
- 2024-11-11
- Publication Date
- 2026-05-12
AI Technical Summary
In application development, when third-party platforms provide data, they cannot provide an identifier for deleted data, leading to the accumulation and redundancy of historical data and wasting system storage resources.
When receiving target data from the target service, historical data is retrieved from the preset cache, the data type is determined, and the data to be updated is identified, including data to be deleted, added, or modified. The corresponding update operations are then performed to generate the updated historical data.
Effectively clean up invalid or redundant data in history, keep local cached data consistent with target service data, avoid data bloat, improve the simplicity and maintainability of data structure, and save system storage resources.
Smart Images
Figure CN122019513A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data synchronization technology, and in particular to a data processing method, apparatus, electronic device, and computer storage medium. Background Technology
[0002] During application development, it's common to integrate data from third-party platforms, such as third-party accounts, organizational structures, push notifications, and product listings. When providing data, these platforms typically offer the full dataset or provide incremental updates after the initial full dataset. However, if the third-party platform cannot identify deleted data or provide deletion records, the application struggles to determine which data no longer exists. This leads to the accumulation and redundancy of historical data, wasting system storage resources. Summary of the Invention
[0003] This application provides a data processing method, apparatus, electronic device, and computer storage medium that can avoid data bloat, maintain the simplicity and maintainability of the data structure, and save system storage resources. The above technical solutions are as follows:
[0004] In a first aspect, embodiments of this application provide a data processing method, the method comprising:
[0005] When target data corresponding to the target service is received, historical data corresponding to the target service is retrieved from the preset cache.
[0006] Determine whether the target data mentioned above is the complete dataset;
[0007] If the target data is the full data, the data to be updated is determined based on the target data and the historical data, including the data to be deleted.
[0008] The historical data is updated based on the data to be updated, resulting in the updated historical data.
[0009] In one possible implementation, after receiving the target data corresponding to the target service, the method further includes:
[0010] Determine whether the target data mentioned above is the data corresponding to the target service received for the first time;
[0011] If not, then perform the steps described above to obtain the historical data corresponding to the target service from the preset cache;
[0012] If so, the target data is stored in the preset cache, and the target data is used as the historical data corresponding to the target service when new target data corresponding to the target service is received next time.
[0013] In one possible implementation, determining whether the target data is the complete dataset includes:
[0014] Obtain the configuration information file corresponding to the target service mentioned above;
[0015] Query the data identifier corresponding to the target service from the above configuration information file;
[0016] Based on the above data identifiers, determine whether the target data is the full dataset.
[0017] In one possible implementation, the target data includes at least one first sub-data and a first identifier for each first sub-data, and the historical data includes at least one second data and a second identifier for each second sub-data. Determining the data to be updated based on the target data and the historical data includes:
[0018] Determine a first difference set between at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the first difference set is a set of second identifiers that belong to the historical data but not to the target data;
[0019] Determine whether the first difference set mentioned above is an empty set;
[0020] If the first difference set is not empty, the second sub-data corresponding to the second identifier in the first difference set is determined as the data to be deleted.
[0021] In one possible implementation, the target data includes at least one first sub-data and a first identifier for each first sub-data; the historical data includes at least one second data and a second identifier for each second sub-data; the data to be updated also includes data to be modified; and determining the data to be updated based on the target data and the historical data includes:
[0022] Determine the intersection of at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the intersection is a set of common identifiers that belong to both the target data and the historical data;
[0023] For each shared identifier included in the above intersection, determine whether the first sub-data corresponding to the shared identifier is consistent with the second sub-data corresponding to the shared identifier;
[0024] If the first sub-data corresponding to the aforementioned shared identifier is inconsistent with the second sub-data corresponding to the aforementioned shared identifier, the second sub-data corresponding to the aforementioned shared identifier will be identified as the data to be modified.
[0025] In one possible implementation, the historical data is updated based on the data to be updated to obtain the updated historical data, including:
[0026] The data to be deleted, which is included in the data to be updated, is removed from the historical data to obtain the updated historical data.
[0027] In one possible implementation, the historical data is updated based on the data to be updated to obtain the updated historical data, including:
[0028] Generate deletion tags for the aforementioned data to be deleted included in the historical data to obtain the updated historical data;
[0029] The above methods also include:
[0030] The second data corresponding to the tags to be deleted in the above historical data is deleted based on a preset period.
[0031] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:
[0032] The acquisition module is used to retrieve historical data corresponding to the target service from a preset cache when target data corresponding to the target service is received.
[0033] The first determining module is used to determine whether the above target data is the complete data;
[0034] The second determining module is used to determine the data to be updated based on the target data and the historical data when the target data is the full data, and the data to be updated includes the data to be deleted.
[0035] The update module is used to update the historical data based on the data to be updated, so as to obtain the updated historical data.
[0036] In one possible implementation, the above-mentioned device further includes:
[0037] The judgment module is used to determine whether the target data is the data corresponding to the target service received for the first time.
[0038] If not, the above-mentioned acquisition module performs the above-mentioned step of acquiring the historical data corresponding to the target service from the preset cache;
[0039] The storage module is used to store the target data in the preset cache when the target data is the data corresponding to the target service received for the first time, and to use the target data as the historical data corresponding to the target service obtained when the new target data corresponding to the target service is received next time.
[0040] In one possible implementation, the first determining module mentioned above includes:
[0041] The acquisition unit is used to acquire the configuration information file corresponding to the target service mentioned above.
[0042] The query unit is used to query the data identifier corresponding to the target service from the above configuration information file.
[0043] The first determining unit is used to determine whether the target data is the full data based on the data identifier.
[0044] In one possible implementation, the target data includes at least one first sub-data and a first identifier for each first sub-data; the historical data includes at least one second data and a second identifier for each second sub-data; and the second determining module includes:
[0045] The second determining unit is used to determine a first difference set between at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the first difference set is a set of second identifiers that belong to the historical data but do not belong to the target data.
[0046] The judgment unit is used to determine whether the first difference set is an empty set.
[0047] The third determining unit is used to determine the second sub-data corresponding to the second identifier in the first difference set as data to be deleted when the first difference set is not an empty set.
[0048] In one possible implementation, the target data includes at least one first sub-data and a first identifier for each first sub-data; the historical data includes at least one second data and a second identifier for each second sub-data; and the second determining module includes:
[0049] The fourth determining unit is used to determine the intersection of at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the intersection is a set of common identifiers that belong to both the target data and the historical data.
[0050] The fifth determining unit is used to determine whether the first sub-data corresponding to the common identifier and the second sub-data corresponding to the common identifier are consistent for each common identifier included in the above intersection;
[0051] The sixth determining unit is used to determine the second sub-data corresponding to the shared identifier as data to be modified when the first sub-data corresponding to the shared identifier is inconsistent with the second sub-data corresponding to the shared identifier.
[0052] In one possible implementation, the above-mentioned update module includes:
[0053] The deletion unit is used to delete the data to be deleted from the historical data included in the data to be updated, so as to obtain the updated historical data.
[0054] In one possible implementation, the above-mentioned update module includes:
[0055] The generation unit is used to generate deletion tags for the data to be deleted included in the above-mentioned historical data, so as to obtain updated historical data.
[0056] The above-mentioned device also includes:
[0057] The deletion module is used to delete the second data corresponding to the tags to be deleted in the above historical data based on a preset period.
[0058] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory;
[0059] The aforementioned memory stores a computer program adapted to be loaded by the aforementioned processor and execute the steps of the method provided by the first aspect of the embodiments of this application or any possible implementation thereof.
[0060] Fourthly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the method provided by the first aspect of the embodiments of this application or any possible implementation thereof.
[0061] This application embodiment retrieves historical data corresponding to the target service from a preset cache when target data corresponding to the target service is received; determines whether the target data is the full set of data; if the target data is the full set of data, determines the data to be updated based on the target data and the historical data, including data to be deleted; and updates the historical data based on the data to be updated to obtain the updated historical data. By comparing the target data and the historical data, it is possible to identify which data needs to be updated and which data needs to be deleted, ensuring that the locally cached data is consistent with the data of the target service; and the data to be updated includes the data to be deleted, which can effectively clean up expired or redundant data in the history, avoiding inconsistencies caused by data backlog and redundancy, preventing data bloat, improving the simplicity and maintainability of the data structure, and effectively saving system storage resources. Attached Figure Description
[0062] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0063] Figure 1 A schematic diagram of the structure of a data processing system provided for an exemplary embodiment of this application;
[0064] Figure 2 A flowchart illustrating a data processing method provided for an exemplary embodiment of this application;
[0065] Figure 3 A detailed flowchart illustrating a method for determining data to be deleted, provided as an exemplary embodiment of this application;
[0066] Figure 4 A flowchart illustrating a method for determining data to be modified, provided as an exemplary embodiment of this application;
[0067] Figure 5 A schematic diagram of the structure of a data processing apparatus provided for an exemplary embodiment of this application;
[0068] Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation
[0069] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0070] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0071] Please refer to the following. Figure 1 The example illustrates a schematic diagram of the structure of a data processing system provided in an embodiment of this application. Figure 1 As shown, the system includes a first device 110 and a second device 120.
[0072] The first device 110 and the second device 120 can be connected directly or indirectly via wired or wireless communication, and this application does not impose any restrictions on this connection.
[0073] In some embodiments, the first device 110 and the second device 120 may be smartphones, tablets, laptops, desktop computers, smart speakers, smart home appliances, etc., but are not limited thereto.
[0074] In some embodiments, the first device 110 and the second device 120 may also be servers. For example, they may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), big data and artificial intelligence platforms.
[0075] In some embodiments, the first device 110 may have an application installed, and the second device 120 may have a third-party platform installed. The third-party platform in the second device 120 provides data target services to the application in the first device 110 and sends data related to the target services to the first device 110.
[0076] In some embodiments, the first device 110 is configured to perform the following steps: when receiving target data corresponding to a target service, obtaining historical data corresponding to the target service from a preset cache; determining whether the target data is full data; if the target data is full data, determining data to be updated based on the target data and the historical data, wherein the data to be updated includes data to be deleted; updating the historical data based on the data to be updated to obtain updated historical data.
[0077] Optionally, a target service is an external service or platform that is interacted with during application development to achieve specific functions or business requirements. Target services can specifically be external social networking platforms, payment systems, map services, data providers, etc., which provide the application with necessary data, functions, or resources.
[0078] Optionally, the target data may be updated user information, new transaction records, or any other information related to the target service.
[0079] Optionally, historical data refers to previous data that the system has received or processed in relation to the current target service.
[0080] Optionally, data to be deleted refers to data that needs to be removed from the application, typically data that has been marked as invalid, no longer used, or completely deleted by a third-party platform.
[0081] Optionally, the updated historical data is the data after performing at least one of the following operations on the historical data: deletion, addition, or modification.
[0082] An exemplary embodiment of this application provides a data processing method. This data processing method can be applied to the first device described above. Please refer to [reference needed] for details. Figure 2 The example illustrates a flowchart of a data processing method provided in an embodiment of this application. Figure 2 As shown, the data processing method includes the following S21-S24:
[0083] S21. When the target data corresponding to the target service is received, retrieve the historical data corresponding to the target service from the preset cache.
[0084] In some embodiments, the target data corresponding to the target service received by the first device may be sent by the second device.
[0085] In this context, a target service refers to an external service or platform that is interacted with during application development to achieve specific functionalities or business requirements. Target services can specifically include external social networking platforms, payment systems, map services, data providers, etc., which provide the application with necessary data, functionality, or resources.
[0086] In some embodiments, the target data may be updated user information, new transaction records, or any other information related to the target service.
[0087] Specifically, the target data mentioned above includes at least one first sub-data and a first identifier for each first sub-data.
[0088] In some embodiments, the preset cache can be implemented in various ways, depending on the application's needs and use cases. Optionally, the preset cache can be any of memory, local storage, or browser cache. Specific memory caching tools include the remote dictionary service Redis and the distributed high-speed caching system Memcached. Redis is an open-source, network-enabled, in-memory or persistent log-structured database that stores data in key-value pairs. Memcached is also a key-value storage system, suitable for caching data that does not change frequently and providing fast data retrieval. Local storage can include lightweight local databases such as SQLite and file caches.
[0089] In some embodiments, historical data refers to previous data that the system has received or processed in relation to the current target service. Historical data can be used for comparison, retrospection, updating, or analysis.
[0090] Optionally, the aforementioned historical data includes at least one second data and a second identifier for each second sub-data.
[0091] S22. Determine whether the above target data is the complete data.
[0092] In this context, "full dataset" refers to all relevant data returned by the third-party platform in a single request. Each time the target service provided by the third-party platform is called or accessed, the complete dataset is retrieved, not just newly added or modified data. For example, when the target data is full dataset, the retrieved user information will include detailed information for all users, regardless of whether that user's information has recently changed.
[0093] In some embodiments, the target data may also be incremental data. Incremental data refers to data that has changed since the last request, returned by the third-party platform in a given request; that is, it only includes newly added, modified, or deleted data records. For example, if the target data is incremental data, the target service may only return information on recently added users, users whose profiles have been modified, and deleted users, without including other unchanged user data.
[0094] S23. If the target data is the full data, determine the data to be updated based on the target data and the historical data. The data to be updated includes the data to be deleted.
[0095] In some embodiments, the data to be updated may also include any one or more of the following: data to be added, data to be modified, etc.
[0096] Optionally, data to be deleted refers to data that needs to be removed from the application, typically data marked as invalid, no longer used, or permanently deleted by a third-party platform. For example, if a user account is canceled or deleted, the data to be deleted will contain that user's relevant information.
[0097] Optionally, "data to be added" refers to new data that needs to be inserted into the application. For example, if a third-party platform receives a new user registration, the "data to be added" will contain the new user's detailed information.
[0098] Optionally, data to be modified refers to records that already exist in the application, but whose fields or information have changed and need to be updated. For example, if a user changes their email address, the third-party platform will receive the user ID and the new email address as data to be modified.
[0099] S24. Update the historical data based on the data to be updated to obtain the updated historical data.
[0100] The updated historical data refers to data after performing at least one of the following operations on historical data: deletion, addition, or modification. After any one or more of these operations, the new dataset can be stored as the updated historical data. This updated data represents the latest data snapshot of the system and can be used as the historical data corresponding to the aforementioned target service when new target data corresponding to the aforementioned target service is received next time.
[0101] This application embodiment retrieves historical data corresponding to the target service from a preset cache when target data corresponding to the target service is received; determines whether the target data is the full set of data; if the target data is the full set of data, determines the data to be updated based on the target data and the historical data, including data to be deleted; and updates the historical data based on the data to be updated to obtain the updated historical data. By comparing the target data and the historical data, it is possible to identify which data needs to be updated and which data needs to be deleted, ensuring that the locally cached data is consistent with the data of the target service; and the data to be updated includes the data to be deleted, which can effectively clean up expired or redundant data in the history, avoiding inconsistencies caused by data backlog and redundancy, preventing data bloat, improving the simplicity and maintainability of the data structure, and effectively saving system storage resources.
[0102] In some embodiments, after receiving the target data corresponding to the target service in S21 above, the method further includes S201-S202:
[0103] S201. Determine whether the target data is the data corresponding to the target service received for the first time. If not, execute the step in S21 above to obtain the historical data corresponding to the target service from the preset cache; if yes, execute S202.
[0104] In some embodiments, determining whether the target data corresponds to the first received data of the target service can be achieved by querying the unique identifier (such as the service ID) of the target service to check whether the preset cache contains historical data corresponding to the target service. If the query result is empty, it means that there is no historical data, i.e., the target data corresponds to the first received data of the target service; if the query result is not empty, it means that the data of the target service has been received before, i.e., the target data does not correspond to the first received data of the target service, and then the step of retrieving the historical data corresponding to the target service from the preset cache can be performed.
[0105] S202. Store the target data in the preset cache and use the target data as historical data corresponding to the target service when receiving new target data corresponding to the target service for the next time.
[0106] If the target data is the data corresponding to the target service received for the first time, the target data can be directly stored in the preset cache without updating the historical data.
[0107] In this embodiment of the application, when the target data is received for the first time, since there is no historical data that needs to be updated, it can be directly stored in the cache. There is no need to retrieve historical data from the preset cache or perform an update operation. This avoids meaningless cache reading and data comparison and saves computing resources.
[0108] In some embodiments, determining whether the target data is the full data in S22 above includes S221-S223:
[0109] S221. Obtain the configuration information file corresponding to the target service mentioned above.
[0110] In some embodiments, the configuration information file corresponding to the target service can be used to describe the parameters, rules, or structured files required for the target service or application to run. The configuration information file can be stored locally on the first device, on a remote server, in a database, or provided by a third-party platform, and the first device can access and load the file.
[0111] S222. Query the data identifier corresponding to the target service from the above configuration information file.
[0112] In some embodiments, the data identifier corresponding to the target service described above can be used to indicate whether the target data is full data or incremental data.
[0113] Specifically, the data identifier can be a flag field indicating the type of the target data (full data or incremental data). For example, a first preset data identifier is used to characterize the target data as full data, and a second preset data identifier is used to characterize the target data as incremental data.
[0114] S223. Determine whether the target data is the full data based on the above data identifier.
[0115] In some embodiments, in S223, it is determined whether the target data is the full data based on the data identifier. If the data identifier includes:
[0116] If the above data identifier is the first preset data identifier, then the above target data is determined to be full data; if the above data identifier is the second preset data identifier, then the above target data is determined to be incremental data.
[0117] In this embodiment of the application, by setting a data identifier in the configuration information file, it is possible to automatically distinguish whether the received target data is full data or incremental data. No human intervention is required, the degree of automation is high, the data processing flow of the system is greatly simplified, errors caused by manually determining the data type are avoided, and the accuracy of data processing is improved.
[0118] In some embodiments, in S23 above, the determination of the data to be updated based on the target data and the historical data includes S231-S233:
[0119] S231. Determine a first difference set between at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the first difference set is a set of second identifiers that belong to the historical data but not to the target data.
[0120] In some embodiments, in S231, determining the first difference between at least one first identifier included in the target data and at least one second identifier included in the historical data specifically includes: determining the first difference between at least one first identifier included in the target data and at least one second identifier included in the historical data using the Redis set operation command Set Difference (SDIFF). Here, SDIFF is a data type in Redis used to calculate the difference between two sets. Specifically, it returns elements that appear in the first set but not in the second set, and these elements constitute a new set.
[0121] For example, historical data may include: second sub-data D1, second sub-data D2, second sub-data D3, second sub-data D4, and second sub-data D5. Each second sub-data includes information corresponding to the following fields: user identifier (IdentityDocument, ID), username, account, organization, and account status. The user ID can be used as the unique identifier for that second sub-data on the third-party platform, i.e., the primary identifier corresponding to the second sub-data. Specifically, the details of the historical data are shown in Table 1.
[0122] Table 1: Historical Data Table
[0123] User ID username account organize state 1 Zhang San zhangsan Product Group Enable 2 Li Si lisi R&D team Enable 3 Wang Wu wangwu test group Enable 4 Zhao Liu zhaoliu R&D team Enable 5 Sun Qi sunqi test group Enable
[0124] As shown in Table 1, the content of the second sub-data D1 is: 1. Zhang San, product group, enabled; the content of the second sub-data D2 is: 2. Li Si, R&D group, enabled; the content of the second sub-data D3 is: 3. Wang Wu, testing group, enabled; the content of the second sub-data D4 is: 4. Zhao Liu, R&D group, enabled; the content of the second sub-data D5 is: 5. Sun Qi, testing group, enabled.
[0125] The target data may include: first sub-data D1, first sub-data D2, first sub-data D3, and first sub-data D4. Each first sub-data includes information corresponding to the following fields: user ID, username, account, organization, and account status. Among them, the user ID can be used as the unique identifier corresponding to the first sub-data on the third-party platform, that is, the first identifier corresponding to the first sub-data.
[0126] Specifically, the details of the target data are shown in Table 2:
[0127] Table 2 Data Table of Target Data
[0128] User ID username account organize state 1 Zhang San zhangsan Product Group Enable 2 Li Si lisi R&D team Disable 3 Wang Wu wangwu test group Enable 4 Zhao Liu zhaoliu R&D team Enable
[0129] As shown in Table 2, the content of the first sub-data D1 is: 1. Zhang San, product group, enabled; the content of the first sub-data D2 is: 2. Li Si, R&D group, disabled; the content of the first sub-data D3 is: 3. Wang Wu, test group, enabled; the content of the first sub-data D4 is: 4. Zhao Liu, R&D group, enabled.
[0130] Furthermore, at least one second identifier included in the historical data can be represented as set A{1,2,3,4,5}, and at least one first identifier included in the target data can be represented as set B{1,2,3,4}. Thus, the first difference set between at least one first identifier included in the target data (set A) and at least one second identifier included in the historical data (set B) is represented as (AB).
[0131] S232. Determine whether the first difference set is an empty set.
[0132] As can be seen from the above, the first difference set (AB) is specifically {5}, and the first difference set is not an empty set.
[0133] S233. If the first difference set is not empty, the second sub-data corresponding to the second identifier in the first difference set is determined as the data to be deleted.
[0134] In other words, the third-party platform deleted the data with user ID 5 during the two data synchronizations. To maintain data consistency, the second sub-data D5 corresponding to this second identifier is designated as data to be deleted.
[0135] If the first difference set is empty, the data to be updated does not include the data to be deleted.
[0136] In this embodiment of the application, by using set difference operation, it is possible to effectively discover which data was deleted by the third-party platform between two data synchronizations, and to perform timely local synchronization deletion operations to ensure that the local data is consistent with the third-party platform, prevent data failure or redundancy, and ensure that the data used locally is up-to-date and valid. This is beneficial for subsequent operations to be based on the latest data, and effectively improves data processing efficiency.
[0137] In some embodiments, the data to be updated also includes data to be modified. In S23, the determination of the data to be updated based on the target data and the historical data includes S234-S236:
[0138] S234. Determine the intersection of at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the intersection is a set of common identifiers that belong to both the target data and the historical data.
[0139] Taking Tables 1 and 2 above as examples, at least one second identifier included in the historical data can be represented as set A{1,2,3,4,5}, and at least one first identifier included in the target data can be represented as set B{1,2,3,4}. Therefore, the intersection of at least one first identifier included in the target data (set A) and at least one second identifier included in the historical data (set B) is represented as {1,2,3,4}.
[0140] S235. For each common identifier included in the above intersection, determine whether the first sub-data corresponding to the common identifier is consistent with the second sub-data corresponding to the common identifier.
[0141] For the common identifiers 1, 2, 3, 4 in the intersection {1,2,3,4}, it can be determined whether the first sub-data corresponding to each common identifier is consistent with the second sub-data corresponding to the common identifier.
[0142] S236. If the first sub-data corresponding to the above-mentioned shared identifier is inconsistent with the second sub-data corresponding to the above-mentioned shared identifier, the second sub-data corresponding to the above-mentioned shared identifier shall be determined as the data to be modified.
[0143] As can be seen from the above, the data corresponding to the first sub-data D2 "2, Li Si, lisi, R&D Group, Disabled" of the shared identifier 2 is different from the data corresponding to the "Account Status" field in the second sub-data D2 "2, Li Si, lisi, R&D Group, Enabled" of the shared identifier 2. Therefore, the first sub-data corresponding to the shared identifier is inconsistent with the second sub-data corresponding to the shared identifier, and thus the second sub-data D2 "2, Li Si, lisi, R&D Group, Enabled" of the shared identifier 2 is determined to be the data to be modified.
[0144] In this embodiment, by calculating the intersection of target data and historical data, it is possible to identify which data exists simultaneously in both, and then compare whether their fields are consistent. Inconsistent parts are marked as data to be modified, ensuring consistency between local data and data on third-party platforms and avoiding errors or conflicts caused by data asynchrony. Furthermore, data modification is only performed when differences are determined in certain fields, avoiding unnecessary processing of consistent data, improving data processing efficiency, and achieving refined data management.
[0145] In some embodiments, the data to be updated also includes data to be added. In S23, the determination of the data to be updated based on the target data and the historical data includes S237-S239:
[0146] S237. Determine a second difference set between at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the second difference set is a set of first identifiers that belong to the target data but not to the historical data.
[0147] In some embodiments, the second difference set between at least one first identifier included in the target data and at least one second identifier included in the historical data can be determined by using the Redis set operation command SDIFF method.
[0148] S238. Determine whether the above second difference set data is an empty set;
[0149] S239. If the second difference set data is not an empty set, the first sub-data corresponding to the first identifier in the second difference set is determined as the data to be added.
[0150] If the second difference set data is not an empty set, the first identifier included in the second difference set is the data added by the third-party platform during the two data synchronization periods. Therefore, the first sub-data corresponding to the first identifier in the second difference set is determined as the data to be added.
[0151] In this embodiment, the second difference between the target data and historical data is calculated using the Redis set operation command SDIFF, and then used to identify newly added data. This method can efficiently handle large-scale data difference calculations, especially when frequent synchronization of large amounts of data is required. The SDIFF command can quickly calculate the difference between the target data and historical data and identify newly added data, avoiding the complexity of manual comparison. Furthermore, by automatically identifying new items in the target data through the difference method, the detection and processing of new data can be automated without human intervention, improving the efficiency and reliability of data processing.
[0152] In some embodiments, in S24 above, updating the historical data based on the data to be updated to obtain the updated historical data includes S241:
[0153] S241. Delete the data to be deleted from the above-mentioned historical data, which is included in the data to be updated, to obtain the updated historical data.
[0154] Through the first difference calculation mentioned above, we have identified which data belongs to the data to be deleted. These data exist in the historical data but no longer appear in the latest target data. Therefore, the data to be deleted can be deleted directly. After the deletion operation is completed, the historical dataset is updated, and the updated historical data is obtained, maintaining the synchronization between the local data and the data on the third-party platform.
[0155] In this embodiment, by deleting data that no longer exists, errors are prevented from occurring locally due to the retention of outdated data, and redundant data in local storage is reduced, thus optimizing data storage efficiency. Especially in large-scale data scenarios, timely deletion of invalid data can significantly reduce wasted storage space.
[0156] In other embodiments, in S24 above, updating the historical data based on the data to be updated to obtain updated historical data includes S242:
[0157] S242. Generate deletion tags for the data to be deleted included in the above historical data to obtain updated historical data;
[0158] Furthermore, the above method also includes: deleting the second data corresponding to the tags to be deleted in the above historical data based on a preset period.
[0159] Generate deletion tags for data to be deleted in historical data, indicating that the sub-data has been identified as pending deletion but has not been physically deleted immediately. Specifically, a specific field can be added to the data record to indicate that the data has been marked as pending deletion. In this way, data that is pending deletion can be marked without immediate deletion.
[0160] In this embodiment, by generating a tag to be deleted instead of immediately physically deleting it, the local first device gains greater control and can flexibly schedule the actual deletion time. Based on business needs, the system can choose an appropriate time to delete without causing system resource conflicts or a decline in user experience due to immediate deletion, thus further improving performance.
[0161] In some embodiments, when the target data is incremental data, only the incremental data (i.e., data that has changed since the last synchronization) can be processed, and data can be updated, deleted, or added by data lookup (i.e., comparison with historical data). This can greatly improve efficiency and avoid unnecessary computation and resource waste caused by full data synchronization.
[0162] In some embodiments, when the data to be updated includes data to be modified, in the above S24, updating the historical data according to the data to be updated to obtain the updated historical data includes: determining the modified data corresponding to the data to be modified from the target data, modifying the data to be modified in the historical data according to the modified data, and obtaining the updated historical data.
[0163] In some embodiments, when the data to be updated includes data to be added, in the above-mentioned S24, updating the historical data according to the data to be updated to obtain the updated historical data includes: using the historical data and the data to be added together as the updated historical data.
[0164] Furthermore, the present application will be described below with reference to specific embodiments. Figure 3 A detailed flowchart illustrating a method for determining data to be deleted, provided as an exemplary embodiment of this application, is shown below. Figure 3 The method for determining the data to be deleted specifically includes S301-S309:
[0165] S301, Receive the target data corresponding to the target service.
[0166] Optionally, the first device may receive target data corresponding to the target service from the second device via a preset communication protocol.
[0167] S302. Determine whether the target data is the data corresponding to the target service received for the first time. If yes, proceed to S303; otherwise, proceed to S304.
[0168] S303. Store the target data in a preset cache, and use the target data as the historical data corresponding to the target service when the next new target data corresponding to the target service is received.
[0169] The steps S302-S303 are the same as those in S201-S202 above, and will not be repeated here.
[0170] S304. Retrieve historical data corresponding to the target service from the preset cache.
[0171] S305. Determine if the target data is the complete set of data. If not, proceed to S306; if yes, proceed to S307.
[0172] Steps S304-S305 are the same as those in S21-S22 above, and will not be repeated here.
[0173] S306. Determine the data to be updated based on the target data.
[0174] Optionally, when the target data is incremental data, the target data may include: new records (records added since the last acquisition); modified records (modification information of existing records); and deletion markers (data records marked as deleted by a third-party platform). This allows for the direct determination of data to be added based on new records, data to be modified based on modified records, and data to be deleted based on deletion markers.
[0175] S307. Determine the first difference set between at least one first identifier included in the target data and at least one second identifier included in the historical data.
[0176] S308. Determine if the first difference set is empty. If yes, there is no data to be deleted, and the process ends; otherwise, execute S309.
[0177] S309. Determine the second sub-data corresponding to the second identifier in the first difference set as the data to be deleted.
[0178] The steps S307-S309 are the same as those in S231-S233 above, and will not be repeated here.
[0179] In this embodiment of the application, by step-by-step judging and comparing data, the data processing process can be reduced, and the consistency between local data and data from third-party platforms can be maintained, thereby reducing data errors.
[0180] Furthermore, the following specific embodiments address the method for determining the data to be modified when the target data is the full dataset. Figure 4 A flowchart illustrating a method for determining data to be modified, as provided in an exemplary embodiment of this application, is shown below. Figure 4 The methods for determining the data to be modified include S401-S405:
[0181] S401. Determine the intersection of at least one first identifier included in the target data and at least one second identifier included in the historical data.
[0182] Step S401 is the same as step S234 above, and will not be repeated here.
[0183] S402. Determine if the intersection is an empty set. If yes, proceed to S403; otherwise, proceed to S404.
[0184] S403. Determine that the data to be updated does not include the data to be modified.
[0185] Optionally, when the intersection is an empty set, it indicates that the third-party platform has not modified the data, and it is not necessary to determine the specific data to be modified.
[0186] S404. Determine whether the first sub-data and the corresponding second sub-data of each common identifier in the intersection are consistent. If yes, proceed to S403; otherwise, proceed to S405.
[0187] S405. The second sub-data corresponding to the common identifier where the first sub-data and the second sub-data are inconsistent is identified as the data to be modified.
[0188] The steps S404-S405 are the same as those in S235-S236 above, and will not be repeated here.
[0189] In this embodiment of the application, by comparing the data in the intersection, the records that need to be modified can be accurately identified, avoiding unnecessary update operations, saving system resources and time, and improving data processing efficiency.
[0190] Please refer to the following. Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in an exemplary embodiment of this application. Figure 5 As shown, the data processing apparatus 500 includes:
[0191] The acquisition module 501 is used to retrieve historical data corresponding to the target service from a preset cache when target data corresponding to the target service is received.
[0192] The first determining module 502 is used to determine whether the above target data is the full data;
[0193] The second determining module 503 is used to determine the data to be updated based on the target data and the historical data when the target data is the full data, and the data to be updated includes the data to be deleted.
[0194] The update module 504 is used to update the historical data based on the data to be updated, so as to obtain the updated historical data.
[0195] In one possible implementation, the device 500 further includes:
[0196] The judgment module is used to determine whether the target data is the data corresponding to the target service received for the first time.
[0197] If not, the above-mentioned acquisition module 501 performs the above-mentioned step of acquiring the historical data corresponding to the target service from the preset cache;
[0198] The storage module is used to store the target data in the preset cache when the target data is the data corresponding to the target service received for the first time, and to use the target data as the historical data corresponding to the target service obtained when the new target data corresponding to the target service is received next time.
[0199] In one possible implementation, the first determining module 502 described above includes:
[0200] The acquisition unit is used to acquire the configuration information file corresponding to the target service mentioned above.
[0201] The query unit is used to query the data identifier corresponding to the target service from the above configuration information file.
[0202] The first determining unit is used to determine whether the target data is the full data based on the data identifier.
[0203] In one possible implementation, the target data includes at least one first sub-data and a first identifier for each first sub-data, the historical data includes at least one second data and a second identifier for each second sub-data, and the second determining module 503 includes:
[0204] The second determining unit is used to determine a first difference set between at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the first difference set is a set of second identifiers that belong to the historical data but do not belong to the target data.
[0205] The judgment unit is used to determine whether the first difference set is an empty set.
[0206] The third determining unit is used to determine the second sub-data corresponding to the second identifier in the first difference set as data to be deleted when the first difference set is not an empty set.
[0207] In one possible implementation, the target data includes at least one first sub-data and a first identifier for each first sub-data, the historical data includes at least one second data and a second identifier for each second sub-data, and the second determining module 503 includes:
[0208] The fourth determining unit is used to determine the intersection of at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the intersection is a set of common identifiers that belong to both the target data and the historical data.
[0209] The fifth determining unit is used to determine whether the first sub-data corresponding to the common identifier and the second sub-data corresponding to the common identifier are consistent for each common identifier included in the above intersection;
[0210] The sixth determining unit is used to determine the second sub-data corresponding to the shared identifier as data to be modified when the first sub-data corresponding to the shared identifier is inconsistent with the second sub-data corresponding to the shared identifier.
[0211] In one possible implementation, the update module 504 mentioned above includes:
[0212] The deletion unit is used to delete the data to be deleted from the historical data included in the data to be updated, so as to obtain the updated historical data.
[0213] In one possible implementation, the update module 504 mentioned above includes:
[0214] The generation unit is used to generate deletion tags for the data to be deleted included in the above-mentioned historical data, so as to obtain updated historical data.
[0215] The aforementioned device 500 also includes:
[0216] The deletion module is used to delete the second data corresponding to the tags to be deleted in the above historical data based on a preset period.
[0217] The division of modules in the data processing apparatus 500 described above is for illustrative purposes only. In other embodiments, the data processing apparatus may be divided into different modules as needed to complete all or part of the functions of the data processing apparatus described above. The implementation of each module in the data processing apparatus provided in the embodiments of this specification can be in the form of a computer program. This computer program can run on a terminal or server. The program modules constituted by this computer program can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements all or part of the steps of the data processing method described in the embodiments of this specification.
[0218] Please refer to the following. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application. Figure 6 As shown, the electronic device 600 may include a processor 610 and a memory 620, and may also include a user interface 630, a network interface 640 and a communication bus 650.
[0219] The processor 610 may include one or more processing cores. The processor 610 connects to various parts within the electronic device 600 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 620, and by calling data stored in the memory 620. Optionally, the processor 610 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 610 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 610 and may be implemented as a separate chip.
[0220] The memory 620 may include random access memory (RAM) or read-only memory. Optionally, the memory 620 may include a non-transitory computer-readable storage medium. The memory 620 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as receiving functions, control functions, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 620 may also be at least one storage device located remotely from the aforementioned processor 610. Figure 6 As shown, the memory 620, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and program instructions.
[0221] Optionally, the communication bus 650 is used to realize the connection and communication between these components. The user interface 630 may include a display screen, a camera, and may also include standard wired interfaces and wireless interfaces. The network interface 640 may optionally include standard wired interfaces and wireless interfaces (such as WIFI interfaces).
[0222] exist Figure 6 In the illustrated electronic device 600, the processor 610 can be used to call program instructions stored in the memory 620 and specifically perform the following operations:
[0223] When target data corresponding to the target service is received, historical data corresponding to the target service is retrieved from the preset cache.
[0224] Determine whether the target data mentioned above is the complete dataset;
[0225] If the target data is the full data, the data to be updated is determined based on the target data and the historical data, including the data to be deleted.
[0226] The historical data is updated based on the data to be updated, resulting in the updated historical data.
[0227] In one possible implementation, after receiving the target data corresponding to the target service, the method further includes:
[0228] Determine whether the target data mentioned above is the data corresponding to the target service received for the first time;
[0229] If not, then perform the steps described above to obtain the historical data corresponding to the target service from the preset cache;
[0230] If so, the target data is stored in the preset cache, and the target data is used as the historical data corresponding to the target service when new target data corresponding to the target service is received next time.
[0231] In one possible implementation, determining whether the target data is the complete dataset includes:
[0232] Obtain the configuration information file corresponding to the target service mentioned above;
[0233] Query the data identifier corresponding to the target service from the above configuration information file;
[0234] Based on the above data identifiers, determine whether the target data is the full dataset.
[0235] In one possible implementation, the target data includes at least one first sub-data and a first identifier for each first sub-data, and the historical data includes at least one second data and a second identifier for each second sub-data. Determining the data to be updated based on the target data and the historical data includes:
[0236] Determine a first difference set between at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the first difference set is a set of second identifiers that belong to the historical data but not to the target data;
[0237] Determine whether the first difference set mentioned above is an empty set;
[0238] If the first difference set is not empty, the second sub-data corresponding to the second identifier in the first difference set is determined as the data to be deleted.
[0239] In one possible implementation, the target data includes at least one first sub-data and a first identifier for each first sub-data; the historical data includes at least one second data and a second identifier for each second sub-data; the data to be updated also includes data to be modified; and determining the data to be updated based on the target data and the historical data includes:
[0240] Determine the intersection of at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the intersection is a set of common identifiers that belong to both the target data and the historical data;
[0241] For each shared identifier included in the above intersection, determine whether the first sub-data corresponding to the shared identifier is consistent with the second sub-data corresponding to the shared identifier;
[0242] If the first sub-data corresponding to the aforementioned shared identifier is inconsistent with the second sub-data corresponding to the aforementioned shared identifier, the second sub-data corresponding to the aforementioned shared identifier will be identified as the data to be modified.
[0243] In one possible implementation, the historical data is updated based on the data to be updated to obtain the updated historical data, including:
[0244] The data to be deleted, which is included in the data to be updated, is removed from the historical data to obtain the updated historical data.
[0245] In one possible implementation, the historical data is updated based on the data to be updated to obtain the updated historical data, including:
[0246] Generate deletion tags for the aforementioned data to be deleted included in the historical data to obtain the updated historical data;
[0247] The above methods also include:
[0248] The second data corresponding to the tags to be deleted in the above historical data is deleted based on a preset period.
[0249] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above data processing apparatus are implemented as software functional units and sold or used as independent products, they can be stored in the above-described computer-readable storage medium.
[0250] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).
[0251] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.
[0252] The above-described embodiments are merely preferred embodiments of this application and are not intended to limit the scope of this application. Any modifications and improvements made by those skilled in the art to the technical solutions of this application without departing from the spirit of this application should fall within the protection scope defined by the claims of this application.
Claims
1. A data processing method, characterized in that, include: When target data corresponding to the target service is received, historical data corresponding to the target service is retrieved from the preset cache. Determine whether the target data is the full dataset; If the target data is the full data, the data to be updated is determined based on the target data and the historical data, and the data to be updated includes the data to be deleted. The historical data is updated based on the data to be updated to obtain the updated historical data.
2. The method as described in claim 1, characterized in that, After receiving the target data corresponding to the target service, the method further includes: Determine whether the target data is the data corresponding to the target service received for the first time; If not, then proceed with the step of retrieving the historical data corresponding to the target service from the preset cache; If so, the target data is stored in the preset cache, and the target data is used as the historical data corresponding to the target service when new target data corresponding to the target service is received next time.
3. The method as described in claim 1, characterized in that, Determining whether the target data is the full dataset includes: Obtain the configuration information file corresponding to the target service; Query the data identifier corresponding to the target service from the configuration information file; Determine whether the target data is the full dataset based on the data identifier.
4. The method as described in claim 1, characterized in that, The target data includes at least one first sub-data and a first identifier for each first sub-data; the historical data includes at least one second data and a second identifier for each second sub-data; the step of determining the data to be updated based on the target data and the historical data includes: Determine a first difference set between at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the first difference set is a set of second identifiers that belong to the historical data but not to the target data; Determine whether the first difference set is an empty set; If the first difference set is not empty, the second sub-data corresponding to the second identifier in the first difference set is determined as the data to be deleted.
5. The method as described in claim 1, characterized in that, The target data includes at least one first sub-data and a first identifier for each first sub-data; the historical data includes at least one second data and a second identifier for each second sub-data; the data to be updated also includes data to be modified; determining the data to be updated based on the target data and the historical data includes: Determine the intersection of at least one first identifier included in the target data and at least one second identifier included in the historical data, wherein the intersection is a set of common identifiers that belong to both the target data and the historical data; For each shared identifier included in the intersection, determine whether the first sub-data corresponding to the shared identifier is consistent with the second sub-data corresponding to the shared identifier; If the first sub-data corresponding to the shared identifier is inconsistent with the second sub-data corresponding to the shared identifier, the second sub-data corresponding to the shared identifier is determined as the data to be modified.
6. The method as described in claim 4, characterized in that, The step of updating the historical data based on the data to be updated to obtain the updated historical data includes: The data to be deleted, which is included in the data to be updated, is deleted from the historical data to obtain the updated historical data.
7. The method as described in claim 4, characterized in that, The step of updating the historical data based on the data to be updated to obtain the updated historical data includes: Generate deletion tags for the data to be deleted included in the historical data to obtain updated historical data; The method further includes: The second data corresponding to the tags to be deleted in the historical data is deleted based on a preset period.
8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire historical data corresponding to the target service from a preset cache when target data corresponding to the target service is received. The first determining module is used to determine whether the target data is the full data; The second determining module is used to determine the data to be updated based on the target data and the historical data when the target data is the full data, wherein the data to be updated includes data to be deleted. The update module is used to update the historical data based on the data to be updated, so as to obtain the updated historical data.
9. An electronic device, characterized in that, include: Processor and memory; The memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as described in any one of claims 1 to 7.