Data processing method, device, equipment, storage medium and computer program product

Abstract

Embodiments of the present application disclose a data processing method and device, equipment, storage medium and computer program product; a project data packet in local storage can be acquired, and invalid data to be repaired and valid data cached at a historical time are determined according to the project data packet; an invalid access rate of the invalid data to be repaired is determined according to data access records cached at the historical time, and an effective access rate of the valid data is determined; a repair gain value at a current time is determined according to the invalid access rate and the effective access rate; a control ratio of the invalid data to be repaired at the current time is determined according to the effective access rate and the repair gain value; and the invalid data to be repaired is repaired according to the control ratio of the invalid data to be repaired and the project data packet. In this way, the invalid data to be repaired in the cache is updated and repaired according to the control ratio, avoiding updating a large amount of data in the cache at the same time, reducing the running load of the system, effectively avoiding the phenomenon of overload, and guaranteeing the system performance.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] With the development of information technology, meeting people's needs for querying and retrieving business information mainly relies on data support from storage systems or caches. While caching can improve the efficiency of users querying business information, when old data in the storage system is republished or updated, the corresponding data in the cache will become invalid, affecting its usability. To ensure the validity of cached data, relevant technologies immediately reload the updated data from the storage system into the cache when modification or publication is detected, ensuring the validity of the cached data.

[0003] In the process of researching and practicing the prior art, the inventors of this application discovered that when the prior art detects that a large amount of data in the storage system has been modified or published, it also updates a large amount of data in the cache. This will increase the system's operating load, easily cause overload, and affect system performance. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, storage medium, and computer program product. These can reduce the system's operating load, effectively avoid overload, and ensure system performance.

[0005] This application provides a data processing method, including:

[0006] Retrieve the project data package from local storage, and determine the data to be repaired and the valid data cached in historical time based on the project data package;

[0007] Based on the cached data access records at historical moments, determine the invalid access rate of the data to be repaired, and determine the valid access rate of the valid data;

[0008] The repair gain value at the current moment is determined based on the invalid access rate and the valid access rate;

[0009] The control ratio of the data to be repaired at the current moment is determined based on the effective access rate and the repair gain value.

[0010] Based on the project data package, the data to be repaired is repaired according to the control ratio of the data to be repaired.

[0011] Accordingly, embodiments of this application provide a data processing apparatus, including:

[0012] The acquisition unit is used to acquire project data packets in local storage and determine the data to be repaired and the valid data cached in historical time based on the project data packets;

[0013] The first determining unit is configured to determine the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in historical time periods.

[0014] The second determining unit is used to determine the repair gain value at the current moment based on the invalid access rate and the valid access rate;

[0015] The third determining unit is used to determine the control ratio of the data to be repaired at the current moment based on the effective access rate and the repair gain value.

[0016] The processing unit is used to perform repair processing on the data to be repaired according to the project data package and the control ratio of the data to be repaired.

[0017] In some embodiments, the acquiring unit is further configured to:

[0018] Get the update time corresponding to each item's data packet in local storage;

[0019] The update time is used to determine the data to be repaired and the valid data cached in the historical time.

[0020] In some embodiments, the acquiring unit is further configured to:

[0021] Obtain the cache update time of the target cache data corresponding to the project data package;

[0022] Based on the cache update time and the corresponding update time, determine the update sequence between the target cache data and the corresponding project data package;

[0023] Based on each update sequence, the cached data to be repaired and the valid data at historical moments are determined.

[0024] In some embodiments, the acquiring unit is further configured to:

[0025] Each target cached data in the cache is compared with the corresponding project data package in the local storage to obtain the comparison result for each target cached data.

[0026] Based on the comparison results, the cached data to be repaired and the valid data at historical moments are determined.

[0027] In some implementations, the acquiring unit is further configured to:

[0028] If the comparison result shows that the target cache data is the same as the corresponding project data package, then the target cache data that is the same as the corresponding project data package is determined as the valid data cached at a historical time; and

[0029] If the comparison result shows that the target cache data is different from the corresponding project data package, then the target cache data that is different from the corresponding project data package is determined as data to be confirmed, and the cache data to be repaired at a historical time is determined based on the data to be confirmed.

[0030] In some implementations, the acquiring unit is further configured to:

[0031] The time difference of the corresponding project data packet at the historical moment is determined based on the update time.

[0032] Obtain the preset update validity period corresponding to the cache, and compare the time difference with the preset update validity period;

[0033] If the time difference is greater than or equal to the preset update time, then the sub-data to be confirmed in the data to be confirmed that corresponds to the data package of the item with a time difference greater than or equal to the preset update time is determined to be invalid data.

[0034] If the time difference is less than the preset update time, then obtain the first control ratio of the data to be repaired corresponding to the previous moment of the historical moment, and obtain the ratio of the data to be repaired corresponding to the historical moment.

[0035] If the ratio of the data to be repaired is less than the first control ratio, then the sub-data of the data to be confirmed whose time difference is less than the preset update time is determined to be invalid data.

[0036] The invalid data is identified as cached data to be repaired at a historical time.

[0037] In some implementations, the first determining unit is further configured to:

[0038] Retrieve the cached data access records at historical moments;

[0039] Identify the total number of accesses in the data access records, the first number of accesses to the data to be repaired, and the second number of accesses to the valid data;

[0040] Based on the ratio of the first number of visits to the total number of visits, the invalid access rate of the data to be repaired is determined; and

[0041] The effective access rate of the valid data is determined based on the ratio of the second number of visits to the total number of visits.

[0042] In some implementations, the second determining unit is further configured to:

[0043] Obtain the access rate difference between the invalid access rate and the valid access rate;

[0044] Obtain the preset data repair time, and determine the repair gain value at the current moment based on the access rate difference and the preset data repair time.

[0045] In some embodiments, the data processing apparatus further includes a fourth determining unit, configured to:

[0046] The target access rate difference at the current moment is determined based on the invalid access rate and the valid access rate.

[0047] Compare the target access rate difference with the access rate difference corresponding to the historical time.

[0048] If the target access rate difference is less than the access rate difference corresponding to the historical time, then obtain the historical control ratio of the cached data to be repaired at the historical time.

[0049] If the historical control ratio is less than the control ratio of the data to be repaired at the current time, then the historical control ratio is determined as the target control ratio of the data to be repaired at the current time.

[0050] The processing unit is further configured to perform repair processing on the data to be repaired according to the target control ratio based on the project data packet.

[0051] Furthermore, this application also provides a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the data processing method provided in this application.

[0052] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the data processing methods provided in embodiments of this application.

[0053] Furthermore, embodiments of this application also provide a computer program product, including computer instructions, which, when executed, implement the steps in any data processing method provided in embodiments of this application.

[0054] This application embodiment can obtain project data packets in local storage, and determine the data to be repaired and valid data cached in historical time based on the project data packets; determine the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in historical time; determine the repair gain value at the current time based on the invalid access rate and the valid access rate; determine the control ratio of the data to be repaired at the current time based on the valid access rate and the repair gain value; and perform repair processing on the data to be repaired according to the control ratio of the data to be repaired based on the project data packets. Therefore, this solution can determine the data to be repaired and the valid data cached locally at historical moments based on the project data packages already published in the local storage system, and determine the invalid access rate and valid access rate at historical moments. Then, based on the valid and invalid access rates at historical moments, the repair gain value at the current moment is estimated. Finally, based on the valid access rate and the repair gain value, a control ratio value for the data to be repaired in the cache is determined. Based on this control ratio value, the data to be repaired in the local cache is repaired. This achieves updating and repairing the data to be repaired in the cache according to the control ratio value, avoiding updating a large amount of data in the cache at the same time, reducing the system's operating load, effectively avoiding overload, and ensuring system performance. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of this application;

[0057] Figure 2 This is a flowchart illustrating the steps of the data processing method provided in the embodiments of this application;

[0058] Figure 3 This is a flowchart illustrating another step of the data processing method provided in the embodiments of this application;

[0059] Figure 4 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0060] Figure 5 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] This application provides a data processing method, apparatus, device, storage medium, and computer program product. Specifically, this application will describe the data processing apparatus from the perspective of a data processing device, which can be integrated into a computer device, such as a server or a terminal. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, vehicle terminal, smart voice interaction device, aircraft, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0063] The solutions provided in this application involve data processing technologies in scenarios such as intelligent transportation and vehicle networking backends. Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively integrates advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing, strengthening the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment, and saves energy. Specific examples are provided below:

[0064] For example, see Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of this application. The scenario includes a terminal or a server.

[0065] The terminal or server can obtain the update time corresponding to each item data packet in local storage, and determine the data to be repaired and the valid data cached in the historical time based on the update time; determine the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in the historical time; determine the repair gain value at the current time based on the invalid access rate and the valid access rate; determine the control ratio of the data to be repaired at the current time based on the valid access rate and the repair gain value; and perform repair processing on the data to be repaired according to the control ratio of the data to be repaired based on the item data packet.

[0066] Data processing may include obtaining the data update time, distinguishing between data to be repaired and valid data in the cache, determining the invalid access rate and the valid access rate, determining the data repair gain value, determining the control ratio of the data to be repaired, and repair processing methods.

[0067] The following sections provide detailed descriptions of each example. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0068] In this embodiment, the description will be from the perspective of a data processing device, which can be specifically integrated into a computer device such as a terminal or server. See also Figure 2 , Figure 2 This is a flowchart illustrating the steps of a data processing method provided in an embodiment of this application. Taking an example where the data processing device is specifically integrated on a server, the specific process when the processor on the server executes the program corresponding to the information processing method is as follows:

[0069] 101. Obtain the project data package in local storage, and determine the data to be repaired and the valid data cached in historical time based on the project data package.

[0070] In this context, local storage refers to the local storage system or storage space within a computer device, used to store relevant data, data packets, etc. It should be noted that the local storage system can include multiple project data packets, each containing one or more business data corresponding to a specific service. This business data is published or uploaded to the local storage system for subsequent retrieval. For example, taking a server of a business platform providing business information query services as an example, this server has a local storage system or space. This local storage is used to store relevant business data acquired, such as data received from uploaded services or data retrieved through web searches. This local storage data supports the business platform's business information query service, allowing users or other personnel to query business information.

[0071] It should be noted that the computer device may also include a cache. Taking a server as an example, the cache can be a cache for the backend system. When the server is running or implementing other data functions, it stores frequently used objects (such as data, files, etc.) in the cache so that the server can quickly call them when needed later, avoiding loading data or creating duplicate instances and improving the server's data utilization efficiency.

[0072] This project data package can be a data package containing directly usable business data, stored in the local storage system for subsequent access. It should be noted that each project data package can contain business data corresponding to multiple services. For example, taking a business query service as an example, its corresponding project data package can contain multiple project business data, each supporting a corresponding business information query service. For instance, a project business data could specifically be business text data that can be queried, such as a "list of mobile phone models of a certain brand," "World Cup schedule," or "uplink and downlink information of a communication network." Furthermore, each project data package can also be a data package containing business data corresponding to a single service. For instance, one project data package might contain project business data corresponding to a "list of mobile phone models of a certain brand," while another project data package might contain project business data corresponding to a "World Cup schedule."

[0073] To determine the status of data in local storage, such as whether data has been updated and the update time, embodiments of this application can obtain all project data packets in local storage, or detect or query all project data packets in local storage to determine relevant information about that project data packet, such as data information and update information. For example, taking the server of a business platform providing business information query services as an example, the project data packets in the local storage system of the server can be obtained to learn about relevant information about that project data packet.

[0074] The historical moment can be a specific moment within a historical period, such as the moment corresponding to the previous second or the previous minute. There is no specific limitation here, and it depends on the actual situation.

[0075] The data to be repaired can be cached data that needs to be repaired or modified. Specifically, when a project's business data or project data package in local storage is modified or updated, in order to ensure that the corresponding business data in the local cache can be quickly accessed and responded to in the future, the modified or updated project's business data or project data package in local storage needs to be reloaded / updated to the cache to replace the corresponding business data in the cache, thereby repairing the cache. The business data to be repaired or replaced in the cache is the aforementioned data to be repaired.

[0076] Specifically, the valid data can be data in the cache that does not require repair or modification and can remain in the cache for subsequent retrieval. In particular, the valid data is data in the cache at a historical point in time, and this valid data is consistent with the corresponding project business data or project data package in local storage at that historical point in time; that is, the original data corresponding to this valid data in local storage has not been updated or modified.

[0077] To determine which data in the cache needs to be updated / modified, and which data is valid and does not require modification / updating, this embodiment of the application, after obtaining the project data package in local storage, can determine the data to be repaired and the valid data in the cache at a historical time based on the relevant information of the project data package. For example, it can determine whether the data information in local storage is consistent with the corresponding cached data; or, it can determine whether the update information corresponding to each project data package in local storage is consistent with the corresponding cached data in the cache. Specifically, it compares the update information corresponding to each project data package with the last load time / repair time / update time of the corresponding cached data in the cache to determine whether the project data package in local storage is consistent with the corresponding cached data in the cache.

[0078] In some implementations, the step "determining the data to be repaired and the valid data cached in historical time based on the project data package" may include:

[0079] (1) Obtain the update time corresponding to each item data packet in local storage;

[0080] (2) Determine the data to be repaired and the valid data cached in the historical time based on the update time.

[0081] The update time can be the last update time of the corresponding project data package. Specifically, the update time can be the time when the corresponding project data package in local storage was last modified and updated.

[0082] Specifically, in order to cache historical data to be repaired and valid data, this application embodiment can identify the update time corresponding to each item data packet in local storage, and determine whether the corresponding cached data in the cache needs to be updated based on the update time corresponding to each item data packet, thereby determining the historical data to be repaired and valid data cached.

[0083] In some implementations, the step "determining the data to be repaired and the valid data cached in the historical time based on the update time" may include:

[0084] (2.1) Obtain the cache update time of the target cache data corresponding to the project data package;

[0085] (2.2) Determine the update sequence between the target cache data and the corresponding project data package based on the cache update time and the corresponding update time;

[0086] (2.3) Based on each update sequence, determine the data to be repaired and the valid data cached in the historical time.

[0087] The target cache data can be related cache data in the local cache, and this target cache data corresponds to the related project data package in the local storage. It should be noted that even if the project data package in the local storage is updated or modified, causing it to differ from the related target cache data loaded into the cache at a historical time, the modified project data package still corresponds to the related target cache data in the cache.

[0088] The cache update time can be the last time the relevant cached data was loaded / updated into the cache or the last time it was updated.

[0089] The update sequence can be the chronological order of update times. In this application, the update sequence refers to the update order between the project data packet in local storage and its corresponding target cache data in the cache; this update sequence can be used to determine whether the relevant cache data in the cache needs to be updated. Specifically, when the project data packet in local storage is redistributed or updated, the update time is determined by the latest update time parameter, and the cache update time of the corresponding target cache data in the local cache is determined by the last load / update time. Thus, the update time of the project data packet is compared with the cache update time of the corresponding target cache data to determine the update sequence between the two, so as to determine whether the relevant cache data in the cache needs to be updated, thereby distinguishing between data to be repaired and valid data.

[0090] To determine the valid data and the data to be repaired in the cache, embodiments of this application can determine this based on the update sequence between the project data packets in local storage and their corresponding target cache data in the cache. Specifically, after obtaining the update time corresponding to each project data packet in local storage, the cache update time of the target cache data corresponding to each project data packet in the cache is obtained; based on the update time of each project data packet and the cache update time of the corresponding target cache data, the update sequence between the project data packets in local storage and the corresponding target cache data in the cache is determined; then, the data to be repaired and the valid data cached at historical moments are determined based on the update sequence.

[0091] The step "(2.3) Determine the data to be repaired and the valid data cached in the historical time frame according to each update sequence" may include:

[0092] Each update sequence is identified, and the identification result corresponding to each update sequence is obtained. On the one hand, if the identification result is that the cache update time is after the corresponding update time, the target cache data whose cache update time is after the corresponding update time is determined as valid data cached in the historical time. On the other hand, if the identification result is that the cache update time is before the corresponding update time, the target cache data whose cache update time is before the corresponding update time is determined as data to be confirmed, and the data to be repaired cached in the historical time is determined based on the data to be confirmed.

[0093] Understandably, when the update time of the project data packet is earlier than the update time of the corresponding target cache data in the cache, the target cache data in the cache is modified and updated based on the earlier updated project data packet in the local storage. In this case, since the project data packet in the local storage has not been changed after the target cache data in the cache is updated, it means that the project data packet in the local storage is consistent with the target cache data in the cache. Therefore, the target cache data in the cache does not need to be updated again, and the target cache data is valid data. Conversely, when the update time of the project data package is later than the update time of the corresponding target cache data in the cache, it indicates that the relevant project data package in the local storage has been redistributed, modified, and updated, while the corresponding target cache data in the current cache has not been reloaded, repaired, or replaced according to the updated project data package in the local storage. That is, there is an inconsistency between the project data package in the local storage and the corresponding target cache data in the cache. In this case, the corresponding target cache data in the cache needs to be reloaded and updated. At this time, the target cache data can be directly identified as data to be repaired, or it can be identified as data to be confirmed, so as to further confirm the data to be repaired that needs to be updated and modified in a timely manner in historical time from the data to be confirmed.

[0094] Furthermore, in this embodiment of the application, in order to determine the data to be repaired and the valid data in the cache at a historical moment, the target cache data in the cache can be compared with the data of the corresponding project data package in the local storage to determine the valid data and the data to be repaired in the cache at a historical moment.

[0095] In some implementations, the step "determining the data to be repaired and the valid data cached in historical time based on the project data package" may further include:

[0096] (a) Compare each target cache data in the cache with the corresponding item data package in the local storage to obtain the comparison result for each target cache data;

[0097] (b) Based on the comparison results, determine the data to be repaired and the valid data cached in the historical time.

[0098] Specifically, the target cache data in the cache is compared with the data in the corresponding project data package in local storage to determine whether the target cache data is consistent with the data in the corresponding project data package. When the comparison result shows that the data is consistent, the target cache data in the cache at a historical time is determined to be valid, and the target cache data is considered valid data. When the comparison result shows that the data is inconsistent, the target cache data in the cache at a historical time is determined to be unusable and needs to be repaired, and the target cache data in this case is considered data to be repaired.

[0099] In some implementations, the step "determining the data to be repaired and the valid data cached in historical time based on the comparison results" may include:

[0100] (b.1) If the comparison result shows that the target cached data is the same as the corresponding project data package, then the target cached data that is the same as the corresponding project data package is determined as valid data cached at a historical time; and (b.2) If the comparison result shows that the target cached data is different from the corresponding project data package, then the target cached data that is different from the corresponding project data package is determined as data to be confirmed, and the data to be repaired cached at a historical time is determined based on the data to be confirmed.

[0101] It should be noted that, in order to control the amount of data repair / update in the local cache at the same time, this embodiment of the application, after determining the target cache data that needs to be updated, designates this target cache data as data to be confirmed, so as to further confirm the data to be repaired that needs to be updated and modified in a timely manner at historical time from the data to be confirmed. Thus, the update amount of related cache data in the cache is controlled, avoiding computer devices (such as servers) from updating a large amount of data in the cache at the same time, and ensuring the performance of computer devices (such as servers).

[0102] In some implementations, after determining the target cache data that needs to be updated in the cache, this target cache data is designated as data to be confirmed, and data to be repaired is determined based on this data. Specifically, the step "determining the data to be repaired cached at a historical time based on the data to be confirmed" may include:

[0103] The time difference of the corresponding project data package in the historical time is determined based on the update time; the preset update validity period corresponding to the cache is obtained, and the time difference is compared with the preset update validity period; if the time difference is greater than or equal to the preset update validity period, the sub-data to be confirmed corresponding to the project data package with a time difference greater than or equal to the preset update validity period in the data to be confirmed is determined as invalid data; if the time difference is less than the preset update validity period, the first control ratio of the data to be repaired corresponding to the previous time in the historical time is obtained, as well as the ratio of the data to be repaired corresponding to the historical time; if the ratio of the data to be repaired is less than the first control ratio, the sub-data to be confirmed corresponding to the time difference less than the preset update validity period in the data to be confirmed is determined as invalid data; the invalid data is determined as the data to be repaired cached in the historical time.

[0104] The preset update validity period can be the time limit for the cache to update data packets that are sent to local storage for updates or modifications. For example, after a related project data packet is redeployed or updated in local storage, the project data packet needs to be loaded into the cache within a certain period of time to replace the original target cache data in the cache. This is the preset update validity period required by the cache when the project data packet that has been modified in local storage is updated.

[0105] For example, after obtaining the update time of each item data packet in local storage during the last update or republishing, the time difference at a historical moment can be determined based on the update time of each item data packet, and the time difference of each item data packet can be compared with the preset update time.

[0106] On the one hand, when the time difference is greater than or equal to the preset update time limit, it means that the project data packet has exceeded the time limit for loading into the local cache, and the corresponding target cache data in the local cache has expired. The expired data is determined to be invalid data, and the corresponding invalid data in the cache needs to be repaired and updated in a timely manner according to the project data packet.

[0107] On the other hand, for project data packets that have been updated or modified in local storage, if the corresponding time difference is less than the preset update timeout, it indicates that the corresponding target cache data in the local cache needs to be updated according to the project data packet, but the corresponding target cache data in the local cache has not expired. At this time, in order to control the amount of data repair or data loading when repairing cache-related cache data, it can be determined whether to repair the cache data in the cache that needs to be updated but has not expired based on the cache miss rate at that moment.

[0108] The decision to repair cached data that needs updating but has not yet expired is based on the cache miss rate at that moment. Specifically, firstly, a first control ratio of the data to be repaired corresponding to the previous moment in the historical time is obtained; then, the ratio of the data to be repaired corresponding to the historical time is obtained. The process of obtaining the ratio of the data to be repaired can be as follows: obtain the amount of data in the cache that has been judged as invalid data, that is, the amount of cached data that has expired in the historical time; determine the cache miss rate of the historical time, i.e., the ratio of the data to be repaired, based on the amount of expired cached data and the total amount of cached data in the cache; next, compare the ratio of the data to be repaired with the first control ratio. When the ratio of the data to be repaired is less than the first control ratio, it indicates that the ratio of the data to be repaired at that moment is low, and the sub-data that needs updating but has not yet expired can be appropriately determined as invalid data; otherwise, the sub-data that needs updating but has not yet expired is determined as valid data. Furthermore, in this embodiment, all invalid data is determined as the data to be repaired at the historical time. It should be noted that the above is only an example. When identifying invalid data at any given time, you can refer to the above example to determine whether to repair the cached data that has not expired.

[0109] By using the above methods, we can identify the data to be repaired and the valid data in the cache at historical moments, so as to determine the access rate of the data to be repaired and the valid data in the future, and control the cache repair based on the access rate.

[0110] 102. Based on the data access records cached in historical time periods, determine the invalid access rate of the data to be repaired, and the valid access rate of the valid data.

[0111] The data access record can be a record of cached data accessed at a historical time in the cache. The data access record can include the total amount of data accessed at a historical time, the number of times valid data was accessed, and the number of times data to be repaired was accessed.

[0112] It's important to note that the access count and total access volume mentioned above refer to the sum of accesses to the corresponding data in the cache, and are unrelated to the total amount of data in the cache. In other words, not all cached data will be accessed at the same time. For example, the cache contains 30 data items: 10 valid data items and 20 data items awaiting repair. During access, the total number of accesses to the valid and unrepaired data items is 15, with the valid data items accessed 5 times and the unrepaired data items accessed 10 times. It's worth noting that the same data can be accessed multiple times at the same time. Taking a platform providing business query services as an example, the same data in the server cache may be accessed by multiple users at the same time, meaning it may be requested and called multiple times.

[0113] In order to determine the effective access rate and invalid access volume in the cache at a historical time, this application embodiment, after determining the data to be repaired and the effective data in the cache at a historical time, can determine each access rate based on the total access volume of cached data, the access volume of data to be repaired, and the access volume of effective data at a historical time.

[0114] Specifically, the steps "determine the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in historical time periods" may include: obtaining the data access records cached in historical time periods; identifying the total number of accesses, the first number of accesses of the data to be repaired, and the second number of accesses of the valid data in the data access records; determining the invalid access rate of the data to be repaired based on the ratio of the first number of accesses to the total number of accesses; and determining the valid access rate of the valid data based on the ratio of the second number of accesses to the total number of accesses.

[0115] By using the above methods, the effective access rate of cached data at a historical time and the invalid access rate of cached data to be repaired at a historical time can be determined based on the cached data access records at a historical time. This allows for the determination of the cached data repair rate at the current time based on the effective and invalid access rates.

[0116] 103. Determine the repair gain value at the current moment based on the invalid access rate and the valid access rate.

[0117] The repair gain value can be the repair increment or repair increment ratio of the data to be repaired, based on the effective access rate of the previous time step. It is used to control the repair progress of the data to be repaired in the cache at the current time step. The repair gain value can reflect the data repair rate to determine whether it is necessary to increase the amount of data to be repaired. For example, in seconds, the repair gain value can be the amount of data repaired per second.

[0118] In order to determine the repair increment for the data to be repaired in the cache at the current moment, this embodiment of the application, after obtaining the invalid access rate and valid access rate at historical moments, can determine the repair gain value at the current moment based on the invalid access rate and valid access rate, so as to subsequently determine the control ratio value of the data to be repaired based on the repair gain value.

[0119] In some implementations, the step "determine the repair gain value at the current moment based on the invalid access rate and the valid access rate" may include: obtaining the access rate difference between the invalid access rate and the valid access rate; obtaining a preset data repair time, and determining the repair gain value at the current moment based on the access rate difference and the preset data repair time.

[0120] The access rate difference can be the difference between the effective access ratios that need to be controlled at the current time or in the future. That is, the amount of cached data that needs to be repaired at the current time or in the future is determined based on the access rate difference. In addition, the access rate difference can reflect the difference in the access rate of effective data at the current time.

[0121] The preset data repair time can be the repair duration of the data to be repaired, or it can be a pre-set repair time. Furthermore, it can be determined based on the invalid access rate or access rate difference at historical times. For example, the time required to increase the effective access rate to the expected ratio can be determined / estimated based on the access rate difference of the data to be repaired accessed at historical times, or the time required to decrease the invalid access rate to the expected ratio can be determined / estimated.

[0122] In this embodiment of the application, after obtaining the invalid access rate and valid access rate at a historical time, the access rate difference at the current time can be determined based on the invalid access rate and valid access rate. Then, the ratio between the access rate difference and the preset data repair time is calculated to obtain the repair gain value cached at the current time.

[0123] Using the above methods, the repair gain of the data to be repaired at the current moment can be determined based on the effective access rate and invalid access rate at historical moments, so that the control ratio of the data to be repaired can be determined subsequently based on the repair gain.

[0124] 104. Determine the control ratio of the data to be repaired at the current moment based on the effective access rate and the repair gain value.

[0125] The control ratio is the target ratio of the data to be repaired in the cache at the current moment, which reflects the proportion of the data to be repaired in the cache after the repair is completed at the current moment.

[0126] In order to control the repair progress of the data to be repaired in the cache, this embodiment of the application needs to determine the ratio to be controlled for the data to be repaired at the current moment. Specifically, after obtaining the repair gain value at the current moment, the effective access rate and the repair gain value can be summed to determine the control ratio of the data to be repaired cached at the current moment, so as to control the repair progress of the data to be repaired in the cache according to the control ratio.

[0127] By using the above methods, the control ratio of the data to be repaired cached at the current moment can be determined. This allows for the control of the repair progress of the data to be repaired in the cache based on the control ratio, thereby controlling the amount of data to be repaired in the cache and avoiding reloading or updating the data to be repaired in the cache at the same time. At the same time, it can prevent the ratio of data to be repaired in the cache from growing too fast, effectively controlling the amount of data to be repaired in the cache and reducing the pressure on the system to repair the data to be repaired in the cache in the later stages.

[0128] 105. Based on the project data package, repair the data to be repaired according to the control ratio of the data to be repaired.

[0129] Specifically, after obtaining the control ratio of the data to be repaired at the current moment, the relevant project data packets in the local cache are used to repair the data to be repaired in the cache until the control ratio of the data to be repaired is reached, thus completing the cache data repair task at the current moment.

[0130] It should be noted that when repairing the data to be repaired in the cache, each target cached data item is treated as a unit, and the amount of repair at each moment can be determined according to the control ratio, specifically, it can be one or more units of target cached data.

[0131] Furthermore, in this embodiment, after the control ratio of the data to be repaired rises to a certain level, since the data to be repaired reaches a certain repair rate and no project data packets are published or modified in the local storage within a certain period of time, the effective access rate of the data to be repaired in the cache will continue to rise after one or more repair cycles. This continues until multiple project data packets in the local storage are distributed and modified in batches, which can easily lead to a sudden surge in the amount of data to be repaired in the cache, thereby reducing the effective access rate. To avoid this phenomenon, this embodiment can control the rise of the control ratio of the data to be repaired in advance, continuously repairing the data to be repaired at this optimal control ratio, thus avoiding increasing the repair rate of the cached data when project data packets in the local storage are published or modified in batches, and ensuring the effective access rate of the cached data.

[0132] To determine the optimal control ratio for the data to be repaired in the cache, and to continuously control the amount of data to be repaired in the cache based on this optimal control ratio after it is reached, thereby alleviating the pressure on the system to repair cached data, the step "determine the control ratio for the data to be repaired at the current moment based on the effective access rate and the repair gain value" may include:

[0133] The target access rate difference for the current time is determined based on the invalid access rate and the valid access rate. This target access rate difference is then compared with the access rate differences corresponding to historical times. If the target access rate difference is less than the historical access rate difference, the historical control ratio of the data to be repaired cached in the historical time is obtained. If the historical control ratio is less than the control ratio of the data to be repaired at the current time, the historical control ratio is determined as the target control ratio of the data to be repaired at the current time. Further, after obtaining the new target control ratio, step 105, "Repairing the data to be repaired according to the control ratio of the data to be repaired based on the project data package," may include: repairing the data to be repaired according to the target control ratio based on the project data package.

[0134] Specifically, the target access rate difference for the current moment is determined based on the invalid access rate and the valid access rate. This target access rate difference is then compared with historical access rate differences to determine the difference in the ratio of data to be repaired that needs to be controlled between the current and historical moments. Understandably, if the ratio of data to be repaired that needs to be controlled at the current moment is less than the ratio at a historical moment, it indicates that the repair progress of the cached data is being effectively controlled. In this case, the historical control ratio of the cached data to be repaired at a historical moment can be obtained and compared with the control ratio at the current moment. The smaller control ratio is selected as the target control ratio for the current moment, and the repair of the cached data to be repaired is then performed based on the target control ratio. This effectively regulates the final target control ratio of the data to be repaired, effectively controls the amount of data to be repaired, and effectively avoids excessively high cache miss rates (i.e., a large amount of cached data being repaired at the same time), which would increase the system's operational load and ensure system performance.

[0135] As can be seen from the above, the embodiments of this application can obtain project data packets in local storage, and determine the data to be repaired and the valid data cached in historical time based on the project data packets; determine the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in historical time; determine the repair gain value at the current time based on the invalid access rate and the valid access rate; determine the control ratio of the data to be repaired at the current time based on the valid access rate and the repair gain value; and perform repair processing on the data to be repaired according to the control ratio of the data to be repaired based on the project data packets. Therefore, this solution can determine the data to be repaired and the valid data cached locally at historical moments based on the project data packages already published in the local storage system, and determine the invalid access rate and valid access rate at historical moments. Then, based on the valid and invalid access rates at historical moments, the repair gain value at the current moment is estimated. Finally, based on the valid access rate and the repair gain value, a control ratio value for the data to be repaired in the cache is determined. Based on this control ratio value, the data to be repaired in the local cache is repaired. This achieves updating and repairing the data to be repaired in the cache according to the control ratio value, avoiding updating a large amount of data in the cache at the same time, reducing the system's operating load, effectively avoiding overload, and ensuring system performance.

[0136] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0137] This application takes data processing as an example to further describe the data processing method provided in this application embodiment.

[0138] Figure 3 This is a schematic flowchart of another step of the data processing method provided in the embodiments of this application. For ease of understanding, the embodiments of this application are combined with... Figure 3 Describe it.

[0139] In this embodiment, the description will be from the perspective of a data processing device, which can be integrated into a computer device such as a terminal and / or a server. For example, taking integration into a terminal or server as an example, when the processor on the terminal or server executes the program corresponding to the data processing method, the specific flow of the data processing method is as follows:

[0140] 201. Obtain the update time corresponding to each item data packet in local storage.

[0141] In this context, local storage refers to the local storage system or storage space within a computer device. It can store multiple project data packages, each containing one or more business data items corresponding to a specific service. This business data is published or uploaded to the local storage system for subsequent access. For example, consider a server on a business platform providing business information query services. This server has a local storage system or space used to store relevant acquired business data, enabling the business data in the local storage to support the business platform's business information query service for users or other personnel to access business information.

[0142] The update time can be the last update time of the corresponding project data package. Specifically, the update time can be the time when the corresponding project data package in local storage was last modified and updated.

[0143] 202. Determine the data to be repaired and the valid data cached in the historical time based on the update time.

[0144] This cache can be a cache for the backend system. Taking a server as a computer device as an example, when the server is running or implementing other data functions, it stores objects (such as data, files, etc.) that the program or system frequently uses in the cache so that the server can quickly call them when it uses them later, avoiding loading data or creating duplicate instances, and improving the server's data utilization efficiency.

[0145] The historical moment can be a specific moment within a historical period, such as the moment corresponding to the previous second or the previous minute. There is no specific limitation here, and it depends on the actual situation.

[0146] The data to be repaired can be cached data that needs to be repaired or modified. For example, when project business data or project data packages in local storage are modified or updated, the corresponding cached data needs to be updated or replaced. The cached data that needs to be updated or replaced is the aforementioned data to be repaired. It should be noted that this data to be repaired is an invalid cache, defined as a "cache miss," meaning the cache is missing, and this data needs to be reloaded and stored in the cache.

[0147] The valid data can be data in the cache that does not require repair or modification. This valid data belongs to a strongly effective cache, meaning that for a specific key-value pair, once the data is cached, no new modifications occur. In this embodiment, this strongly effective cache is defined as a "cache hit," meaning that the data can be directly reused.

[0148] In addition, the embodiment of the present application further includes a weakly effective cache. For a specific key-value, after the data is cached in the cache, there are new modifications, but the interval from the first modification time after caching to the current time is shorter than the update timeliness requirement (preset update timeliness). Specifically, for the project data packet that has been updated or modified in the local storage, when the corresponding time difference is less than the preset update timeliness, it indicates that the target cache data corresponding in the local cache needs to be updated according to the project data packet, but the target cache data corresponding in the local cache has not expired. In order to control the amount of data repair or data loading when repairing cache-related cache data, it can be determined whether to repair the weakly effective cache data according to the cache miss rate at this moment, that is, to determine whether it is the data to be repaired at this moment. Exemplarily, for the weakly effective cache, it is judged whether the weakly effective cache data needs to be defined as "cache miss" according to the cache update rule; wherein, the cache update rule is specifically as follows: if the current cache miss rate < the cache miss rate control target, it is judged as cache miss; otherwise, it is judged as cache hit. That is, when the ratio of invalid data in the cache is less than the control ratio of the data to be repaired, the weakly effective cache data and the invalid data are used as the data to be repaired together; otherwise, only the invalid data is used as the data to be repaired.

[0149] In order to determine the valid data and the data to be repaired in the cache, the embodiment of the present application can be determined according to the update time sequence between the project data packet in the local storage and the corresponding target cache data in the cache. Specifically, after obtaining the update time corresponding to each project data packet in the local storage, the cache update time of the target cache data corresponding to each project data packet in the cache is obtained; according to the update time of each project data packet and the cache update time of the corresponding target cache data, the update time sequence between the project data packet in the local storage and the corresponding target cache data in the cache is determined; furthermore, the data to be repaired and the valid data in the cache at the historical moment are determined according to the update time sequence.

[0150] 203. Obtain the data access record of the cache at the historical moment, and identify the total access volume, the first access volume of the data to be repaired, and the second access volume of the valid data in the data access record.

[0151] Among them, the data access record can be the record of the cache data accessed in the cache at the historical moment, and the data access record can include the total data access volume, the accessed volume of the valid data, and the accessed volume of the data to be repaired at the historical moment.

[0152] It's important to note that the access count and total access count mentioned above refer to the sum of accesses to the corresponding data in the cache, and are unrelated to the total amount of data in the cache. That is, not all cached data will be accessed at the same time. For example, the cache contains 30 data items: 10 valid items and 20 items awaiting repair. During access, the total number of accesses to the valid and repairable data is 15. Specifically, the valid data is accessed 5 times, and the repairable data is accessed 10 times. Therefore, the first access count is 10, the second access count is 5, and the total access count is 15.

[0153] 204. Determine the invalid access rate of the data to be repaired based on the ratio of the first number of visits to the total number of visits, and determine the valid access rate of the valid data based on the ratio of the second number of visits to the total number of visits.

[0154] For example, the invalid access rate of cached historical data to be repaired is as follows:

[0155] Invalid access rate = (Number of invalid cache accesses + Number of weak valid cache accesses) / Total number of accesses.

[0156] The effective access rate of cached historical data awaiting repair is as follows:

[0157] Effective access rate = number of accesses with "cache hit" / total number of accesses.

[0158] 205. Determine the repair gain value at the current moment based on the invalid access rate and the valid access rate.

[0159] The repair gain value can be the repair increment or repair increment ratio of the data to be repaired, based on the effective access rate of the previous time step. It is used to control the repair progress of the data to be repaired in the cache at the current time step. The repair gain value can reflect the data repair rate to determine whether it is necessary to increase the amount of data to be repaired. For example, in seconds, the repair gain value can be the amount of data repaired per second.

[0160] Specifically, the difference between the invalid access rate and the valid access rate is obtained; the preset data repair time is obtained, and the repair gain value at the current moment is determined based on the access rate difference and the preset data repair time.

[0161] The access rate difference can be the difference between the effective access ratios that need to be controlled at the current time or in the future. That is, the amount of cached data that needs to be repaired at the current time or in the future is determined based on the access rate difference. In addition, the access rate difference can reflect the difference in the access rate of effective data at the current time.

[0162] The preset data repair time can be determined based on the invalid access rate or access rate difference at historical times. For example, it can be determined / estimated based on the access rate difference of the data to be repaired accessed at historical times to determine / estimate the time required to increase the effective access rate to the expected ratio, or to determine / estimate the time required to decrease the invalid access rate to the expected ratio. For example, the data repair time can be determined as follows: Time required when the expected effective access rate equals the invalid access rate.

[0163] For example, the repair gain value is calculated as follows: Repair gain value = (Invalid access rate - Valid access rate) / Preset data repair time.

[0164] 206. Determine the control ratio of the data to be repaired at the current moment based on the effective access rate and the repair gain value.

[0165] The control ratio is the target ratio of the data to be repaired in the cache at the current moment, which reflects the proportion of the data to be repaired in the cache after the repair is completed at the current moment.

[0166] In order to control the repair progress of the data to be repaired in the cache, this embodiment of the application needs to determine the ratio to be controlled for the data to be repaired at the current moment. Specifically, after obtaining the repair gain value at the current moment, the effective access rate and the repair gain value can be summed to determine the control ratio of the data to be repaired cached at the current moment, so as to control the repair progress of the data to be repaired in the cache according to the control ratio.

[0167] For example, the control ratio of the data to be repaired is calculated as follows:

[0168] The control ratio of the data to be repaired = effective access rate + (invalid access rate - effective access rate) / preset data repair time.

[0169] 207. Obtain the historical control ratio cached at a historical time, and select the target control ratio from the historical control ratio and the control ratio of the data to be repaired at the current time.

[0170] To determine the optimal control ratio for the data to be repaired in the cache, once the optimal control ratio is reached, the amount of data to be repaired in the cache can be continuously controlled based on this optimal control ratio to reduce the pressure on the system to repair cached data.

[0171] Specifically, the target access rate difference for the current moment is obtained, which is calculated as invalid access rate minus valid access rate. The access rate difference for historical moments is also obtained, and the target access rate difference is compared with the historical access rate differences to determine the gap between the ratio of data to be controlled and repaired at the current moment and at historical moments. Furthermore, if the ratio of data to be controlled and repaired at the current moment is less than the ratio at historical moments, the smaller control ratio is selected from the historical control ratio and the current control ratio as the target control ratio for the current moment.

[0172] The algorithm for determining the target control ratio is as follows: when the control ratio of the data to be repaired at the current moment is lower than the historical control ratio at the previous moment, the target control ratio = min(historical control ratio at the previous moment, control ratio at the current moment).

[0173] By using the above methods, the control ratio of the data to be repaired cached at the current moment can be determined. This allows for the control of the repair progress of the data to be repaired in the cache based on the control ratio, thereby controlling the amount of data to be repaired in the cache and avoiding reloading or updating the data to be repaired in the cache at the same time. At the same time, it can prevent the ratio of data to be repaired in the cache from growing too fast, effectively controlling the amount of data to be repaired in the cache and reducing the pressure on the system to repair the data to be repaired in the cache in the later stages.

[0174] 208. Based on the project data package, perform repair processing on the data to be repaired according to the target control ratio.

[0175] Specifically, after obtaining the target control ratio of the data to be repaired at the current moment, the relevant project data packets in the local cache are used to repair the data to be repaired in the cache until the target control ratio of the data to be repaired is reached, thus completing the cache data repair task at the current moment.

[0176] The above embodiments can effectively reduce the maximum value of cached data missing rate, thereby reducing the performance impact of the system when repairing cache missing data and improving the system's throughput.

[0177] As can be seen from the above, the embodiments of this application can obtain project data packets in local storage, and determine the data to be repaired and the valid data cached in historical time based on the project data packets; determine the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in historical time; determine the repair gain value at the current time based on the invalid access rate and the valid access rate; determine the control ratio of the data to be repaired at the current time based on the valid access rate and the repair gain value; and perform repair processing on the data to be repaired according to the control ratio of the data to be repaired based on the project data packets. Therefore, this solution can determine the data to be repaired and the valid data cached locally at historical moments based on the project data packages already published in the local storage system, and determine the invalid access rate and valid access rate at historical moments. Then, based on the valid and invalid access rates at historical moments, the repair gain value at the current moment is estimated. Finally, based on the valid access rate and the repair gain value, a control ratio value for the data to be repaired in the cache is determined. Based on this control ratio value, the data to be repaired in the local cache is repaired. This achieves updating and repairing the data to be repaired in the cache according to the control ratio value, avoiding updating a large amount of data in the cache at the same time, reducing the system's operating load, effectively avoiding overload, and ensuring system performance.

[0178] To better implement the above methods, this application also provides a data processing device that can be integrated into a network device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0179] For example, such as Figure 4 As shown, the data processing device may include an acquisition unit 401, a first determination unit 402, a second determination unit 403, a third determination unit 404, and a processing unit 405.

[0180] The acquisition unit 401 is used to acquire project data packets in local storage and determine the data to be repaired and the valid data cached in historical time based on the project data packets;

[0181] The first determining unit 402 is used to determine the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in historical time periods.

[0182] The second determining unit 403 is used to determine the repair gain value at the current moment based on the invalid access rate and the valid access rate;

[0183] The third determining unit 404 is used to determine the control ratio of the data to be repaired at the current moment based on the effective access rate and the repair gain value.

[0184] The processing unit 405 is used to perform repair processing on the data to be repaired according to the control ratio of the data to be repaired, based on the project data package.

[0185] In some embodiments, the obtaining unit 401 is further configured to:

[0186] Retrieve the update time corresponding to each item data packet in local storage; determine the data to be repaired and the valid data cached in historical time based on the update time.

[0187] In some embodiments, the obtaining unit 401 is further configured to:

[0188] Obtain the cache update time of the target cached data corresponding to the project data package; determine the update sequence between the target cached data and the corresponding project data package based on the cache update time and the corresponding update time; determine the data to be repaired and the valid data cached at historical moments based on each update sequence.

[0189] In some embodiments, the obtaining unit 401 is further configured to:

[0190] Each target cached data in the cache is compared with the corresponding project data package in local storage to obtain the comparison result for each target cached data; based on the comparison result, the data to be repaired and the valid data in the cache at a historical time are determined.

[0191] In some implementations, the acquisition unit 401 is further configured to:

[0192] If the comparison result shows that the target cached data is the same as the corresponding project data package, then the target cached data that is the same as the corresponding project data package is determined as valid data cached in the historical time. If the comparison result shows that the target cached data is different from the corresponding project data package, then the target cached data that is different from the corresponding project data package is determined as data to be confirmed, and the data to be repaired cached in the historical time is determined based on the data to be confirmed.

[0193] In some implementations, the acquisition unit 401 is further configured to:

[0194] The time difference of the corresponding project data package in the historical time is determined based on the update time; the preset update validity period corresponding to the cache is obtained, and the time difference is compared with the preset update validity period; if the time difference is greater than or equal to the preset update validity period, the sub-data to be confirmed corresponding to the project data package with a time difference greater than or equal to the preset update validity period in the data to be confirmed is determined as invalid data; if the time difference is less than the preset update validity period, the first control ratio of the data to be repaired corresponding to the previous time in the historical time is obtained, as well as the ratio of the data to be repaired corresponding to the historical time; if the ratio of the data to be repaired is less than the first control ratio, the sub-data to be confirmed corresponding to the time difference less than the preset update validity period in the data to be confirmed is determined as invalid data; the invalid data is determined as the data to be repaired cached in the historical time.

[0195] In some implementations, the acquisition unit 401 is further configured to:

[0196] Each target cached data in the cache is compared with the corresponding project data package in local storage to obtain the comparison result for each target cached data; based on the comparison result, the data to be repaired and the valid data in the cache at a historical time are determined.

[0197] In some implementations, the acquisition unit 401 is further configured to:

[0198] If the comparison result shows that the target cached data is the same as the corresponding project data package, then the target cached data that is the same as the corresponding project data package is determined as valid data cached in the historical time. If the comparison result shows that the target cached data is different from the corresponding project data package, then the target cached data that is different from the corresponding project data package is determined as data to be confirmed, and the data to be repaired cached in the historical time is determined based on the data to be confirmed.

[0199] In some implementations, the first determining unit 402 is further configured to:

[0200] Retrieve data access records cached in historical time periods; identify the total number of accesses, the first number of accesses to the data to be repaired, and the second number of accesses to the valid data in the data access records; determine the invalid access rate of the data to be repaired based on the ratio of the first number of accesses to the total number of accesses; and determine the valid access rate of the valid data based on the ratio of the second number of accesses to the total number of accesses.

[0201] In some implementations, the second determining unit 403 is further configured to:

[0202] Obtain the difference between the invalid access rate and the valid access rate; obtain the preset data repair time, and determine the repair gain value at the current moment based on the access rate difference and the preset data repair time.

[0203] In some embodiments, the data processing apparatus further includes a fourth determining unit, specifically used for:

[0204] The target access rate difference at the current moment is determined based on the invalid access rate and the valid access rate; the target access rate difference is compared with the access rate difference at the corresponding historical moment; if the target access rate difference is less than the access rate difference at the corresponding historical moment, the historical control ratio of the data to be repaired cached at the historical moment is obtained; if the historical control ratio is less than the control ratio of the data to be repaired at the current moment, the historical control ratio is determined as the target control ratio of the data to be repaired at the current moment.

[0205] The processing unit 405 is also used to perform repair processing on the data to be repaired according to the target control ratio based on the project data packet.

[0206] As can be seen from the above, the embodiments of this application can obtain project data packets in local storage through the acquisition unit 401, and determine the data to be repaired and the valid data cached in historical time based on the project data packets; the first determination unit 402 determines the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in historical time based on the first determination unit 402; the second determination unit 403 determines the repair gain value at the current time based on the invalid access rate and the valid access rate based on the second determination unit 403; the third determination unit 404 determines the control ratio of the data to be repaired at the current time based on the valid access rate and the repair gain value based on the third determination unit 404; and the processing unit 405 performs repair processing on the data to be repaired according to the control ratio of the data to be repaired based on the project data packets. Therefore, this solution can determine the data to be repaired and the valid data cached locally at historical moments based on the project data packages already published in the local storage system, and determine the invalid access rate and valid access rate at historical moments. Then, based on the valid and invalid access rates at historical moments, the repair gain value at the current moment is estimated. Finally, based on the valid access rate and the repair gain value, a control ratio value for the data to be repaired in the cache is determined. Based on this control ratio value, the data to be repaired in the local cache is repaired. This achieves updating and repairing the data to be repaired in the cache according to the control ratio value, avoiding updating a large amount of data in the cache at the same time, reducing the system's operating load, effectively avoiding overload, and ensuring system performance.

[0207] This application also provides a computer device, such as... Figure 5 As shown, it illustrates a structural schematic diagram of the computer device involved in the embodiments of this application, specifically:

[0208] The computer device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that... Figure 5The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0209] The processor 501 is the control center of the computer device, connecting various parts of the computer device through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 502, and by calling data stored in the memory 502. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

[0210] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0211] The computer equipment also includes a power supply 503 that supplies power to the various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0212] The computer device may also include an input unit 504, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0213] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in the embodiments of this application, the processor 501 in the computer device loads the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 runs the application programs stored in the memory 502 to realize various functions, as follows:

[0214] Retrieve the project data package from local storage, and determine the data to be repaired and the valid data cached in historical time based on the project data package; determine the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in historical time; determine the repair gain value at the current time based on the invalid access rate and the valid access rate; determine the control ratio of the data to be repaired at the current time based on the valid access rate and the repair gain value; and perform repair processing on the data to be repaired according to the control ratio of the data to be repaired based on the project data package.

[0215] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0216] Therefore, this solution can determine the data to be repaired and the valid data cached locally at historical moments based on the project data packages already published in the local storage system, and determine the invalid access rate and valid access rate at historical moments. Then, based on the valid and invalid access rates at historical moments, the repair gain value at the current moment is estimated. Finally, based on the valid access rate and the repair gain value, a control ratio value for the data to be repaired in the cache is determined. Based on this control ratio value, the data to be repaired in the local cache is repaired. This achieves updating and repairing the data to be repaired in the cache according to the control ratio value, avoiding updating a large amount of data in the cache at the same time, reducing the system's operating load, effectively avoiding overload, and ensuring system performance.

[0217] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0218] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the data processing methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0219] Retrieve the project data package from local storage, and determine the data to be repaired and the valid data cached in historical time based on the project data package; determine the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in historical time; determine the repair gain value at the current time based on the invalid access rate and the valid access rate; determine the control ratio of the data to be repaired at the current time based on the valid access rate and the repair gain value; and perform repair processing on the data to be repaired according to the control ratio of the data to be repaired based on the project data package.

[0220] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0221] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0222] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0223] The foregoing has provided a detailed description of a data processing method, apparatus, device, storage medium, and computer program product provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, include: Retrieve the project data package from local storage, and determine the data to be repaired and the valid data cached in historical time based on the project data package; Based on the cached data access records at historical moments, determine the invalid access rate of the data to be repaired, and determine the valid access rate of the valid data; Determining the repair gain value at the current moment based on the invalid access rate and the valid access rate includes: obtaining the access rate difference between the invalid access rate and the valid access rate; obtaining a preset data repair time, and determining the repair gain value at the current moment based on the access rate difference and the preset data repair time; wherein, the repair gain value is the repair increment or repair increment ratio of the data to be repaired under the premise of the valid access rate at the previous moment. The effective access rate and the repair gain value are summed to determine the control ratio of the data to be repaired at the current moment; wherein, the control ratio of the data to be repaired is the target ratio of the data to be repaired in the cache to be controlled at the current moment. Based on the project data package, the data to be repaired is repaired according to the control ratio of the data to be repaired.

2. The method according to claim 1, characterized in that, The step of determining the data to be repaired and the valid data cached in historical time based on the project data package includes: Identify the update time corresponding to the project data packet; The data to be repaired and the valid data cached at historical moments are determined based on the update time.

3. The method according to claim 2, characterized in that, The step of determining the data to be repaired and the valid data cached in historical time based on the update time includes: Obtain the cache update time of the target cache data corresponding to the project data package; Based on the cache update time and the corresponding update time, determine the update sequence between the target cache data and the corresponding project data package; Based on each update sequence, the cached data to be repaired and the valid data at historical moments are determined.

4. The method according to claim 3, characterized in that, The step of determining the cached data to be repaired and the valid data at each update time based on the historical time sequence includes: Each update sequence is identified to obtain the identification result corresponding to each update sequence; If the identification result is that the cache update time is after the corresponding update time, then the target cache data whose cache update time is after the corresponding update time is determined as the valid data of the cache at the historical time. If the identification result indicates that the cache update time is before the corresponding update time, then the target cache data whose cache update time is before the corresponding update time is determined as data to be confirmed, and the cache data to be repaired at a historical time is determined based on the data to be confirmed.

5. The method according to claim 4, characterized in that, The step of determining the cached data to be repaired at a historical time based on the data to be confirmed includes: The time difference of the corresponding project data packet at the historical moment is determined based on the update time. Obtain the preset update validity period corresponding to the cache, and compare the time difference with the preset update validity period; If the time difference is greater than or equal to the preset update time, then the sub-data to be confirmed in the data to be confirmed that corresponds to the data package of the item with a time difference greater than or equal to the preset update time is determined to be invalid data. If the time difference is less than the preset update time, then obtain the first control ratio of the data to be repaired corresponding to the previous moment of the historical moment, and obtain the ratio of the data to be repaired corresponding to the historical moment. If the ratio of the data to be repaired is less than the first control ratio, then the sub-data of the data to be confirmed whose time difference is less than the preset update time is determined to be invalid data. The invalid data is identified as cached data to be repaired at a historical time.

6. The method according to claim 1, characterized in that, The step of determining the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the cached data access records at historical times includes: Retrieve the cached data access records at historical moments; Identify the total number of accesses in the data access records, the first number of accesses to the data to be repaired, and the second number of accesses to the valid data; Based on the ratio of the first number of visits to the total number of visits, the invalid access rate of the data to be repaired is determined; and The effective access rate of the valid data is determined based on the ratio of the second number of visits to the total number of visits.

7. The method according to claim 1, characterized in that, After determining the control ratio of the data to be repaired at the current moment by summing the effective access rate and the repair gain value, the method further includes: The target access rate difference at the current moment is determined based on the invalid access rate and the valid access rate. Compare the target access rate difference with the access rate difference corresponding to the historical time. If the target access rate difference is less than the access rate difference corresponding to the historical time, then obtain the historical control ratio of the cached data to be repaired at the historical time. If the historical control ratio is less than the control ratio of the data to be repaired at the current moment, then the historical control ratio is determined as the target control ratio of the data to be repaired at the current moment, and the data to be repaired is repaired according to the target control ratio based on the project data package.

8. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire project data packets in local storage and determine the data to be repaired and the valid data cached in historical time based on the project data packets; The first determining unit is configured to determine the invalid access rate of the data to be repaired and the valid access rate of the valid data based on the data access records cached in historical time periods. The second determining unit is used to determine the repair gain value at the current moment based on the invalid access rate and the valid access rate, including: obtaining the access rate difference between the invalid access rate and the valid access rate; obtaining a preset data repair time, and determining the repair gain value at the current moment based on the access rate difference and the preset data repair time; wherein, the repair gain value is the repair increment or repair increment ratio of the data to be repaired under the premise of the valid access rate at the previous moment. The third determining unit is used to sum the effective access rate and the repair gain value to determine the control ratio of the data to be repaired at the current time; wherein, the control ratio of the data to be repaired is the target ratio of the data to be repaired in the cache to be controlled at the current time. The processing unit is used to perform repair processing on the data to be repaired according to the project data package and the control ratio of the data to be repaired.

9. The apparatus according to claim 8, characterized in that, The acquisition unit is further configured to: Get the update time corresponding to each item's data packet in local storage; The update time is used to determine the data to be repaired and the valid data cached in the historical time.

10. The apparatus according to claim 9, characterized in that, The acquisition unit is further configured to: Obtain the cache update time of the target cache data corresponding to the project data package; Based on the cache update time and the corresponding update time, determine the update sequence between the target cache data and the corresponding project data package; Based on each update sequence, the cached data to be repaired and the valid data at historical moments are determined.

11. The apparatus according to claim 9, characterized in that, The acquisition unit is further configured to: Each target cached data in the cache is compared with the corresponding project data package in the local storage to obtain the comparison result for each target cached data. Based on the comparison results, the cached data to be repaired and the valid data at historical moments are determined.

12. The apparatus according to claim 11, characterized in that, The acquisition unit is further configured to: If the comparison result shows that the target cache data is the same as the corresponding project data packet, then the target cache data that is the same as the corresponding project data packet is determined as the valid data cached at a historical time. as well as If the comparison result shows that the target cache data is different from the corresponding project data package, then the target cache data that is different from the corresponding project data package is determined as data to be confirmed, and the cache data to be repaired at a historical time is determined based on the data to be confirmed.

13. The apparatus according to claim 12, characterized in that, The acquisition unit is further configured to: The time difference of the corresponding project data packet at the historical moment is determined based on the update time. Obtain the preset update validity period corresponding to the cache, and compare the time difference with the preset update validity period; If the time difference is greater than or equal to the preset update time, then the sub-data to be confirmed in the data to be confirmed that corresponds to the data package of the item with a time difference greater than or equal to the preset update time is determined to be invalid data. If the time difference is less than the preset update time, then obtain the first control ratio of the data to be repaired corresponding to the previous moment of the historical moment, and obtain the ratio of the data to be repaired corresponding to the historical moment. If the ratio of the data to be repaired is less than the first control ratio, then the sub-data of the data to be confirmed whose time difference is less than the preset update time is determined to be invalid data. The invalid data is identified as cached data to be repaired at a historical time.

14. The apparatus according to claim 8, characterized in that, The first determining unit is further configured to: Retrieve the cached data access records at historical moments; Identify the total number of accesses in the data access records, the first number of accesses to the data to be repaired, and the second number of accesses to the valid data; Based on the ratio of the first number of visits to the total number of visits, the invalid access rate of the data to be repaired is determined; and The effective access rate of the valid data is determined based on the ratio of the second number of visits to the total number of visits.

15. The apparatus according to claim 8, characterized in that, The data processing device further includes a fourth determining unit, used for: The target access rate difference at the current moment is determined based on the invalid access rate and the valid access rate. Compare the target access rate difference with the access rate difference corresponding to the historical time. If the target access rate difference is less than the access rate difference corresponding to the historical time, then obtain the historical control ratio of the cached data to be repaired at the historical time. If the historical control ratio is less than the control ratio of the data to be repaired at the current time, then the historical control ratio is determined as the target control ratio of the data to be repaired at the current time. The processing unit is further configured to perform repair processing on the data to be repaired according to the target control ratio based on the project data packet.

16. A computer device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor being used to run the computer program in the memory to implement the steps of the data processing method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the data processing method according to any one of claims 1 to 7.

18. A computer program product, characterized in that, Includes computer instructions, which, when executed, implement the steps in the data processing method according to any one of claims 1 to 7.

Images