Resource data processing method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of data processing, and discloses a resource data processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: after incremental data sent by a resource data source system is received, adding the incremental data into a corresponding queue; updating the target resource data in the total data cache list according to the to-be-processed incremental data in the queue; if the accumulation condition of the to-be-processed incremental data in the queue meets a target condition, pushing the updated total data cache list to a resource data publishing system for publishing processing when the updating frequency of the total data cache list reaches a first frequency; if the accumulation condition does not meet the target condition, pushing the updated total data cache list to a resource data publishing system for publishing processing when the updating frequency of the total data cache list reaches a second frequency; the second number is smaller than the first number. By adopting the method, the requirement on the network bandwidth can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a resource data processing method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] Due to the needs of resource processing business, a resource data source system can send resource data of various resources to a front-end application connected to a resource data publishing system, and the front-end application connected to the resource data publishing system can push the received resource data of various resources to the resource data publishing system. The resource data publishing system can publish the received resource data of various resources, thereby providing relevant data support for users related to resource processing business.

[0003] The front-end application connected to the resource data publishing system can access resource data from the resource data source system. Due to the differences in various resource data source systems and various resources, the front-end application connected to the resource data publishing system can adopt a full subscription mode. In the full subscription mode, when the resource data of any resource changes, the resource data source system can push full resource data to the front-end application connected to the resource data publishing system. This way can occupy a large amount of network bandwidth. SUMMARY

[0004] Therefore, it is necessary to provide a resource data processing method, device, computer equipment, computer readable storage medium and computer program product to solve the above technical problems.

[0005] In a first aspect, the present application provides a resource data processing method, comprising:

[0006] In response to receiving incremental data corresponding to a target data type of a target resource sent by a resource data source system, the incremental data is added to a queue corresponding to a resource type of the target resource; the queue is used to store to-be-processed incremental data;

[0007] In sequence, the target resource data in a full data cache list is updated according to the to-be-processed incremental data in the queue; the full data cache list is used to store resource data of each data type of each resource type of resource; the target resource data is resource data corresponding to the to-be-processed incremental data;

[0008] If the accumulation condition of the to-be-processed incremental data in the queue meets a target condition, the updated full data cache list is pushed to a resource data publishing system for publishing processing every time the updating frequency of the full data cache list reaches a first frequency.

[0009] If the accumulation of the to-be-processed incremental data in the queue does not satisfy the target condition, the updated full-data cache list is pushed to the resource data publishing system for publishing processing when the number of updates to the full-data cache list reaches a second number, which is less than the first number.

[0010] In one of the embodiments, the updating of the target resource data in the full-data cache list according to the to-be-processed incremental data in the queue comprises:

[0011] If the update type corresponding to the to-be-processed incremental data in the queue is a replacement type, it is determined whether the target resource data in the full-data cache list belongs to non-empty resource data.

[0012] If the target resource data belongs to non-empty resource data, the target resource data in the full-data cache list is replaced according to the to-be-processed incremental data.

[0013] If the target resource data belongs to empty resource data, it is determined whether the expired data list includes target expired resource data; the expired data list is used to store expired resource data of each data type of each resource type in the full-data cache list; the target expired resource data is the expired resource data corresponding to the to-be-processed incremental data.

[0014] If the expired data list includes the target expired resource data, the target expired resource data in the expired data list is restored to the target resource data in the full-data cache list, and the target resource data in the full-data cache list is replaced according to the to-be-processed incremental data.

[0015] If the expired data list does not include the target expired resource data, it is determined whether the update type corresponding to the to-be-processed incremental data is adjusted to an addition type according to whether the target data type corresponding to the to-be-processed incremental data includes each data type, and the target resource data in the full-data cache list is processed according to the adjusted update type.

[0016] In one of the embodiments, the determination of whether the update type corresponding to the to-be-processed incremental data is adjusted to an addition type according to whether the target data type corresponding to the to-be-processed incremental data includes each data type comprises:

[0017] If the target data type corresponding to the to-be-processed incremental data includes each data type, the update type corresponding to the to-be-processed incremental data is adjusted to an addition type.

[0018] If the target data type corresponding to the to-be-processed incremental data comprises a partial data type, the update type corresponding to the to-be-processed incremental data is not adjusted;

[0019] The target resource data in the full-data cache list is processed according to the adjusted update type.

[0020] If the update type corresponding to the to-be-processed incremental data is adjusted to an increase type, the to-be-processed incremental data is added to the target resource data in the full-data cache list.

[0021] If the update type corresponding to the to-be-processed incremental data is not adjusted, the target resource data in the full-data cache list is not replaced according to the to-be-processed incremental data.

[0022] In one embodiment, the target resource data in the full-data cache list is updated according to the to-be-processed incremental data in the queue, including:

[0023] If the to-be-processed incremental data in the queue comprises abnormal data, a target processing mode for the to-be-processed incremental data is determined according to a preset processing rule and other data in the to-be-processed incremental data except the abnormal data.

[0024] After the to-be-processed incremental data is processed according to the target processing mode, the target resource data in the full-data cache list is updated.

[0025] In one embodiment, after the number of updates of the full-data cache list reaches a first number of times, the updated full-data cache list is pushed to a resource data publishing system for publishing processing, and the method further comprises:

[0026] If the accumulation condition of the to-be-processed incremental data in the queue meets the target condition, a data accumulation alarm message is generated; the data accumulation alarm message is used to prompt a user to perform intervention processing.

[0027] In response to the intervention processing operation triggered by the user, a target queue is determined in a plurality of queues other than the queue; the target queue is a queue in which the accumulation condition of the to-be-processed incremental data does not meet the target condition.

[0028] The part of the processing thread corresponding to the target queue is determined as the processing thread corresponding to the queue; the processing thread is used to update the target resource data in the full-data cache list according to the to-be-processed incremental data in the corresponding queue.

[0029] In one of the embodiments, the pushing the updated full data cache list to the resource data publishing system for publishing processing comprises:

[0030] If the resource data of one of the resources in the updated full data cache list is missing, determining the associated data of the missing data in the resource data;

[0031] After removing the associated data in the updated full data cache list, pushing the updated full data cache list to the resource data publishing system; the updated full data cache list is used for the resource data publishing system to display the missing data and the display area corresponding to the associated data according to a preset display mode.

[0032] In a second aspect, the application further provides a resource data processing device, comprising:

[0033] A data distribution module is configured to add the incremental data corresponding to the target data type of the target resource sent by the resource data source system into a queue corresponding to the resource type of the target resource in response to receiving the incremental data; the queue is used for storing the to-be-processed incremental data;

[0034] An update processing module is configured to sequentially update the target resource data in a full data cache list according to the to-be-processed incremental data in the queue; the full data cache list is used for storing the resource data of each data type of each resource type; the target resource data is the resource data corresponding to the to-be-processed incremental data;

[0035] A pushing processing module is configured to push the updated full data cache list to the resource data publishing system for publishing processing when the number of updates of the full data cache list reaches a first number if the accumulation condition of the to-be-processed incremental data in the queue meets a target condition.

[0036] The pushing processing module is further configured to push the updated full data cache list to the resource data publishing system for publishing processing when the number of updates of the full data cache list reaches a second number if the accumulation condition of the to-be-processed incremental data in the queue does not meet the target condition; the second number is less than the first number.

[0037] In a third aspect, the application further provides a computer device comprising a memory and a processor; the memory stores a computer program; and the processor implements the following steps when executing the computer program:

[0038] In response to receiving the incremental data of the target data type of the target resource sent by the resource data source system, the incremental data is added to a queue corresponding to the resource type of the target resource; the queue is used to store the incremental data to be processed;

[0039] The target resource data in the full data cache list is updated in sequence according to the incremental data to be processed in the queue; the full data cache list is used to store resource data of each data type of resources of each resource type; the target resource data is resource data corresponding to the incremental data to be processed;

[0040] If the accumulation condition of the incremental data to be processed in the queue meets a target condition, the updated full data cache list is pushed to the resource data publishing system for publishing processing every time the updating times of the full data cache list reach a first number of times.

[0041] If the accumulation condition of the incremental data to be processed in the queue does not meet the target condition, the updated full data cache list is pushed to the resource data publishing system for publishing processing every time the updating times of the full data cache list reach a second number of times; the second number of times is less than the first number of times.

[0042] In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:

[0043] In response to receiving the incremental data of the target data type of the target resource sent by the resource data source system, the incremental data is added to a queue corresponding to the resource type of the target resource; the queue is used to store the incremental data to be processed;

[0044] The target resource data in the full data cache list is updated in sequence according to the incremental data to be processed in the queue; the full data cache list is used to store resource data of each data type of resources of each resource type; the target resource data is resource data corresponding to the incremental data to be processed;

[0045] If the accumulation condition of the incremental data to be processed in the queue meets a target condition, the updated full data cache list is pushed to the resource data publishing system for publishing processing every time the updating times of the full data cache list reach a first number of times.

[0046] If the accumulation condition of the incremental data to be processed in the queue does not meet the target condition, the updated full data cache list is pushed to the resource data publishing system for publishing processing every time the updating times of the full data cache list reach a second number of times; the second number of times is less than the first number of times.

[0047] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0048] In response to receiving incremental data corresponding to the target data type of the target resource sent by the resource data source system, the incremental data is added to a queue corresponding to the resource type of the target resource; the queue is used to store incremental data to be processed.

[0049] The target resource data in the full data cache list is updated sequentially based on the incremental data to be processed in the queue; the full data cache list is used to store resource data of each data type of resource of each resource type; the target resource data is the resource data corresponding to the incremental data to be processed.

[0050] If the accumulation of incremental data to be processed in the queue meets the target condition, then whenever the number of updates to the full data cache list reaches the first count, the updated full data cache list will be pushed to the resource data publishing system for publishing processing.

[0051] If the accumulation of incremental data to be processed in the queue does not meet the target condition, then whenever the number of updates to the full data cache list reaches the second count, the updated full data cache list will be pushed to the resource data publishing system for publishing processing; the second count is less than the first count.

[0052] The aforementioned resource data processing methods, apparatus, computer equipment, computer-readable storage media, and computer program products allow the external front-end application to adopt an incremental subscription model. In this model, if a resource's data of a certain data type changes, that resource can be called the target resource, and that data type can be called the target data type. The resource data source system can send the incremental data corresponding to the target data type of the target resource to the external front-end application. Other unchanged resource data does not need to be sent to the external front-end application, thus reducing the need for significant network bandwidth and lowering network bandwidth requirements. Furthermore, after receiving incremental data, the external front-end application can add it to a queue corresponding to the resource type of the target resource. This avoids interference between data processing of different resource types and allows for differentiated processing strategies for different queues (such as setting the number of processing threads and processing priorities for different queues), ensuring more timely processing of incremental data for critical resource types. The queue can be used to store pending incremental data. According to the algorithm, the target resource data in the full data cache list can be updated sequentially based on the incremental data to be processed in the queue. The full data cache list can be used to store resource data of various data types for each resource type, and the target resource data can be the resource data corresponding to the incremental data to be processed. If the backlog of incremental data to be processed in the queue meets the target condition, the updated full data cache list is pushed to the resource data publishing system for publishing processing whenever the number of updates to the full data cache list reaches the first count. If the backlog of incremental data to be processed in the queue does not meet the target condition, the updated full data cache list is pushed to the resource data publishing system for publishing processing whenever the number of updates to the full data cache list reaches the second count, and the second count is less than the first count. In this way, when the backlog of incremental data to be processed in the queue is serious, the number of times to push to the resource data publishing system can be reduced, the processing speed of the incremental data to be processed in the queue can be accelerated, and thus the timeliness of resource data publishing can be improved. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is an application environment diagram of a resource data processing method in one embodiment;

[0055] Figure 2This is a flowchart illustrating a resource data processing method in one embodiment;

[0056] Figure 3 This is a flowchart illustrating the processing of target resource data in one embodiment;

[0057] Figure 4 This is a flowchart illustrating a resource data processing method in another embodiment;

[0058] Figure 5 This is a flowchart illustrating the resource data processing method in yet another embodiment;

[0059] Figure 6 This is a structural block diagram of a resource data processing device in one embodiment;

[0060] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. It should be noted that existing industry solutions such as software, components, and models may be mentioned in the embodiments of this application. These should be considered exemplary and are intended only to illustrate the feasibility of implementing the technical solutions of this application, but do not imply that the applicant has already used or necessarily used such solutions.

[0062] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations. The acquisition, storage, use and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0063] The terms "first," "second," etc., used in this application may be used to describe various objects, but these objects are not limited by these terms. These terms are only used to distinguish the first object from the second object. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0064] The resource data processing method provided in this application embodiment can be applied to, for example... Figure 1The application environment is illustrated. The resource data source system can communicate with external front-end applications targeting the resource data publishing system via a network, and the external front-end applications can communicate with the resource data publishing system via a network. The resource data source system may include a data storage system, which stores the data that the resource data source system needs to process; the data storage system can be integrated into the resource data source system or located in the cloud or on a web server. The resource data publishing system may also include a data storage system, which stores the data that the resource data publishing system needs to process; the data storage system can be integrated into the resource data publishing system or located in the cloud or on a web server.

[0065] The resource data source system can send incremental data corresponding to the target data type of the target resource to the external front-end application. The external front-end application can add the incremental data to a queue corresponding to the resource type of the target resource; the queue is used to store incremental data to be processed. The external front-end application can sequentially update the target resource data in the full data cache list based on the incremental data to be processed in the queue; the full data cache list is used to store resource data of each data type for each resource type; the target resource data is the resource data corresponding to the incremental data to be processed. If the backlog of incremental data to be processed in the queue meets the target condition, the external front-end application can push the updated full data cache list to the resource data publishing system for publishing processing whenever the number of updates to the full data cache list reaches the first count. If the backlog of incremental data to be processed in the queue does not meet the target condition, the external front-end application can push the updated full data cache list to the resource data publishing system for publishing processing whenever the number of updates to the full data cache list reaches the second count; the second count is less than the first count. The external front-end application can be deployed on a computer device, which can be, but is not limited to, a personal computer. Resource data source systems can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide cloud computing services; resource data publishing systems can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide cloud computing services.

[0066] In one exemplary embodiment, such as Figure 2 As shown, a resource data processing method is provided, which can be applied to... Figure 1 The method may include the following steps:

[0067] Step S201: In response to receiving incremental data corresponding to the target data type of the target resource sent by the resource data source system, the incremental data is added to the queue corresponding to the resource type of the target resource.

[0068] The resource data source system can provide resource data for each resource type across various data types. In incremental subscription mode, the initial data sent by the resource data source system to the external front-end application can be the full set of resource data, which may include resource data for each resource type across various data types. Accordingly, the external front-end application can create a full data cache list based on the full set of resource data; the external front-end application can then push this full data cache list to the resource data publishing system. Upon receiving the full data cache list, the resource data publishing system can process it according to preset rules before publishing.

[0069] After the initial transmission, resource data can change. If the resource data of a certain resource changes in a certain data type, that resource can be called the target resource, and that data type can be called the target data type. Based on the changed resource data of the target resource's target data type, the resource data source system can determine the corresponding incremental data. For example, if the changed resource data of the target resource's target data type is 5, then the incremental data can include the information "5".

[0070] As an implementation method, after determining the incremental data corresponding to the target data type of the target resource, the resource data source system can send the incremental data to the external front-end application. The incremental data may have a corresponding resource number (which can be called a resource ID value), which can be used to uniquely identify the corresponding resource. The external front-end application can determine the target resource corresponding to the incremental data based on the resource number, thereby determining the queue corresponding to the resource type of the target resource, and adding the incremental data to that queue. The queue can be used to store incremental data to be processed.

[0071] Step S202: Update the target resource data in the full data cache list according to the incremental data to be processed in the queue.

[0072] The full data cache list is used to store resource data of various data types for each resource type; the target resource data is the resource data corresponding to the incremental data to be processed.

[0073] The queue can include multiple pending incremental data items corresponding to the resource type of the target resource. As an implementation, the external front-end application can sequentially retrieve multiple pending incremental data items from the queue. For each pending incremental data item retrieved, the external front-end application can perform verification and data protocol conversion processing. After processing, it can determine the target resource data corresponding to the pending incremental data item in the full data cache list. The target resource data can be the resource data of the target resource. After determining the target resource data, the external front-end application can update the target resource data in the full data cache list based on the pending incremental data. Updates can include, but are not limited to, adding, replacing, and deleting.

[0074] Step S203: If the accumulation of incremental data to be processed in the queue meets the target condition, then whenever the number of updates to the full data cache list reaches the first count, the updated full data cache list is pushed to the resource data publishing system for publishing processing.

[0075] Step S204: If the accumulation of incremental data to be processed in the queue does not meet the target condition, then whenever the number of updates to the full data cache list reaches the second count, the updated full data cache list is pushed to the resource data publishing system for publishing processing.

[0076] The second count is less than the first count. The external front-end application can also determine whether the accumulation of incremental data to be processed in the queue meets the target condition. As an implementation method, before retrieving a specific incremental data item from the queue, the external front-end application can determine the number of incremental data items to be processed in the queue. If the number is lower than a preset threshold, it can be determined that the accumulation of incremental data items to be processed in the queue does not meet the target condition. If the accumulation of incremental data items to be processed in the queue does not meet the target condition, the external front-end application can push the full data cache list obtained from the Nth update to the resource data publishing system every N updates to the full data cache list. If the number is higher than a preset threshold, it can be determined that the accumulation of incremental data items to be processed in the queue meets the target condition. If the accumulation of incremental data items to be processed in the queue meets the target condition, the external front-end application can push the full data cache list obtained from the Mth update to the resource data publishing system every M updates to the full data cache list.

[0077] Where M can be less than N, N can be a positive integer greater than or equal to 1, and M can be a positive integer greater than or equal to 2. For example, N can be 1 and M can be 10. As an example, let's take N = 1 and M = 10 as an example:

[0078] Before retrieving the first incremental data to be processed from the queue, the external front-end application can determine whether the accumulation of incremental data in the queue meets the target conditions. If the target conditions are not met, the external front-end application can retrieve the first incremental data to be processed from the queue and update the full data cache list based on the first incremental data. At this time, the update count of the full data cache list is 1. The external front-end application can push the full data cache list obtained from this update to the resource data publishing system. The external front-end application can also reset the update count of the full data cache list to 0.

[0079] Before retrieving the second incremental data to be processed from the queue, the external front-end application can determine whether the backlog of incremental data in the queue meets the target condition. If the target condition is not met, the external front-end application can retrieve the second incremental data to be processed from the queue and update the full data cache list based on the second incremental data. At this time, the update count of the full data cache list is 1. The external front-end application can push the full data cache list obtained from this update to the resource data publishing system. The external front-end application can also reset the update count of the full data cache list to 0.

[0080] Before retrieving the third incremental data to be processed from the queue, the external front-end application can determine whether the backlog of incremental data in the queue meets the target condition. If the target condition is met, the external front-end application can retrieve the third incremental data to be processed from the queue and update the full data cache list based on the third incremental data. At this time, the update count of the full data cache list is 1. For the fourth to thirteenth incremental data to be processed, the external front-end application does not need to determine whether the backlog of incremental data in the queue meets the target condition before retrieving the corresponding incremental data from the queue. For the fourth to thirteenth incremental data to be processed, the external front-end application can update the full data cache list after retrieving the corresponding incremental data from the queue, and the update count of the full data cache list is incremented by one, instead of being reset to 0. After retrieving the thirteenth incremental data to be processed from the queue, the external front-end application can update the full data cache list. At this time, the update count of the full data cache list is 13. The external front-end application can push the full data cache list obtained from this update to the resource data publishing system. The external front-end application can also reset the update count of the full data cache list to 0.

[0081] When the external application is about to retrieve the fourteenth incremental data to be processed from the queue, it can determine whether the accumulation of incremental data in the queue meets the target condition. If the target condition is not met, it can be processed in the same way as the first incremental data. If the target condition is met, it can be processed in the same way as the third incremental data.

[0082] Each time the resource data publishing system receives a full data cache list, it can process and publish it according to preset rules. There can be multiple resource data source systems; in this case, the full data cache list can carry resource data from multiple sources, and the resource data publishing system can process and publish this multi-source resource data according to preset rules.

[0083] In the above embodiments, the external front-end application can adopt an incremental subscription mode. In this mode, if a resource changes in a certain data type, the resource can be called the target resource, and the data type can be called the target data type. The resource data source system can send the incremental data corresponding to the target data type of the target resource to the external front-end application. Other unchanged resource data does not need to be sent to the external front-end application, thus avoiding the consumption of a large amount of network bandwidth and having lower network bandwidth requirements. Moreover, after receiving the incremental data, the external front-end application can add the incremental data to the queue corresponding to the resource type of the target resource. This can avoid interference between the processing of data of different resource types, and differentiated processing strategies can be formulated for different queues (such as setting the corresponding number of processing threads and processing priorities for different queues), allowing the incremental data of more critical resource types to be processed more promptly. The queue can be used to store incremental data to be processed, and the data can be processed sequentially according to the incremental data to be processed in the queue. The process updates the target resource data in the full data cache list. The full data cache list stores resource data of various data types for each resource type, and the target resource data corresponds to the incremental data to be processed. If the backlog of incremental data in the queue meets the target condition, the updated full data cache list is pushed to the resource data publishing system for publication whenever the number of updates to the full data cache list reaches the first count. If the backlog of incremental data in the queue does not meet the target condition, the updated full data cache list is pushed to the resource data publishing system for publication whenever the number of updates to the full data cache list reaches the second count (the second count being less than the first count). This reduces the number of pushes to the resource data publishing system when there is significant backlog of incremental data in the queue, speeding up the processing of incremental data and improving the timeliness of resource data publication.

[0084] In one exemplary embodiment, such as Figure 3 As shown, step S202, which updates the target resource data in the full data cache list based on the incremental data to be processed in the queue, may include the following steps:

[0085] Step S301: If the update type corresponding to the incremental data to be processed in the queue is a replacement type, then determine whether the target resource data in the full data cache list is non-empty resource data; Step S302: If the target resource data is non-empty resource data, then perform replacement processing on the target resource data in the full data cache list according to the incremental data to be processed; Step S303: If the target resource data is empty resource data, then determine whether the expired data list includes the target expired resource data; The expired data list is used to store the expired resource data of each data type of resource of each resource type in the full data cache list; The target expired resource data is the data corresponding to the incremental data to be processed. The expired resource data; Step S304, if the expired data list includes the target expired resource data, then restore the target expired resource data in the expired data list to the target resource data in the full data cache list, and replace the target resource data in the full data cache list according to the incremental data to be processed; Step S305, if the expired data list does not include the target expired resource data, then determine whether to adjust the update type corresponding to the incremental data to be processed to the addition type according to whether the target data type corresponding to the incremental data to be processed includes each data type, and process the target resource data in the full data cache list accordingly according to the adjusted update type.

[0086] As an implementation method, after retrieving a certain incremental data to be processed from the queue, the external front-end application can parse the incremental data to be processed, thereby determining the update type corresponding to the incremental data to be processed.

[0087] As an implementation method, if the update type is an increment type, the external front-end application can determine the target resource data in the full data cache list, and the external front-end application can add resource data that matches the incremental data to be processed to the target resource data in the full data cache list, thereby completing the increment processing.

[0088] As an implementation method, if the update type is deletion, the external front-end application can determine the target resource data in the full data cache list, and the external front-end application can delete the resource data that matches the incremental data to be processed from the target resource data in the full data cache list, thereby completing the deletion process.

[0089] As an implementation method, if the update type is replacement, the external front-end application can determine the target resource data in the full data cache list. The external front-end application can determine whether the target resource data is non-empty. If the target resource data is non-empty, it indicates that there is data to be replaced. In this case, the external front-end application can replace the target resource data in the full data cache list based on the incremental data to be processed. If the target resource data is empty, it indicates that there is no data to be replaced. In this case, the external front-end application can attempt to recover data based on the expired data list. The expired data list can be used to store expired resource data of each data type of resource in the full data cache list. An expiration threshold can be set for the expired resource data in the expired data list. The expiration threshold can be used to characterize the time that the corresponding expired resource data can be retained in the expired data list. As an implementation method, the external front-end application can determine whether the expired data list includes target expired resource data. The target expired resource data is the expired resource data of the target resource corresponding to the incremental data to be processed, and the target expired resource data corresponds to the incremental data to be processed.

[0090] If the expired data list includes the target expired resource data, the external front-end application can restore the target expired resource data in the expired data list to the target resource data in the full data cache list. At this time, the target resource data can be considered as non-empty resource data, which indicates that there is data that has been replaced. The external front-end application can replace the target resource data in the full data cache list according to the incremental data to be processed.

[0091] If the expired data list does not include the target expired resource data, it indicates that the corresponding resource data cannot be recovered from the expired data list. In this case, the external front-end application can determine whether to adjust the update type of the incremental data to be processed to the addition type based on whether the target data type corresponding to the incremental data to be processed includes each data type, and process the target resource data in the full data cache list accordingly based on the adjusted update type.

[0092] In this embodiment, if the update type corresponding to the incremental data to be processed is a replacement type, and the target resource data in the full data cache list is empty, the corresponding resource data can be restored from the expired data list first before replacement, thereby ensuring normal replacement and avoiding errors. If the target resource data in the full data cache list is empty and the expired data list does not include the target expired resource data, it can be determined whether to change the update type corresponding to the incremental data to be processed to an addition type based on whether the target data type corresponding to the incremental data to be processed includes each data type, thereby enabling appropriate processing. This avoids the deletion of potentially valid incremental data to be processed due to issues with the full data cache list and the expired data list, and prevents potentially valid incremental data to be processed from failing to be pushed to the resource data publishing system, thus ensuring the accuracy of the resource data.

[0093] In an exemplary embodiment, determining whether to adjust the update type of the incremental data to an increment type based on whether the target data type corresponding to the incremental data to be processed includes:

[0094] If the target data type corresponding to the incremental data to be processed includes every data type, then the update type corresponding to the incremental data to be processed will be adjusted to the addition type; if the target data type corresponding to the incremental data to be processed includes some data types, then the update type corresponding to the incremental data to be processed will not be adjusted.

[0095] Based on the adjusted update type, the target resource data in the full data cache list is processed accordingly, including:

[0096] If the update type corresponding to the incremental data to be processed is adjusted to the add type, then the incremental data to be processed will be added to the target resource data in the full data cache list; if the update type corresponding to the incremental data to be processed is not adjusted, then the target resource data in the full data cache list will not be replaced based on the incremental data to be processed.

[0097] As an implementation method, if the target data type corresponding to the incremental data to be processed includes every data type, it can be indicated that the validity of the incremental data to be processed is high. In this case, the external front-end application can change the update type corresponding to the incremental data to be processed to the addition type. As a result, the resource data corresponding to the incremental data to be processed can be added to the target resource data in the full data cache list, so that the resource data corresponding to the incremental data to be processed can be pushed to the resource data publishing system and avoid being accidentally deleted.

[0098] As an implementation method, if the target data type corresponding to the incremental data to be processed includes some data types, it can be indicated that the validity of the incremental data to be processed is low. In this case, the external front-end application may not change the update type corresponding to the incremental data to be processed to the addition type, but may keep the update type corresponding to the incremental data to be processed in the replacement type. In this case, the external front-end application may not replace the target resource data in the full data cache list according to the incremental data to be processed, and the external front-end application may also discard the incremental data to be processed.

[0099] In this embodiment, when the target data type corresponding to the incremental data to be processed includes every data type, the update type corresponding to the incremental data to be processed can be changed to an add type. This allows resource data corresponding to the incremental data to be processed to be added to the target resource data in the full data cache list, enabling the resource data corresponding to the incremental data to be processed to be pushed to the resource data publishing system and preventing accidental deletion. When the target data type corresponding to the incremental data to be processed includes only some data types, it indicates that the validity of the incremental data to be processed is low. In this case, the update type can be left unchanged, and the full data cache list can be left unreplaced, thus preventing the incremental data with low validity from affecting the accuracy of the data in the full data cache list.

[0100] In an exemplary embodiment, updating the target resource data in the full data cache list based on the incremental data to be processed in the queue in step S202 may include the following steps:

[0101] If the incremental data to be processed in the queue includes abnormal data, then the target processing method for the incremental data to be processed is determined according to the preset processing rules and other data in the incremental data to be processed excluding abnormal data; after processing the incremental data to be processed according to the target processing method, the target resource data in the full data cache list is updated.

[0102] As an implementation method, the incremental data to be processed includes resource data of the target resource after changes in the target data type. The resource data can have a normal numerical range corresponding to its data type. If the value of a certain changed resource data included in the incremental data to be processed is outside the normal numerical range, it indicates that the incremental data to be processed includes abnormal data. After determining that the incremental data to be processed includes abnormal data, the external front-end application can determine, based on preset processing rules and other data in the incremental data to be processed (excluding abnormal data), whether to discard the incremental data to be processed or discard only a portion of the data, thereby obtaining the target processing method for the incremental data to be processed. If the incremental data to be processed is discarded according to the target processing method, the target resource data in the full data cache list does not need to be updated based on the incremental data to be processed. If only a portion of the incremental data to be processed is discarded according to the target processing method, the target resource data in the full data cache list can be updated based on the remaining data in the incremental data to be processed.

[0103] When the incremental data to be processed includes abnormal data, this embodiment provides a corresponding abnormal handling mechanism. The processing method for the incremental data to be processed is determined according to the preset processing rules and in combination with other data in the incremental data to be processed, excluding abnormal data. This can avoid mishandling the incremental data to be processed, thereby more accurately affecting the full data cache list and improving the accuracy of resource data.

[0104] In an exemplary embodiment, after pushing the updated full data cache list to the resource data publishing system for publishing processing whenever the number of updates to the full data cache list reaches the first count, the method provided in this application further includes the following steps:

[0105] If the accumulation of incremental data to be processed in the queue meets the target condition, a data accumulation alarm message is generated. The data accumulation alarm message is used to prompt the user to intervene. In response to the user-triggered intervention, a target queue is determined from multiple queues other than the main queue. The target queue is the queue where the accumulation of incremental data to be processed does not meet the target condition. The processing threads corresponding to the target queue are determined as the processing threads corresponding to the queue. The processing threads are used to update the target resource data in the full data cache list based on the incremental data to be processed in the corresponding queue.

[0106] As an implementation method, for queues corresponding to various resource types, the external front-end application can configure corresponding processing threads for each queue. The processing threads can be used to update the target resource data in the full data cache list based on the incremental data to be processed in the corresponding queue.

[0107] As an implementation method, for a specific queue, after the external front-end application reduces the push frequency to the resource data publishing system according to step S203 above, the external front-end application can determine whether the accumulation of incremental data to be processed in the queue still meets the target condition. If so, it indicates that the data accumulation in the queue is relatively serious, and the mitigation effect of step S203 is limited. The external front-end application can generate a data accumulation alarm message. The data accumulation alarm message can be used to prompt the user to intervene. If the user confirms to intervene, the intervention operation can be triggered by a button or other means. In response to the user-triggered intervention operation, the external front-end application can determine the target queue from multiple queues other than the current queue. The accumulation of incremental data to be processed in the target queue may not meet the target condition, that is, the accumulation of incremental data to be processed in the target queue is mild or non-existent. The external front-end application can determine some of the processing threads configured for the target queue as the processing threads corresponding to this queue. As a result, the number of processing threads corresponding to this queue increases, and the external front-end application can start more processing threads to process the incremental data to be processed in this queue, thereby alleviating the data accumulation in this queue.

[0108] In one exemplary embodiment, pushing the updated full data cache list to the resource data publishing system for publishing processing includes:

[0109] If resource data for one of the resources in the updated full data cache list is missing, then the associated data for the missing data is determined in the resource data. After removing the associated data from the updated full data cache list, the updated full data cache list is pushed to the resource data publishing system. The updated full data cache list is used by the resource data publishing system to display the display areas corresponding to the missing data and associated data according to the preset display method.

[0110] As an implementation method, before pushing the updated full data cache list to the resource data publishing system for publishing, the external front-end application can determine whether there is any missing resource data in the updated full data cache list. If the resource data of one type of resource in the updated full data cache list is missing, it indicates that the information of that resource in a certain dimension may not be fully conveyed. The external front-end application can identify data related to the missing data in the resource data of that resource, thus obtaining the associated data of the missing data. The external front-end application can remove the associated data from the updated full data cache list and then push the updated full data cache list to the resource data publishing system. The updated full data cache list received by the resource data publishing system may not include the missing data and associated data. The resource data publishing system can determine the display area corresponding to the missing data and associated data and display the display area according to a preset display method, such as dimming the display area.

[0111] In this embodiment, if the resource data of one of the resources in the updated full data cache list is missing, the external front-end application can remove the associated data corresponding to the missing data from the updated full data cache list. This can prevent the associated data from being published incorrectly. Furthermore, the resource data publishing system displays the display areas corresponding to the missing data and associated data according to a preset display method, which can remind users of the abnormality of the data from a display perspective and reduce the possibility of misleading users.

[0112] In one exemplary embodiment, such as Figure 4 As shown, a resource data processing method is also provided, which may include the following steps:

[0113] Step S401: In response to receiving incremental data corresponding to the target data type of the target resource sent by the resource data source system, the incremental data is added to the queue corresponding to the resource type of the target resource. The queue is used to store incremental data to be processed.

[0114] Step S402: Update the target resource data in the full data cache list sequentially based on the incremental data to be processed in the queue. The full data cache list stores resource data of various data types for each resource type; the target resource data is the resource data corresponding to the incremental data to be processed.

[0115] If the incremental data to be processed in the queue includes abnormal data, the target processing method for the incremental data to be processed is determined based on the preset processing rules and the other data in the incremental data to be processed, excluding the abnormal data. If the incremental data to be processed is discarded according to the target processing method, the target resource data in the full data cache list can be updated without considering the incremental data to be processed, and the next incremental data to be processed in the queue can be retrieved and processed. If part of the incremental data to be processed is discarded according to the target processing method, the target resource data in the full data cache list can be updated based on the remaining data in the incremental data to be processed.

[0116] When updating the target resource data in the full data cache list based on the remaining data in the incremental data to be processed, the following steps may be included:

[0117] If the update type corresponding to the incremental data to be processed is a replacement type, then determine whether the target resource data in the full data cache list is non-empty resource data; if the target resource data is non-empty resource data, then replace the target resource data in the full data cache list according to the remaining data in the incremental data to be processed; if the target resource data is empty resource data, then determine whether the expired data list includes the target expired resource data; the expired data list is used to store the expired resource data of each data type of each resource type in the full data cache list; the target expired resource data is the expired resource data corresponding to the incremental data to be processed; if the expired data list includes the target expired resource data, then restore the target expired resource data in the expired data list to the target in the full data cache list. For target resource data, based on the remaining data in the pending incremental data, the target resource data in the full data cache list is replaced. If the expired data list does not include the target expired resource data, and the target data type corresponding to the pending incremental data includes every data type, then the update type corresponding to the pending incremental data is adjusted to the add type, and the resource data corresponding to the remaining data in the pending incremental data is added to the target resource data in the full data cache list. If the expired data list does not include the target expired resource data, and the target data type corresponding to the pending incremental data includes some data types, then the update type corresponding to the pending incremental data is not adjusted, and the target resource data in the full data cache list is not replaced based on the remaining data in the pending incremental data.

[0118] Step S403: When the accumulation of incremental data to be processed in the queue meets the target condition, whenever the number of updates to the full data cache list reaches the first count, if there is missing resource data for one of the resources in the updated full data cache list, then the associated data of the missing data is determined in the resource data; after removing the associated data from the updated full data cache list, the updated full data cache list is pushed to the resource data publishing system; the updated full data cache list is used by the resource data publishing system to display the display areas corresponding to the missing data and associated data according to the preset display method.

[0119] Step S404: If the accumulation of incremental data in the queue still meets the target condition, a data accumulation alarm message is generated. The data accumulation alarm message is used to prompt the user to intervene.

[0120] Step S405: In response to the user-triggered intervention operation, a target queue is determined from multiple queues other than the main queue. The target queue is the queue where the accumulation of incremental data to be processed does not meet the target conditions.

[0121] Step S406: The processing threads corresponding to the target queue are determined as the processing threads corresponding to the queue. These processing threads are used to update the target resource data in the full data cache list based on the incremental data to be processed in the corresponding queue.

[0122] Step S407: If the accumulation of incremental data to be processed in the queue does not meet the target condition, whenever the number of updates to the full data cache list reaches the second count, if resource data of one type of resource in the updated full data cache list is missing, then the associated data of the missing data is determined in the resource data; after removing the associated data from the updated full data cache list, the updated full data cache list is pushed to the resource data publishing system; the updated full data cache list is used by the resource data publishing system to display the display areas corresponding to the missing data and associated data according to the preset display method; the second count is less than the first count.

[0123] As an example, the above resource data processing method can be applied to the resource data processing of financial products. Financial products may include, but are not limited to, futures. Taking futures as an example, the data types involved in futures may include, but are not limited to, price, quantity, price levels, buy / sell direction, and validity period, with 5 or 10 price levels. In the incremental subscription mode provided in this embodiment, the first data obtained by the external front-end application from the resource data source system can be the full amount of resource data, and subsequent data can be incremental data, which may include resource data with changed data types. Figure 5 As shown,Figure 5 The process of an external front-end application from access to push is illustrated. Step S501: The external front-end application can receive incremental data sent by the resource data source system and can perform traffic distribution processing on the incremental data. Steps S502 and S503: Utilizing the thread pool of the access processor, the application can process incremental data for various futures products according to flow control rules, such as performing verification processing and data protocol conversion processing on the incremental data. Step S504: The external front-end application can utilize the thread pool of the sending processor to update the full data cache list according to flow control rules and based on the processed incremental data. Step S505: The external front-end application can push the updated full data cache list to the resource data publishing system. The data protocol conversion processing may include adjusting the data structure of the incremental data according to the corresponding data protocol; correspondingly, the data protocol conversion processing can also be referred to as data entity flattening processing.

[0124] In the access and distribution phase, after receiving multiple incremental data messages from the resource data source system, the external front-end application can add these messages to a blocking queue. The blocking queue can be used to store incremental data corresponding to various resource types. The external front-end application can then add the incremental data from the blocking queue to the queue of the corresponding resource type for subsequent verification and data protocol conversion.

[0125] This embodiment provides a full data cache list, which can be used to cache the latest resource data of various data types for each resource type. When applying this embodiment to futures data processing, futures resource data changes frequently and there are many types of futures. Database operations can be avoided; the processing and update process data can be written to a log, and the full data cache list can be stored in the cache.

[0126] When incremental data appears, the full data cache list can be updated. If the backlog of pending incremental data in the queue is not severe, the updated full data cache list can be pushed to the resource data publishing platform each time it is updated. Updates can include adding, replacing, or deleting. When an update includes deletion, the corresponding resource data can be removed from both the full data cache list and the expired data list.

[0127] When applying this embodiment to futures data processing, there are many types of futures contracts, and prices include standard term prices and forward contract calibration prices. Therefore, when futures prices fluctuate frequently, the data volume is large, and the processing capacity of the external front-end application is limited, easily leading to data backlog in the queue. The solution provided in this embodiment reduces the number of times the full data cache list is pushed to the resource data publishing platform when incremental data to be processed accumulates in the queue, thereby alleviating data backlog in the queue and accelerating data processing speed.

[0128] When applying this embodiment to futures data processing, the incremental data to be processed may include anomalous data, such as a price of 0 or abnormal data, or data that is 0 or abnormal. The external front-end application can determine, based on preset processing rules and other data in the incremental data to be processed (excluding anomalous data), whether to discard the incremental data to be processed or discard only a portion of it, thereby more accurately influencing the full data cache list.

[0129] When applying this embodiment to futures data processing, if expired prices appear in the full data cache list, the expired prices can be added to the expired data list. This process can be called the price cleanup process.

[0130] When applying this embodiment to futures data processing, this embodiment can also provide the following exception handling mechanism:

[0131] (1) Severe backlog: If the queue still has backlog after pushing according to step S403, the external front-end application can generate a data backlog alarm message. This data backlog alarm message can be output through the monitoring system so that relevant users can intervene in a timely manner.

[0132] (2) No corresponding record can be found during replacement: If the update type corresponding to the incremental data to be processed is the replacement type, and the target resource data in the full data cache list is empty, the corresponding resource data can be restored from the expired data list first and then replaced, so as to ensure that the replacement is carried out normally and avoid errors; If the target resource data in the full data cache list is empty and the expired data list does not include the target expired resource data, and the target data type corresponding to the incremental data to be processed includes each data type, the update type corresponding to the incremental data to be processed can be changed to the addition type, so that the resource data corresponding to the incremental data to be processed can be added to the target resource data in the full data cache list, so that the resource data corresponding to the incremental data to be processed can be pushed to the resource data publishing system and avoid being accidentally deleted.

[0133] (3) If the full data cache list has the following missing data: missing quantity of a certain grade, extra quantity of a certain grade, missing price of a certain grade, or multiple prices, the associated data of these missing data in the full data cache list can be removed before pushing (the associated data is, for example, the price or quantity of these grades); the resource data publishing system can display the missing data according to the corresponding display method, such as graying out the display area corresponding to the price or quantity of these grades.

[0134] In futures data processing applications, this embodiment reduces network load between the resource data source system and external front-end applications by incrementally receiving futures resource data, thereby lowering network bandwidth requirements and improving performance. This embodiment is message-parsing driven; after receiving messages from the resource data source system, the messages are parsed to obtain incremental data, which is then stored in a blocking queue. The incremental data in the blocking queue is then added to the queue for the corresponding futures type, and the pending incremental data in the queue is processed asynchronously. For situations where there is data backlog in the queue, this embodiment also provides corresponding processing mechanisms to improve data processing efficiency. This embodiment ensures the timeliness of resource data. It also provides a full data cache list, which can be updated based on the incremental data to be processed, ensuring the real-time nature and accuracy of resource data. When the incremental data to be processed includes abnormal data, this embodiment provides a corresponding processing mechanism. This mechanism determines the processing method for the incremental data to be processed based on preset processing rules and other data besides abnormal data, avoiding mishandling of the incremental data and thus more accurately influencing the full data cache list, improving the accuracy of resource data. Furthermore, this embodiment provides a corresponding display processing mechanism when data is missing from the full data cache list.

[0135] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0136] Based on the same inventive concept, this application also provides a resource data processing apparatus for implementing the resource data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more resource data processing apparatus embodiments provided below can be found in the limitations of the resource data processing method described above, and will not be repeated here.

[0137] In one exemplary embodiment, such as Figure 6 As shown, a resource data processing apparatus is provided, comprising:

[0138] The data distribution module 601 is used to add incremental data corresponding to the target data type of the target resource to a queue corresponding to the resource type of the target resource in response to receiving incremental data sent by the resource data source system; the queue is used to store incremental data to be processed.

[0139] The update processing module 602 is used to update the target resource data in the full data cache list sequentially according to the incremental data to be processed in the queue; the full data cache list is used to store resource data of each data type of resource of each resource type; the target resource data is the resource data corresponding to the incremental data to be processed;

[0140] The push processing module 603 is used to push the updated full data cache list to the resource data publishing system for publishing processing whenever the number of updates to the full data cache list reaches the first number if the accumulation of incremental data to be processed in the queue meets the target conditions.

[0141] The push processing module 603 is further configured to, if the accumulation of incremental data to be processed in the queue does not meet the target condition, push the updated full data cache list to the resource data publishing system for publishing processing whenever the number of updates to the full data cache list reaches the second number; the second number is less than the first number.

[0142] In an exemplary embodiment, the update processing module 602 is further configured to:

[0143] If the update type corresponding to the incremental data to be processed in the queue is a replacement type, then it is determined whether the target resource data in the full data cache list is non-empty resource data; if the target resource data is non-empty resource data, then the target resource data in the full data cache list is replaced according to the incremental data to be processed; if the target resource data is empty resource data, then it is determined whether the expired data list includes the target expired resource data; the expired data list is used to store the expired resource data of each data type of each resource type in the full data cache list; the target expired resource data is the expired resource data corresponding to the incremental data to be processed. Source data; if the expired data list includes the target expired resource data, then the target expired resource data in the expired data list is restored to the target resource data in the full data cache list, and the target resource data in the full data cache list is replaced according to the incremental data to be processed; if the expired data list does not include the target expired resource data, then it is determined whether the update type corresponding to the incremental data to be processed is adjusted to the addition type according to whether the target data type corresponding to the incremental data to be processed includes each data type, and the target resource data in the full data cache list is processed accordingly according to the adjusted update type.

[0144] In an exemplary embodiment, the update processing module 602 is further configured to:

[0145] If the target data type corresponding to the incremental data to be processed includes every data type, then the update type corresponding to the incremental data to be processed is adjusted to the add type; if the target data type corresponding to the incremental data to be processed includes some data types, then the update type corresponding to the incremental data to be processed is not adjusted; if the update type corresponding to the incremental data to be processed is adjusted to the add type, then the incremental data to be processed is added to the target resource data in the full data cache list; if the update type corresponding to the incremental data to be processed is not adjusted, then the target resource data in the full data cache list is not replaced according to the incremental data to be processed.

[0146] In an exemplary embodiment, the update processing module 602 is further configured to:

[0147] If the incremental data to be processed in the queue includes abnormal data, then a target processing method for the incremental data to be processed is determined according to the preset processing rules and other data in the incremental data to be processed excluding abnormal data; after processing the incremental data to be processed according to the target processing method, the target resource data in the full data cache list is updated.

[0148] In one exemplary embodiment, the apparatus further includes:

[0149] The alarm processing module is used to generate a data accumulation alarm message if the accumulation of incremental data to be processed in the queue meets the target condition; the data accumulation alarm message is used to prompt the user to intervene.

[0150] The queue processing module is used to respond to the user-triggered intervention operation and determine the target queue from multiple queues other than the queue mentioned above; the target queue is the queue where the accumulation of incremental data to be processed does not meet the target condition.

[0151] The thread processing module is used to determine a portion of the processing threads corresponding to the target queue as the processing threads corresponding to the queue; the processing threads are used to update the target resource data in the full data cache list according to the incremental data to be processed in the corresponding queue.

[0152] In an exemplary embodiment, the push processing module 603 is further configured to:

[0153] If resource data for one of the resources in the updated full data cache list is missing, then the associated data of the missing data is determined in the resource data; after removing the associated data from the updated full data cache list, the updated full data cache list is pushed to the resource data publishing system; the updated full data cache list is used by the resource data publishing system to display the display areas corresponding to the missing data and the associated data according to a preset display method.

[0154] Each module in the aforementioned resource data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0155] In one exemplary embodiment, a computer device is provided, which may be a server and can be used to deploy external front-end applications. The internal structure diagram of the computer device may be as follows: Figure 7As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores the data described in the above embodiments. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a resource data processing method.

[0156] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0157] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the various method embodiments described above.

[0158] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the various method embodiments described above.

[0159] In one exemplary embodiment, a computer program product is provided having a computer program stored thereon, the computer program being executed by a processor of the steps described in the various method embodiments above.

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0162] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A resource data processing method, characterized in that, The method includes: In response to receiving incremental data corresponding to the target data type of the target resource sent by the resource data source system, the incremental data is added to a queue corresponding to the resource type of the target resource; the queue is used to store incremental data to be processed. The target resource data in the full data cache list is updated sequentially based on the incremental data to be processed in the queue; the full data cache list is used to store resource data of each data type of resource of each resource type; the target resource data is the resource data corresponding to the incremental data to be processed. If the accumulation of incremental data to be processed in the queue meets the target condition, then whenever the number of updates to the full data cache list reaches the first count, the updated full data cache list will be pushed to the resource data publishing system for publishing processing. If the accumulation of incremental data to be processed in the queue does not meet the target condition, then whenever the number of updates to the full data cache list reaches the second count, the updated full data cache list will be pushed to the resource data publishing system for publishing processing; the second count is less than the first count.

2. The method according to claim 1, characterized in that, The step of updating the target resource data in the full data cache list based on the incremental data to be processed in the queue includes: If the update type corresponding to the incremental data to be processed in the queue is the replacement type, then determine whether the target resource data in the full data cache list belongs to non-empty resource data; If the target resource data is non-empty resource data, then the target resource data in the full data cache list is replaced according to the incremental data to be processed; If the target resource data is empty resource data, then determine whether the expired data list includes the target expired resource data; the expired data list is used to store the expired resource data of each data type of each resource type in the full data cache list; the target expired resource data is the expired resource data corresponding to the incremental data to be processed; If the expired data list includes the target expired resource data, then the target expired resource data in the expired data list is restored to the target resource data in the full data cache list, and the target resource data in the full data cache list is replaced according to the incremental data to be processed; If the expired data list does not include the target expired resource data, then based on whether the target data type corresponding to the incremental data to be processed includes each data type, it is determined whether to adjust the update type corresponding to the incremental data to be processed to the addition type, and based on the adjusted update type, the target resource data in the full data cache list is processed accordingly.

3. The method according to claim 2, characterized in that, The step of determining whether to adjust the update type of the incremental data to an add type based on whether the target data type corresponding to the incremental data to be processed includes: If the target data type corresponding to the incremental data to be processed includes every data type, then the update type corresponding to the incremental data to be processed is adjusted to the addition type; If the target data type corresponding to the incremental data to be processed includes some data types, then the update type corresponding to the incremental data to be processed will not be adjusted. The step of processing the target resource data in the full data cache list according to the adjusted update type includes: If the update type corresponding to the incremental data to be processed is adjusted to the addition type, then the incremental data to be processed is added to the target resource data in the full data cache list; If the update type corresponding to the incremental data to be processed is not adjusted, then the target resource data in the full data cache list will not be replaced based on the incremental data to be processed.

4. The method according to claim 1, characterized in that, The step of updating the target resource data in the full data cache list based on the incremental data to be processed in the queue includes: If the incremental data to be processed in the queue includes abnormal data, then the target processing method for the incremental data to be processed is determined according to the preset processing rules and other data in the incremental data to be processed excluding abnormal data. After processing the incremental data to be processed according to the target processing method, the target resource data in the full data cache list is updated.

5. The method according to claim 1, characterized in that, After pushing the updated full data cache list to the resource data publishing system for publishing processing whenever the number of updates to the full data cache list reaches the first count, the method further includes: If the accumulation of incremental data to be processed in the queue meets the target condition, a data accumulation alarm message is generated; the data accumulation alarm message is used to prompt the user to intervene. In response to the user-triggered intervention operation, a target queue is determined from multiple queues other than the aforementioned queue; the target queue is the queue where the accumulation of incremental data to be processed does not meet the target condition. The processing threads corresponding to the target queue are determined as the processing threads corresponding to the queue; the processing threads are used to update the target resource data in the full data cache list according to the incremental data to be processed in the corresponding queue.

6. The method according to any one of claims 1 to 5, characterized in that, The step of pushing the updated full data cache list to the resource data publishing system for publishing includes: If resource data for one of the resources in the updated full data cache list is missing, then the associated data for the missing data is determined from the resource data. After removing the associated data from the updated full data cache list, the updated full data cache list is pushed to the resource data publishing system. The updated full data cache list is used by the resource data publishing system to display the missing data and the display area corresponding to the associated data according to a preset display method.

7. A resource data processing device, characterized in that, The device includes: The data distribution module is used to respond to receiving incremental data corresponding to the target data type of the target resource sent by the resource data source system, and to add the incremental data to a queue corresponding to the resource type of the target resource; the queue is used to store incremental data to be processed. The update processing module is used to sequentially update the target resource data in the full data cache list according to the incremental data to be processed in the queue; the full data cache list is used to store resource data of each data type of resource of each resource type; the target resource data is the resource data corresponding to the incremental data to be processed; The push processing module is used to push the updated full data cache list to the resource data publishing system for publishing processing whenever the number of updates to the full data cache list reaches the first number if the accumulation of incremental data to be processed in the queue meets the target conditions. The push processing module is further configured to push the updated full data cache list to the resource data publishing system for publishing processing whenever the number of updates to the full data cache list reaches the second number, if the accumulation of incremental data to be processed in the queue does not meet the target condition; the second number is less than the first number.

8. The apparatus according to claim 7, characterized in that, The update processing module is also used for: If the update type corresponding to the incremental data to be processed in the queue is the replacement type, then determine whether the target resource data in the full data cache list belongs to non-empty resource data; If the target resource data is non-empty resource data, then the target resource data in the full data cache list is replaced according to the incremental data to be processed; If the target resource data is empty resource data, then determine whether the expired data list includes the target expired resource data; the expired data list is used to store the expired resource data of each data type of each resource type in the full data cache list; The target expired resource data is the expired resource data corresponding to the incremental data to be processed; If the expired data list includes the target expired resource data, then the target expired resource data in the expired data list is restored to the target resource data in the full data cache list, and the target resource data in the full data cache list is replaced according to the incremental data to be processed; If the expired data list does not include the target expired resource data, then based on whether the target data type corresponding to the incremental data to be processed includes each data type, it is determined whether to adjust the update type corresponding to the incremental data to be processed to the addition type, and based on the adjusted update type, the target resource data in the full data cache list is processed accordingly.

9. The apparatus according to claim 8, characterized in that, The update processing module is also used for: If the target data type corresponding to the incremental data to be processed includes every data type, then the update type corresponding to the incremental data to be processed is adjusted to the addition type; If the target data type corresponding to the incremental data to be processed includes some data types, then the update type corresponding to the incremental data to be processed will not be adjusted. If the update type corresponding to the incremental data to be processed is adjusted to the addition type, then the incremental data to be processed is added to the target resource data in the full data cache list; If the update type corresponding to the incremental data to be processed is not adjusted, then the target resource data in the full data cache list will not be replaced based on the incremental data to be processed.

10. The apparatus according to claim 7, characterized in that, The update processing module is also used for: If the incremental data to be processed in the queue includes abnormal data, then the target processing method for the incremental data to be processed is determined according to the preset processing rules and other data in the incremental data to be processed excluding abnormal data. After processing the incremental data to be processed according to the target processing method, the target resource data in the full data cache list is updated.

11. The apparatus according to claim 7, characterized in that, The device further includes: The alarm processing module is used to generate a data accumulation alarm message if the accumulation of incremental data to be processed in the queue meets the target condition; the data accumulation alarm message is used to prompt the user to intervene. The queue processing module is used to respond to the user-triggered intervention operation and determine the target queue from multiple queues other than the queue mentioned above; the target queue is the queue where the accumulation of incremental data to be processed does not meet the target condition. The thread processing module is used to determine a portion of the processing threads corresponding to the target queue as the processing threads corresponding to the queue; the processing threads are used to update the target resource data in the full data cache list according to the incremental data to be processed in the corresponding queue.

12. The apparatus according to any one of claims 7 to 11, characterized in that, The push processing module is also used for: If resource data for one of the resources in the updated full data cache list is missing, then the associated data for the missing data is determined from the resource data. After removing the associated data from the updated full data cache list, the updated full data cache list is pushed to the resource data publishing system. The updated full data cache list is used by the resource data publishing system to display the missing data and the display area corresponding to the associated data according to a preset display method.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images