Data processing methods, apparatus, equipment and storage media

Abstract

This disclosure provides a data processing method, apparatus, device, and storage medium, relating to the field of artificial intelligence technology, specifically cloud computing, big data, and the Internet of Things. The data processing method includes: receiving data to be processed sent by a data source; if the processing capability of the processing source is abnormal, determining data to be stored from the data to be processed; wherein the processing source includes: a rule engine, and / or, the data destination corresponding to the data to be processed; and storing the data to be stored in the local memory of the rule engine. This disclosure can improve the fault tolerance of the rule engine and handle abnormal processing capabilities in a lightweight manner.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to cloud computing, big data, the Internet of Things and other technical fields, and in particular to a data processing method, apparatus, device and storage medium. Background Technology

[0002] A rule engine, evolved from an inference engine, is a component embedded in an application that separates business decisions from application code and uses predefined semantic modules to write those decisions. In IoT scenarios, a rule engine can process data sent by IoT devices and forward it to the data's destination.

[0003] In a private deployment scenario, the rule engine is deployed in a private environment provided by the user, where computing and human resources are limited. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, device, and storage medium.

[0005] According to one aspect of this disclosure, a data processing method is provided, the method comprising: receiving data to be processed sent by a data source; if the processing capability of the processing party is abnormal, determining data to be stored in the data to be processed; wherein the processing party includes: a rule engine, and / or, a data destination corresponding to the data to be processed; and storing the data to be stored in the local memory of the rule engine.

[0006] According to another aspect of this disclosure, a rule engine is provided, comprising: a data forwarding module for receiving data to be processed sent by a data source; if the processing capability of the processor is abnormal, determining data to be stored in the data to be processed; wherein the processor includes: the rule engine, and / or, the data destination corresponding to the data to be processed; and storing the data to be stored in the local memory of the rule engine; the local memory is used to store the data to be stored.

[0007] According to another aspect of this disclosure, a data processing apparatus is provided, the apparatus comprising: a receiving module for receiving data to be processed sent by a data source; a determining module for determining data to be stored in the data to be processed if the processing capability of the processing source is abnormal; wherein the processing source includes: a rule engine, and / or a data destination corresponding to the data to be processed; and a storage module for storing the data to be stored in the local memory of the rule engine.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the foregoing aspects.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any of the preceding aspects.

[0010] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any of the preceding aspects.

[0011] According to the technical solution disclosed herein, a lightweight rule engine can be provided to deal with situations of abnormal processing capabilities.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0015] Figure 2 This is a schematic diagram of an application scenario corresponding to an embodiment of this disclosure;

[0016] Figure 3 This is a schematic diagram according to the second embodiment of the present disclosure;

[0017] Figure 4 This is a partial processing flow diagram of the data processing module according to an embodiment of this disclosure;

[0018] Figure 5 This is a partial processing flow diagram of the data forwarding module according to an embodiment of this disclosure;

[0019] Figure 6 This is a schematic diagram according to the third embodiment of the present disclosure;

[0020] Figure 7 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0021] Figure 8 This is a schematic diagram according to the fifth embodiment of the present disclosure.

[0022] Figure 9 This is a schematic diagram of an electronic device used to implement the data processing method of the embodiments of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] In related technologies, when the processing capabilities of the processing party (rule engine and / or data destination) are abnormal, the rule engine can directly send a rejection message to the data source (such as IoT devices). However, directly sending a rejection message results in poor fault tolerance of the rule engine, reduces user experience, and increases the processing burden on the data source. Alternatively, the rule engine can use message middleware for processing, but due to the need to introduce message middleware, it is relatively complex and lacks lightweight design, making it unsuitable for application in resource-constrained scenarios such as privatization.

[0025] To improve the fault tolerance of the rule engine and to handle abnormal processing capabilities in a lightweight manner, this disclosure provides the following embodiments.

[0026] Figure 1 Based on a schematic diagram of the first embodiment of this disclosure, this embodiment provides a data processing method, the method comprising:

[0027] 101. Receive data to be processed from the data source.

[0028] 102. If the processing capability of the processing party is abnormal, determine the data to be stored from the data to be processed; wherein the processing party includes: a rule engine, and / or the data destination party corresponding to the data to be processed.

[0029] 103. Store the data to be stored in the local memory of the rule engine.

[0030] The method in this embodiment can be applied to a rule engine.

[0031] The rules engine provides a configuration interface, allowing data sources to configure rules according to their needs. These rules can record how data is processed. Once the data sent by the data source is transmitted to the rules engine, the rules engine can process the data based on the pre-configured rules and forward the processed data to the data destination.

[0032] In the context of the Internet of Things (IoT), the data source can be IoT devices.

[0033] The data sent by IoT devices can carry information about the data destination, and the rule engine can determine the data destination based on this information.

[0034] The data sent by the data source can be called the data to be processed, and the amount of the data to be processed can be called the current data volume.

[0035] The processor may include a rules engine and / or a data destination.

[0036] For the rule engine, since it has already received data, it has the ability to receive data. An abnormal processing capacity of the rule engine means that it exceeds its processing capacity, that is, the current amount of data is greater than the maximum amount of data that the rule engine can process.

[0037] When the processing capability of the processor is abnormal, at least part of the data to be processed cannot be processed. This unprocessable data can be called data to be stored. For data to be stored, the data to be stored can be stored in the local storage of the rule engine, such as a hard disk.

[0038] In this embodiment, when the processing capability of the processor is abnormal, the rule engine determines the data to be stored in the data to be processed and stores the data to be stored in the local memory. Compared with the method of directly feeding back rejection messages, the fault tolerance of the rule engine can be improved. In addition, compared with the method of relying on message middleware, the method of storing locally in this embodiment can eliminate the need to introduce other components, reduce complexity, and thus deal with the situation of abnormal processing capability in a lightweight way.

[0039] To better understand this disclosure, the application scenarios of the embodiments of this disclosure will be described.

[0040] Figure 2 This is a schematic diagram illustrating an application scenario corresponding to an embodiment of this disclosure. For example... Figure 2 As shown, the data transmission system includes: data source 201, access device 202, rule engine 203, and data destination 204.

[0041] In IoT scenarios, the data source is IoT devices, such as... Figure 2 In Chinese, we use equipment A, equipment B, etc. to represent them.

[0042] IoT devices can interact with the rules engine through a unified access device. Figure 2 The unified access device should be represented by the IoT access service.

[0043] The data destination can be configured as needed. Figure 2 This example includes Kafka, Hypertext Transfer Protocol (HTTP), and Time Series Database (TSDB) services. Kafka is an open-source stream processing platform.

[0044] Data generated by data source 201 can be transmitted to rule engine 203 via access device 202. Rule engine 203 can process the data based on preset rules and send the processed data to data destination 204. The data sent by data source 201 carries information about the data destination, and rule engine 203 can determine the data destination 204 based on this information.

[0045] The rules engine includes a data forwarding module and a data processing module. The data forwarding module receives the data to be processed sent by the data source and stores the data to be stored in a local file. The data to be forwarded is forwarded to the data processing module, which processes the received data and sends it to the data destination.

[0046] In addition, the data forwarding module can also read data from local files and forward it to the data processing module for processing before sending it to the data destination.

[0047] The data forwarding module can determine whether the processing capability of the processor is abnormal based on the feedback information sent by the data processing module, so as to perform operations such as storage, forwarding, and reading.

[0048] In conjunction with the above application scenarios, this disclosure also provides a data processing method.

[0049] Figure 3 Based on a schematic diagram of the second embodiment of this disclosure, this embodiment provides a data processing method, the method comprising:

[0050] 301. Data forwarding module: Receives data to be processed from the data source.

[0051] Among them, combined Figure 2 In the scenario shown, the data source can be an IoT device, which can send the data to be processed to the data forwarding module in the rule engine through the access device.

[0052] 302. Data forwarding module: Determine if the processing capability of the processor is abnormal. If so, execute steps 303-304; otherwise, execute step 305 and subsequent steps.

[0053] The data forwarding module can determine whether the processing capability of the processor is abnormal based on the feedback information sent by the data processing module.

[0054] Abnormal processing capacity of the processor may include: the processor has the ability to receive data, but the current amount of data to be processed exceeds its processing capacity; or, the processor does not have the ability to receive data.

[0055] Taking the processing party as an example, which includes a rule engine and a data destination, abnormal processing capacity of the processing party may include: the current amount of data to be processed is greater than the maximum amount of data that the rule engine can process; or, the data destination does not have the ability to receive data, such as the data destination experiencing a system crash; or, the current amount of data to be processed is greater than the maximum amount of data that the data destination can process.

[0056] Based on the three abnormal situations described above, the data processing module can provide three different types of feedback information, which can be referred to as the first feedback information, the third feedback information, and the second feedback information, respectively. For details on the generation of feedback information, please refer to [link to relevant documentation]. Figure 4 The embodiments shown are described in detail below.

[0057] Specifically, for the first feedback information (exceeding the processing capacity of the data module itself) and the second feedback information (exceeding the processing capacity of the data destination), the feedback information can carry the maximum amount of data that the corresponding processor (data processing module or rule engine, or data destination) can process. Thus, the data forwarding module can determine whether the processing capacity is abnormal based on the current amount of data to be processed and the maximum amount of data carried in the feedback information.

[0058] In response to third feedback information (the data destination does not have the ability to receive data), the data forwarding module can determine that the processing capability of the processor is abnormal after receiving the third feedback information.

[0059] That is, if the processor has the ability to receive data, the maximum amount of data that the processor can process is obtained; if the current amount of data to be processed is greater than the maximum amount of data, the processing capability is determined to be abnormal. Alternatively,

[0060] If the processing party includes the data destination party, and it is determined that the data destination party does not have the ability to receive data, then the processing capability is determined to be abnormal.

[0061] In this embodiment, the processing capacity of the processor can be determined based on the current amount of data to be processed and the maximum amount of data that the processor can process. This allows for timely identification of situations that exceed the processor's processing capacity, thus enabling timely and accurate identification of abnormal situations.

[0062] In this embodiment, it is possible to determine whether the processing capability of the processor (specifically the data destination) is abnormal based on whether the processor has the ability to receive data. This allows for timely identification of situations where the processor does not have the ability to receive data, thus enabling timely and accurate identification of abnormal situations.

[0063] 303. Data forwarding module, which determines the data to be stored from the data to be processed.

[0064] Specifically, for anomalies exceeding processing capacity, i.e., when the current data volume is greater than the maximum data volume, a first difference between the current data volume and the maximum data volume can be determined; among the data to be processed, the data with a data volume equal to the first difference is selected as the data to be stored.

[0065] For example, if the maximum data size is 4K and the current data size is 6K, then the first difference can be calculated as 2K. The remaining 2K of data to be processed can then be used as the data to be stored. This selection can be random or based on other preset rules.

[0066] In this embodiment, the data to be stored is determined based on the first difference between the current data volume and the maximum data volume. Data exceeding the processing capacity can be stored locally to achieve a fault-tolerant rule engine in a lightweight manner.

[0067] In case of an anomaly where the data destination does not have the ability to receive data, a preset proportion of the data to be processed can be selected as the data to be stored.

[0068] The preset ratio can be 100%, meaning that all the data to be processed can be used as data to be stored. Alternatively, the preset ratio can be a larger value close to 100%, and the specific value can be set according to actual needs.

[0069] In this embodiment, a preset proportion of data is selected as the data to be stored. When the data destination does not have the ability to receive data, all or nearly all of the data to be processed can be stored locally, so as to realize a rule engine with fault tolerance in a lightweight manner.

[0070] In addition, for cases where the data destination lacks the ability to receive data, the data forwarding module can select a small amount of data (the amount of data can be set) as test data and send it to the data destination to determine whether the data destination has recovered. For example, if the rule engine performs a write operation on the data destination and receives a write timeout response message, it can be determined that the data destination has not recovered. Alternatively, if the rule engine performs a write operation on the data destination and receives a write success response message, it can be determined that the data destination has recovered. Here, "recovered" means having the ability to receive data. Furthermore, based on the current processing status, a write success response message can indicate whether the processing capacity of the data destination has been exceeded.

[0071] 304. The data forwarding module stores the data to be stored in the local memory of the rule engine.

[0072] This can be stored as a file, i.e., a reference. Figure 2 It can be stored in a local file.

[0073] Additionally, for data that is not intended for storage, it can be forwarded to the data processing module for processing. For example, if the maximum data size is 4K and the current data size is 6K, the first difference can be calculated as 2K. Then, 2K of the data to be processed can be stored locally as data to be stored, while the other 4K of data to be processed can be forwarded to the data processing module for processing.

[0074] 305. Data forwarding module, forwards data to the data processing module.

[0075] The data forwarding module forwards data that can be data to be processed, or it can include both data to be processed and data read from local storage. Details will follow later. Figure 5 The example shown.

[0076] In addition, before forwarding data, the data forwarding module can also perform format conversion and other processing on the data to convert it into the data format inside the rule engine.

[0077] 306. Data processing module, which processes the received data.

[0078] 307. The data processing module sends the processed data to the data destination.

[0079] The data processing module can process the received data according to preset rules when it does not exceed its own processing capacity, and send the processed data to the data destination when the data destination is normal (does not exceed its processing capacity and has the ability to receive data).

[0080] The data processing module can send feedback information to the data forwarding module when it encounters an anomaly (exceeding its own processing capacity) and / or when the data destination encounters an anomaly (exceeding the processing capacity of the data destination or the data destination does not have the ability to receive data). Different feedback information can be sent for different anomaly situations.

[0081] In this embodiment, when the data forwarding module determines that the processing capability of the processor is abnormal, it can identify the data to be stored in the data to be processed and store the data to be stored in the local memory of the rule engine. Since the rejection message is not directly sent to the data source in case of an anomaly, the fault tolerance of the rule engine can be improved, the user experience can be enhanced, and the processing burden of the data source can be reduced. Storing the data to be stored locally, instead of introducing a message middleware, can reduce system complexity and implement a rule engine with a certain degree of fault tolerance in a lightweight manner, which is particularly suitable for resource-limited scenarios such as user private environments.

[0082] Figure 4 This is a partial processing flow diagram of the data processing module according to an embodiment of this disclosure, such as... Figure 4 As shown, the process may include:

[0083] 401. Data processing module: receives data sent by the data forwarding module.

[0084] Among them, reference Figure 3 When the data forwarding module determines that the processing capability of the processor is normal, it can forward data to the data processing module.

[0085] 402. Data processing module: Determine if the data exceeds its processing capacity. If so, execute 403; otherwise, execute 404.

[0086] In specific implementation, the data processing module may include at least one processing resource. If all processing resources are occupied after receiving data, it indicates that the processing capacity has been exceeded; conversely, if there are idle processing resources after receiving data, it indicates that the processing capacity has not been exceeded.

[0087] Processing resources refer to resources used to process data, such as processes and containers.

[0088] Furthermore, processing resources can be dynamically increased. Taking containers as an example, initially, the data processing module can start N (N is a positive integer) containers. After receiving data, the received data can be allocated to these N containers for processing. Assuming that all N containers are occupied, and new data is received, the data processing module can dynamically add a preset number of containers. For example, a new container can be added, and this new container can be used to process the new data. This process continues until the number of containers increases to a threshold M (M is a positive integer, greater than or equal to N). If, after the number of containers increases from N to M, new data is received, and all M containers are occupied, it can be determined that the module's processing capacity has been exceeded.

[0089] 403. The data processing module sends the first feedback information to the data forwarding module.

[0090] The first feedback information is used to indicate that the data processing module's processing capacity has been exceeded (or the rule engine's processing capacity has been exceeded).

[0091] In addition, the first feedback information can carry the maximum amount of data that the data processing module can process (or the maximum amount of data that the rule engine can process), so that the data forwarding module can determine whether the processing capacity is abnormal based on the current amount of data to be processed and the maximum amount of data.

[0092] The data processing module can obtain the maximum amount of data mentioned above based on historical statistical information.

[0093] That is, the amount of historical data processed by the processor within a preset time period is counted; based on the amount of historical data processed, the maximum amount of data is determined.

[0094] The preset time period can refer to the time period from the time point t1 when data processing begins to the time point t2 when it is determined that the processing capacity is exceeded. By statistically analyzing the amount of data processed during the time period [t1, t2], the historical data volume is obtained, and then the maximum data volume is determined.

[0095] For example, t1 can be used as the starting point for statistics. The statistics of the processed data can be counted from the starting point until time t2 is reached to obtain the historical data volume.

[0096] Then, the historical data volume can be used as the maximum data volume, or the maximum data volume can be obtained by performing calculations on the historical data volume based on a preset algorithm.

[0097] In this embodiment, the maximum data volume is obtained based on historical statistical information. The maximum data volume can be flexibly determined according to the actual situation. Compared with fixed settings, this method is more flexible and the data is more accurate.

[0098] 404. Data processing module: Determine whether the data destination has the ability to receive data. If yes, execute 406; otherwise, execute 405.

[0099] After processing the data, the data processing module can send the processed data to the data destination. If the data destination does not have the ability to receive data, the data processing module can receive the corresponding response message. For example, if the response message from the data destination is a write timeout, it can be determined that the data destination does not have the ability to receive data.

[0100] 405. The data processing module sends third feedback information to the data forwarding module.

[0101] When the data destination does not have the ability to receive data, the data processing module can send a third feedback message to the data forwarding module. The third feedback message is used to indicate that the data destination does not have the ability to receive data.

[0102] 406. Data processing module: Determine if the data exceeds the processing capacity of the data destination. If yes, proceed to 407; otherwise, proceed to 408.

[0103] When the data destination exceeds its processing capacity, it can send a response message to the data processing module indicating that it has exceeded its processing capacity. The data processing module can then determine whether the data has exceeded its processing capacity based on this response message. The data destination's process for determining whether it has exceeded its processing capacity is similar to the data processing module's decision-making process.

[0104] In addition, the data processing module can also count the amount of data sent to the data destination to obtain the maximum amount of data that the data destination can process. The time period counted is, for example, from the time of the start of sending to the time of receiving a response message indicating that the data destination's processing capacity has been exceeded.

[0105] 407. The data processing module sends the second feedback information to the data forwarding module.

[0106] The second feedback information is used to indicate that the data exceeds the processing capacity of the data destination.

[0107] In addition, the second feedback information can carry the maximum amount of data that the data destination can process, so that the data forwarding module can determine whether the processing capacity is abnormal based on the current amount of data to be processed and the maximum amount of data.

[0108] 408. Data processing module, processes data.

[0109] The data processing module can process the received data based on preset rules, such as converting the data into the format required by the data destination or taking the maximum value of all data.

[0110] Among them, 408 and 402-407 have no timing constraint relationship. That is, the data processing module can process the received data as long as it is determined that it does not exceed its own processing capacity, without having to wait until all judgments are completed before processing the data.

[0111] After processing the data, if the data destination has the ability to receive the data and the processing capacity is not exceeded, the data processing module can send the processed data to the data destination.

[0112] If the data destination is abnormal (lacking data reception capability, or exceeding the data destination's processing capability), the data processing module can choose not to send data to the data destination, thus implementing a circuit breaker mechanism. This avoids the impact of meaningless write operations on the rule engine, improves resource utilization, and prevents resource waste.

[0113] In this embodiment, when the processing capacity of the rule engine is exceeded, the data processing module can send a first feedback message to the data forwarding module, which carries the maximum amount of data that the rule engine can process. The data forwarding module can then determine whether an anomaly has occurred based on the first feedback message. Alternatively, when the data destination does not have the ability to receive data, the data processing module can send a third feedback message to the data forwarding module, which can then determine whether an anomaly has occurred based on the third feedback message. Or, when the processing capacity of the data destination is exceeded, the data processing module can send a second feedback message to the data forwarding module, which carries the maximum amount of data that the data destination can process. The data forwarding module can then determine whether an anomaly has occurred based on the second feedback message. That is, the data forwarding module can accurately identify whether the processing capacity of the processing party is abnormal based on the feedback message from the data processing module. Furthermore, the data processing module can have a statistical function to obtain the maximum data volume and send it to the data forwarding module in the feedback message, so that the data forwarding module can determine whether the processing capacity is abnormal based on the maximum data volume.

[0114] Figure 5 This is a partial processing flow diagram of the data forwarding module according to an embodiment of this disclosure, such as... Figure 5 As shown, the process includes:

[0115] 501. Data forwarding module: Determine if the processing capability of the processor is abnormal. If not, execute 502.

[0116] The determination of whether the processing capability is abnormal, and the handling process after the processing capability is abnormal, can be found in the relevant descriptions of the above embodiments, and will not be described in detail here.

[0117] 502. Data forwarding module: Determine whether local data is stored in the local memory. If yes, execute 503-504; otherwise, execute 505.

[0118] 503. Data forwarding module: Read at least a portion of the local data to obtain the read data.

[0119] The local data read by the data forwarding module can be referred to as read data.

[0120] Normal processing capacity means that the processing capacity of the rule engine is not exceeded, and the data destination has the ability to receive data, and the processing capacity of the data destination is not exceeded.

[0121] Since the processing capacity of either the rule engine or the data destination is not exceeded, the current data volume to be processed is less than or equal to the maximum data volume of either party. If the current data volume is less than the minimum between the maximum data volume of the rule engine and the maximum data volume of the data destination, the difference between the minimum between these two maximum data volumes and the current data volume can be calculated. This difference can be called the second difference. From the stored data, local data with a volume equal to this second difference is randomly selected or obtained using other methods and used as the read data.

[0122] 504. The data forwarding module forwards the data to be processed and the data to be read to the data processing module.

[0123] For example, if the second difference is 2K and the current data volume is 1K, then 1K of local data can be read as the read data.

[0124] Accordingly, 1K of data to be processed and 1K of data to be read can be sent to the data processing module.

[0125] 505. Data forwarding module: forwards the data to be processed to the data processing module.

[0126] If no data is stored in the local memory, the data to be processed can be forwarded directly (in real time) to the data processing module.

[0127] After the data processing module receives the data, it can refer to... Figure 3 and / or Figure 4 The relevant descriptions of the embodiments shown will not be repeated here.

[0128] In this embodiment, when the processing capability of the processor is normal, the data to be processed can be processed, or local data can be read and processed to achieve the fault tolerance of the rule engine.

[0129] Figure 6This is a schematic diagram based on the third embodiment of the present disclosure. This embodiment provides a rule engine 600, which includes a data forwarding module 601 and a local storage 602.

[0130] The data forwarding module 601 is used to receive data to be processed sent by the data source; if the processing capability of the processor is abnormal, it determines data to be stored in the data to be processed; wherein, the processor includes: the rule engine, and / or, the data destination corresponding to the data to be processed; and, the data to be stored is stored in the local memory of the rule engine.

[0131] The local memory 602 is used to store the data to be stored.

[0132] In this embodiment, when the processing capability of the processor is abnormal, the rule engine determines the data to be stored in the data to be processed and stores the data to be stored in the local memory. Compared with the method of directly feeding back rejection messages, the fault tolerance of the rule engine can be improved. In addition, compared with the method of relying on message middleware, the method of storing locally in this embodiment can eliminate the need to introduce other components, reduce complexity, and thus deal with the situation of abnormal processing capability in a lightweight way.

[0133] like Figure 7 As shown, the rule engine 600 may also include a data processing module 603.

[0134] The data processing module 603 is used to send feedback information to the data forwarding module when the processing capability of the processor is abnormal; the data forwarding module 601 is also used to determine whether the processing capability is abnormal based on the feedback information.

[0135] In this embodiment, when the data processing module determines that the processing capability of the processor is abnormal, it can send feedback information to the data forwarding module. The data forwarding module can determine whether the processing capability is abnormal based on the feedback information, so as to carry out corresponding processing based on whether it is abnormal, and thus deal with the situation of abnormal processing capability in a lightweight manner.

[0136] In some embodiments, the feedback information carries the maximum amount of data that the processor can process; the data forwarding module 601 is further configured to: obtain the maximum amount of data based on the feedback information; and, if the current amount of data to be processed is greater than the maximum amount of data, determine that the processing capability is abnormal; and / or, determine a first difference between the current amount of data and the maximum amount of data; and, among the data to be processed, select the data to be processed whose amount is the first difference as the data to be stored.

[0137] The feedback information at this time can be the first feedback information and / or the second feedback information in the above embodiments.

[0138] In this embodiment, the processing capacity of the processor can be determined based on the current amount of data to be processed and the maximum amount of data that the processor can process. This allows for timely identification of situations that exceed the processor's processing capacity, thus enabling timely and accurate identification of abnormal situations.

[0139] In some embodiments, the data forwarding module 601 is further configured to: if the current data volume of the data to be processed is less than the maximum data volume, and there is no local data in the local storage, send the data to be processed to the data processing module; or, if the current data volume of the data to be processed is less than the maximum data volume, and there is local data in the local storage, determine a second difference between the maximum data volume and the current data volume, read local data with a data volume equal to the second difference as read data; and send the data to be processed and the read data to the data processing module.

[0140] In this embodiment, when the processing capability of the processor is normal, the data to be processed can be processed, or local data can be read and processed to achieve the fault tolerance of the rule engine.

[0141] In some embodiments, the data processing module 603 is further configured to: count the amount of historical data processed by the processor within a preset time period; determine the maximum amount of data based on the amount of historical data processed; and carry the maximum amount of data in the feedback information.

[0142] In this embodiment, the data processing module can have a statistical function to obtain the maximum amount of data and send it to the data forwarding module in the feedback information, so that the data forwarding module can determine whether the processing capacity is abnormal based on the maximum amount of data.

[0143] In some embodiments, the feedback information is used to indicate that the data destination does not have the ability to receive data; the data forwarding module 601 is further used to: determine the abnormal processing capability based on the feedback information; and / or, select a preset proportion of data from the data to be processed as the data to be stored.

[0144] The feedback information at this time can be the third feedback information in the above embodiment.

[0145] In this embodiment, it is possible to determine whether the processing capability of the processor (specifically the data destination) is abnormal based on whether the processor has the ability to receive data. This allows for timely identification of situations where the processor does not have the ability to receive data, thus enabling timely and accurate identification of abnormal situations.

[0146] Figure 8This is a schematic diagram based on the fifth embodiment of the present disclosure. This embodiment provides a data processing device 800, which includes: a receiving module 801, a determining module 802, and a storage module 803.

[0147] The receiving module 801 is used to receive data to be processed sent by the data source;

[0148] The determination module 802 is used to determine data to be stored in the data to be processed if the processing capability of the processor is abnormal; wherein, the processor includes: a rule engine, and / or, the data destination corresponding to the data to be processed;

[0149] The storage module 803 is used to store the data to be stored in the local memory of the rule engine.

[0150] This device can be located within the rules engine.

[0151] In this embodiment, when the processing capability of the processor is abnormal, the rule engine determines the data to be stored in the data to be processed and stores the data to be stored in the local memory. Compared with the method of directly feeding back rejection messages, the fault tolerance of the rule engine can be improved. In addition, compared with the method of relying on message middleware, the method of storing locally in this embodiment can eliminate the need to introduce other components, reduce complexity, and thus deal with the situation of abnormal processing capability in a lightweight way.

[0152] In some embodiments, the determining module 802 is further configured to: if the processor has data receiving capability, obtain the maximum amount of data that the processor can process; if the current amount of data to be processed is greater than the maximum amount of data, determine that the processing capability is abnormal.

[0153] In this embodiment, the processing capacity of the processor can be determined based on the current amount of data to be processed and the maximum amount of data that the processor can process. This allows for timely identification of situations that exceed the processor's processing capacity, thus enabling timely and accurate identification of abnormal situations.

[0154] In some embodiments, the determining module 802 is further configured to: determine a first difference between the current data volume and the maximum data volume; and select, from the data to be processed, the data volume that is the first difference as the data to be stored.

[0155] In this embodiment, the data to be stored is determined based on the first difference between the current data volume and the maximum data volume. Data exceeding the processing capacity can be stored locally to achieve a fault-tolerant rule engine in a lightweight manner.

[0156] In some embodiments, the device 800 further includes:

[0157] The forwarding module is configured to: if the current data volume of the data to be processed is less than the maximum data volume, and there is no local data in the local storage, process the data to be processed and send it to the data destination; or, if the current data volume of the data to be processed is less than the maximum data volume, and there is local data in the local storage, determine a second difference between the maximum data volume and the current data volume, read local data with a data volume equal to the second difference, and use it as read data; and process the data to be processed and the read data, and then send it to the data destination.

[0158] In this embodiment, when the processing capability of the processor is normal, the data to be processed can be processed, or local data can be read and processed to achieve the fault tolerance of the rule engine.

[0159] In some embodiments, the device 800 further includes:

[0160] The statistics module is used to count the amount of historical data processed by the processor within a preset time period; and to determine the maximum amount of data based on the amount of historical data processed.

[0161] In this embodiment, the data processing module can have a statistical function to obtain the maximum amount of data and send it to the data forwarding module in the feedback information, so that the data forwarding module can determine whether the processing capacity is abnormal based on the maximum amount of data.

[0162] In some embodiments, the determining module 802 is further configured to: if the processing party includes the data destination party, and it is determined that the data destination party does not have the ability to receive data, determine that the processing capability is abnormal.

[0163] In this embodiment, it is possible to determine whether the processing capability of the processor (specifically the data destination) is abnormal based on whether the processor has the ability to receive data. This allows for timely identification of situations where the processor does not have the ability to receive data, thus enabling timely and accurate identification of abnormal situations.

[0164] In some embodiments, the determining module 802 is further configured to: select a preset proportion of data from the data to be processed as the data to be stored.

[0165] In this embodiment, a preset proportion of data is selected as the data to be stored. When the data destination does not have the ability to receive data, all or nearly all of the data to be processed can be stored locally, so as to realize a rule engine with fault tolerance in a lightweight manner.

[0166] It is understood that the same or similar content in different embodiments of this disclosure can be referred to each other.

[0167] It is understood that the terms "first" and "second" in the embodiments of this disclosure are only used for distinction and do not indicate the degree of importance or the order of events.

[0168] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0169] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0170] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 900 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0171] like Figure 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 909 into a random access memory (RAM) 903. The RAM 903 may also store various programs and data required for the operation of the electronic device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0172] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0173] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0174] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable load balancing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0179] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0180] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0181] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, the method comprising: Receive pending data sent by the data source; If the processor has data receiving capabilities, obtain the maximum amount of data that the processor can process. If the current data volume of the data to be processed is greater than the maximum data volume, the processing capacity is determined to be abnormal. If the processing capability of the processor is abnormal, the data to be stored is determined from the data to be processed, including: determining a first difference between the current data volume and the maximum data volume; selecting the data to be processed with a data volume equal to the first difference from the data to be processed, as the data to be stored; wherein, the processor includes: a rule engine, and / or, the data destination corresponding to the data to be processed; The data to be stored is stored in the local memory of the rule engine.

2. The method according to claim 1, further comprising: If the current data volume of the data to be processed is less than the maximum data volume, and there is no local data in the local storage, the data to be processed is then sent to the data destination after processing. or, If the current data volume of the data to be processed is less than the maximum data volume, and there is local data in the local storage, determine the second difference between the maximum data volume and the current data volume, and read the local data with a data volume equal to the second difference as the read data; After processing the data to be processed and the data to be read, the data is sent to the data destination.

3. The method according to claim 1, wherein, The step of obtaining the maximum amount of data that the processor can process includes: The amount of historical data processed by the processor within a preset time period is statistically analyzed. The maximum data volume is determined based on the amount of historical data that has been processed.

4. The method according to claim 1, further comprising: If the processing party includes the data destination party, and it is determined that the data destination party does not have the ability to receive data, then the processing capability is determined to be abnormal.

5. The method according to claim 4, wherein, The step of determining the data to be stored from the data to be processed includes: From the data to be processed, a preset proportion of the data is selected as the data to be stored.

6. A rules engine, comprising: The data forwarding module is used to receive unprocessed data sent by the data source. If the processing capability of the processor is abnormal, data to be stored is determined from the data to be processed; wherein, the processor includes: the rule engine, and / or, the data destination corresponding to the data to be processed; and, the data to be stored is stored in the local memory of the rule engine; Local storage is used to store the data to be stored; The data processing module is used to send feedback information to the data forwarding module when the processing capability of the processor is abnormal; the feedback information carries the maximum amount of data that the processor can process. The data forwarding module is further configured to obtain the maximum data volume based on the feedback information; and, if the current data volume of the data to be processed is greater than the maximum data volume, determine that the processing capacity is abnormal; determine a first difference between the current data volume and the maximum data volume; and, among the data to be processed, select the data to be processed whose data volume is the first difference as the data to be stored.

7. The rule engine according to claim 6, wherein, The data forwarding module is also used for: If the current data volume of the data to be processed is less than the maximum data volume, and there is no local data in the local storage, the data to be processed is sent to the data processing module. or, If the current data volume of the data to be processed is less than the maximum data volume, and there is local data in the local storage, determine the second difference between the maximum data volume and the current data volume, and read the local data with a data volume equal to the second difference as the read data; The data to be processed and the data to be read are sent to the data processing module.

8. The rule engine according to claim 6, wherein, The data processing module is also used for: The amount of historical data processed by the processor within a preset time period is statistically analyzed. The maximum data volume is determined based on the amount of historical data that has been processed; The maximum amount of data is carried in the feedback information.

9. The rule engine according to claim 6, wherein, The feedback information is used to indicate that the data destination does not have the ability to receive data; The data forwarding module is further used for: Based on the feedback information, it is determined that the processing capability is abnormal; and / or, From the data to be processed, a preset proportion of the data is selected as the data to be stored.

10. A data processing apparatus, the apparatus comprising: The receiving module is used to receive data to be processed sent by the data source. The determination module is used to determine the maximum amount of data that the processor can process if the processor has data receiving capabilities. If the current data volume of the data to be processed is greater than the maximum data volume, the processing capacity is determined to be abnormal. If the processing capability of the processor is abnormal, the data to be stored is determined from the data to be processed, including: determining a first difference between the current data volume and the maximum data volume; selecting the data to be processed with a data volume equal to the first difference from the data to be processed, as the data to be stored; wherein, the processor includes: a rule engine, and / or, the data destination corresponding to the data to be processed; The storage module is used to store the data to be stored in the local memory of the rule engine.

11. The apparatus of claim 10, further comprising: The forwarding module is used to process the data to be processed and send it to the data destination if the current data volume of the data to be processed is less than the maximum data volume and there is no local data in the local storage. Alternatively, if the current data volume of the data to be processed is less than the maximum data volume, and there is local data in the local storage, determine a second difference between the maximum data volume and the current data volume, and read the local data with a data volume equal to the second difference as the read data; Furthermore, after processing the data to be processed and the data read, the data is sent to the data destination.

12. The apparatus of claim 10, further comprising: The statistics module is used to count the amount of historical data processed by the processor within a preset time period; And, based on the amount of historical data that has been processed, determine the maximum amount of data.

13. The apparatus according to claim 10, wherein, The determining module is also used for: If the processing party includes the data destination party, and it is determined that the data destination party does not have the ability to receive data, then the processing capability is determined to be abnormal.

14. The apparatus according to claim 13, wherein, The determining module is further used for: From the data to be processed, a preset proportion of the data is selected as the data to be stored.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images