An intelligent data processing method, system, device, equipment and medium

Abstract

The present invention discloses an intelligent data processing method, system, device, equipment and medium, belonging to the field of data processing technology, the method comprises: splitting a processing strategy into a plurality of sub-strategies, combining the sub-strategies into a strategy group, and storing the strategy group; a first module collects device data, obtains a corresponding strategy group, processes the device data into first data based on the strategy group, and stores the first data in a storage module; the first module obtains storage information of the first data, if the second module is in a normal state, sends the storage information to the second module, if the second module is in an abnormal state, sends the storage information to the backup module; the second module or the backup module processes the first data into second data; transmits to a third module, the third module processes the second data into third data based on the strategy group, and updates a database based on the third data. The present invention can not only avoid data loss due to a server failure, but also realize rapid replacement of a failed server.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to an intelligent data processing method, system, device, equipment and medium. Background Art

[0002] The AFC intelligent operation and maintenance system is an automated system that realizes comprehensive operation and maintenance management of the rail transit automatic ticket vending and checking system based on computer, communication, network, automatic control and other technologies. The AFC intelligent operation and maintenance system needs to collect many types of data and large amounts of data, including basic data of equipment and software, system operation data, equipment status data and application software log data. The above data that needs to be collected includes semi-structured data and unstructured data, which need to be analyzed and processed before being collected through specific interfaces. Due to the large amount of data, processing the data only through a single server will inevitably cause excessive pressure on the server, thereby affecting the data processing efficiency.

[0003] In order to improve the data processing performance of the server, the following methods are proposed in the prior art. For example, the Chinese patent application with publication number CN107015867A discloses an efficient data processing server system, which includes a data source server, a first data processing server, a second data processing server, a standby data processing server and a task allocation server. The task allocation server receives data input and provides the first data processing server, the second data processing server and the standby data processing server with source data to be processed. The first data processing server and the second data processing server process one type of data respectively; the standby data processing server can process any one of the two types of data. The data processing efficiency is improved by setting up multiple servers. For another example, the Chinese patent application with publication number CN114816751A discloses a data processing method and server, which performs sharding processing on the data to be processed to obtain multiple target sharding data; determines the target batch executor group corresponding to each target sharding data according to the first correspondence between the multiple sharding data and the multiple batch executor groups; determines the target server corresponding to each target sharding data from the server cluster based on the allocation rule, and generates a second correspondence between the multiple target servers and the multiple target batch executor groups, so that the target batch executor group processes the target sharding data stored on the corresponding target server. In this way, efficient data processing is achieved.

[0004] When a large amount of data needs to be processed, when one of the servers fails, there may be a risk of data loss. When data loss occurs, the data processing process needs to be re-executed, which not only increases the processing time but also easily causes network congestion. Summary of the invention

[0005] To solve the above problems, the present invention provides an intelligent data processing method and system to solve the problems in the prior art.

[0006] In order to achieve the above-mentioned object of the invention, the present invention proposes an intelligent data processing method, comprising:

[0007] Setting a processing strategy for module input device data, splitting the processing strategy into a plurality of sub-strategies, and determining a first number of first modules, second modules, and third modules;

[0008] Combining the sub-strategies into the first number of strategy groups, and storing the strategy groups in a storage module;

[0009] The first module collects the device data, obtains the corresponding policy group from the storage module, processes the device data into first data based on the policy group, and stores the first data in the storage module;

[0010] The first module acquires storage information of the first data, and if the second module is in a normal state, sends the storage information to the second module; if the second module is in an abnormal state, the second module is replaced by a spare module, and the first module sends the storage information to the spare module;

[0011] The second module or the standby module obtains the first data and the corresponding policy group from the storage module according to the storage information, and processes the first data into second data;

[0012] The second module or the backup module transmits the second data to the third module, and the third module processes the second data into third data based on the policy group, updates the database based on the third data, and deletes the first data and the second data in the storage module.

[0013] Further, updating the database based on the third data comprises the following steps:

[0014] The third module includes a second number of processing threads, establishes multiple databases based on the device data, calculates and obtains the standard time of each processing thread in different operating states, splits the third data into the second number of to-be-processed data, and the processing threads process the to-be-processed data respectively, defines the first processing thread that completes the processing of the to-be-processed data as the first thread, and sets it to the first type, and the first thread calculates the unit processing time based on the time of processing the to-be-processed data and the size of the to-be-processed data;

[0015] Determine the running state of the third module based on the unit processing time, determine the standard time of the remaining processing threads based on the running state, calculate the unit processing time of each processing thread except the first thread at every preset time interval, and define it as the second type if the unit processing time of the processing thread is less than or equal to the corresponding standard time, otherwise define it as the third type;

[0016] After the processing thread completes processing the data to be processed, it will be converted to the first type. When there are only processing threads of the first type and the third type, the corresponding database is updated based on the processing thread of the first type. After the processing thread of the third type is converted to the first type, the remaining database is updated.

[0017] Further, calculating the standard time includes the following steps:

[0018] Obtain historical data of each processing thread, the historical data including the time required for each processing thread to complete processing the second data and the time segment for processing the second data, cluster the processing threads based on the time segment to obtain multiple clustering results, the historical data in the same clustering result have overlapping time segments, calculate the median of the unit processing time corresponding to the same processing thread in each clustering result, and use the median as the corresponding standard time.

[0019] Furthermore, if the historical data has overlapping time segments with the other multiple historical data, the historical data corresponding to the time segment with the largest overlapping value are divided into the same clustering result.

[0020] Further, the first module counts a second amount of the collected device data, and when the first module is in a first state, when the second amount is greater than a first threshold, the first module processes the currently collected device data into the first data, and stores the first data in the storage module;

[0021] When a third amount of the first data that has not been processed in the storage module is greater than a third threshold, the storage module sends a stop instruction to the first module, and the first module switches to a second state after receiving the stop instruction. In the second state, when the first amount does not exceed the second threshold, the first module stops uploading the first data to the storage module;

[0022] When the fourth number of unprocessed first data in the storage module is greater than a fourth threshold, the storage module sends a recovery instruction to the first module, the first module switches to the first state based on the recovery instruction, and the second module uses the backup module to replace itself to request the first data.

[0023] Furthermore, the first thread calculates the unit processing time t based on a first formula, and the first formula is: , where S is the time for the first thread to process the data to be processed, D is the size of the data to be processed, and W is the weight.

[0024] The present invention also provides an intelligent data processing method system, which is used to implement the above-mentioned intelligent data processing method, and the system includes:

[0025] A terminal device, used to generate the device data;

[0026] The server is used to execute the above method.

[0027] The present invention also provides an intelligent data processing method and device, the system is used to implement the above-mentioned intelligent data processing method, the device comprises:

[0028] A setting module, for inputting a processing strategy of device data, splitting the processing strategy into a plurality of sub-strategies, determining a first number of first modules, second modules, and third modules, combining the sub-strategies into the first number of strategy groups, and storing the strategy groups in a storage module;

[0029] A first module collects the device data, obtains the corresponding policy group from the storage module, processes the device data into first data based on the policy group, stores the first data in the storage module, obtains storage information of the first data, sends the storage information to the second module if the second module is in a normal state, and replaces the second module with a spare module if the second module is in an abnormal state, and the first module sends the storage information to the spare module;

[0030] A second module, acquiring the first data and the corresponding policy group from the storage module according to the storage information, processing the first data into second data, and transmitting the second data to the third module;

[0031] a standby module, which obtains the first data and the corresponding policy group from the storage module according to the storage information, processes the first data into second data, and transmits the second data to the third module;

[0032] a third module, processing the second data into third data based on the policy group, updating a database based on the third data, and deleting the first data and the second data in the storage module;

[0033] A storage module is used to store the policy group, the first data and the second data.

[0034] The present invention also discloses an electronic device, which includes a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to implement the above-mentioned intelligent data processing method when executing the computer program stored in the memory.

[0035] The present invention also discloses a computer storage medium, wherein the computer storage medium stores program instructions, wherein when the program instructions are executed, the device where the computer storage medium is located is controlled to execute the above-mentioned method.

[0036] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0037] The present invention processes data in segments by setting up multiple modules, thereby reducing the pressure on the server to process data. After each module completes data processing, the processed data is placed in a storage module. On the one hand, it can avoid module failures that may cause loss of processed data. On the other hand, the processed data can be centrally stored in the storage module, thereby improving data security.

[0038] During the segmented processing, the policies for processing data are split into multiple policy groups and stored centrally in the storage module. When device data needs to be processed, the first module, the second module, and the third module only need to obtain the corresponding policy groups from the storage module to process the data. When a server fails, since the policy groups are stored in the storage module, a spare module can be directly used for replacement, which improves the convenience of server replacement. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of the steps of an intelligent data processing method of the present invention;

[0040] Figure 2 The figure is a schematic diagram of the structure of an intelligent data processing device of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0043] like Figure 1 As shown, an intelligent data processing method includes:

[0044] Step S1: setting a processing strategy for module input device data, splitting the processing strategy into multiple sub-strategies, and determining a first number of first modules, second modules, and third modules.

[0045] Step S2: Combine the sub-policies into a first number of policy groups, and store the policy groups in a storage module.

[0046] Step S3: The first module collects device data, obtains the corresponding policy group from the storage module, processes the device data into first data based on the policy group, and stores the first data in the storage module.

[0047] In this embodiment, the processing strategy is divided into 9 sub-strategies, such as missing value supplementation, outlier elimination, structured data conversion, etc.; the first module, the second module and the third module are each set one, and the corresponding first quantity is 3. In other embodiments, multiple second modules can also be set, and multiple second modules are cascaded front and back; then the 9 sub-strategies are combined into 3 strategy groups, each strategy group includes 3 sub-strategies, and the device data to be processed is 1-6. In this embodiment, strategy group 1 includes sub-strategies 1-3, strategy group 2 includes sub-strategies 4-6, strategy group 3 includes sub-strategies 7-9, and the complete device data includes sub-data 1-9. When the first module receives the device data, it collects strategy group 1 from the storage module, uses sub-strategy 1 to process sub-data 1-3, sub-strategy 2 to process sub-data 4, and sub-strategy 3 to process sub-data 3 and 6. The remaining sub-data do not have corresponding sub-strategies, so they are not processed in the first module for the time being, and the device data processed by the first module is defined as the first data.

[0048] After processing the first data, the first module does not directly send the first data to the second module, but stores the first data in the storage module. In this way, even if the first module fails, the processed data will not be lost, and because the data is stored in the processing module, the security of data storage is improved.

[0049] Step S4: The first module obtains the storage information of the first data. If the second module is in a normal state, the storage information is sent to the second module. If the second module is in an abnormal state, the spare module is used to replace the second module, and the first module sends the storage information to the spare module.

[0050] After the first data is successfully stored, the storage module returns the storage address and corresponding description of the first data to the first module. Before the first module sends the storage information to the second module, it first determines whether the second module is in a normal state. If it is in a normal state, the storage information is sent to the second module. If the second module is in an abnormal state, the storage information is sent to the backup module. Since the policy group is stored in the storage module, the backup module can be used directly to replace the second module. Compared with the method of storing the policy group in the server, the method of the present invention can directly replace the failed server, and there is no need to write the processing strategy into the server in advance, thereby improving the convenience of replacing the server.

[0051] Step S5: The second module or the standby module obtains the first data and the corresponding policy group from the storage module according to the storage information, and processes the first data into the second data.

[0052] Step S6: the second module or the standby module transmits the second data to the third module, the third module processes the second data into third data based on the policy group, updates the database based on the third data, and deletes the first data and the second data in the storage module.

[0053] As described above, the second module or the backup module extracts the strategy group 2 from the storage module to process the first data to obtain the second data. The second data is then sent to the third module, which processes the second data using the strategy group 3 to obtain the third data, and then uses the third data to update the corresponding database.

[0054] The present invention processes data in segments by setting up multiple modules, thereby reducing the pressure on the server to process data. After each module completes data processing, the processed data is placed in a storage module. On the one hand, it can avoid module failures that may cause loss of processed data. On the other hand, the processed data can be centrally stored in the storage module, thereby improving data security.

[0055] During the segmented processing, the policies for processing data are split into multiple policy groups and stored centrally in the storage module. When device data needs to be processed, the first module, the second module, and the third module only need to obtain the corresponding policy groups from the storage module to process the data. When a server fails, since the policy groups are stored in the storage module, a spare module can be directly used for replacement, which improves the convenience of server replacement.

[0056] It is particularly noteworthy that the present invention can not only avoid data loss due to server failure, but also realize rapid replacement of failed servers.

[0057] In this embodiment, updating the database based on the third data includes the following steps:

[0058] The third module includes a second number of processing threads, establishes multiple databases based on device data, calculates and obtains the standard time of each processing thread under different operating conditions, splits the third data into a second number of data to be processed, and the processing threads process the data to be processed respectively. The first processing thread that completes the processing of the data to be processed is defined as the first thread and is set to the first type. The first thread calculates the unit processing time based on the time for processing the data to be processed and the size of the data to be processed.

[0059] The second number is 4, and the third module includes 4 processing threads, namely processing threads 1-4. This embodiment includes 4 databases, namely databases 1-4. The present invention first obtains the time each processing thread has used to process data in the past, and calculates the standard time for the processing thread to complete data processing under different operating states based on the time. The operating state is determined according to the processing thread that first completes the processing task. For example, if the first processing thread completes the processing faster, it is considered that the server is in a relatively idle state, and if the processing thread completes the processing slower, it is considered that the server is in a relatively busy state. On this basis, different operating states correspond to different standard times, that is, when the server is in an idle state, how much data processing speed each processing thread should have, and when the server is in a busy state, how much data processing speed each processing thread should have.

[0060] After the third module generates the third data, based on the previous, a piece of equipment data includes multiple sub-data, each sub-data is used to update different databases. Therefore, here the sub-data is divided into 4 types of data to be processed according to the database to be updated, and input into different corresponding processing threads 1-4 respectively; when processing thread 1 completes the processing of the data first, processing thread 1 is positioned as the first thread, and the first thread is set to the first type, which represents the thread that has completed the data processing. Then the unit processing time of the first thread processing the sub-data is calculated, and the unit processing time is KB / S, and the calculation method is introduced later.

[0061] The operating status of the third module is determined based on the unit processing time, and the standard time of the remaining processing threads is determined based on the operating status. The unit processing time of each processing thread except the first thread is calculated at every preset time interval. If the unit processing time of the processing thread is less than or equal to the corresponding standard time, it is defined as the second type, otherwise it is defined as the third type.

[0062] By comparing the unit processing time of the first thread with the multiple standard times calculated previously, a standard time close to the unit processing time is determined, and the running state of the current server is determined according to the matched standard time. For example, if the server is in a busy state, the standard time of the remaining processing threads in this running state is obtained, and the unit processing time of the remaining processing threads is calculated at intervals of a preset time, such as 150ms, after the first thread completes the processing. Specifically, the calculation is performed in the following way: in the first 150ms after the first thread completes the processing, the size of the data processed by each processing thread in this 150ms is obtained, and the unit processing time is obtained by dividing the data size by 150. In the second 150ms, the size of the data processed by the processing thread within 300ms is divided by 300 to update the unit processing time.

[0063] When the updated unit processing time is less than or equal to the standard time, it indicates that the processing thread is processing data normally, and is accordingly classified as the second type. When the unit processing time is greater than the standard time, it indicates that the processing speed of the processing thread is too slow, and is classified as the abnormal third type.

[0064] After the processing thread completes processing the pending data, it will be converted to the first type. When there are only processing threads of the first type and the third type, the corresponding database is updated based on the processing thread of the first type. After the processing thread of the third type is converted to the first type, the remaining database is updated.

[0065] Regardless of whether it is the second type or the third type of processing thread, after completing the processing of the sub-data, it will be set to the first type. The above steps include the following process: processing thread 1 is the first type. After 150ms, processing threads 2-4 are the second type. Since the upload condition is not met, processing thread 1 does not update database 1. After 300ms, processing threads 2 and 4 are classified as the third type, and processing thread 3 is classified as the first type. Since only the first type and the third type exist, databases 1 and 3 are updated based on processing threads 1 and 3. After processing threads 2 and 4 are changed to the first type, databases 2 and 4 are updated accordingly.

[0066] Through the above steps, the corresponding database can be updated in batches according to the processing speed of the processing thread. The thread that has completed data processing will be idle to receive new data, thereby improving the data processing efficiency. During the update process, by dividing the processing threads into multiple types, some processing threads can be concentrated to update the database, which reduces the number of requests from the third module to the database and reduces the number of communications.

[0067] The calculation of the standard time in this embodiment includes the following steps:

[0068] Obtain historical data of each processing thread, the historical data including the time required for each processing thread to complete processing the second data and the time segment for processing the second data, cluster the processing threads based on the time segment to obtain multiple clustering results, the historical data in the same clustering result have overlapping time segments, calculate the median of the unit processing time corresponding to the same processing thread in each clustering result, and use the median as the corresponding standard time.

[0069] Specifically, the historical data has the following form: processing thread 1 has historical data 1, specifically, on January 1, 9:00:00-9:00:02, 3000KB of data was processed; historical data 2, on January 2, 9:00:00-9:00:03, 1500KB of data was processed; processing thread 2 has historical data 3, on January 1, 9:00:01-9:00:03, 1600KB of data was processed; 9:00:00-9:00:02 is the time segment in the historical data, and the duration of 2s in the middle is the time required for the historical data to complete processing. Then, all historical data of different processing threads are clustered based on the time segment; since historical data 1-3 have overlapping time segments, historical data 1 and 2 are classified into one category. Finally, the unit processing speeds of processing threads 1-4 on different dates can be calculated from the same clustering result. Processing threads 1-4 will correspond to a set of unit processing speed data respectively, and the median of each set of data is extracted as the standard time of the corresponding processing thread.

[0070] In this embodiment, if the historical data and the other multiple historical data have overlapping time segments, the historical data corresponding to the time segment with the largest overlapping value are divided into the same clustering result.

[0071] For example, the time segment of historical data 1 is 9:00:00-9:00:05, the time segment of historical data 2 is 9:00:00-9:00:04, and the time segment of historical data 3 is 8:00:58-9:00:01. The overlapping value of historical data 1 and 2 is 3, and the overlapping value of historical data 1 and 3 is 1. Therefore, historical data 1 and 2 are divided into one clustering result.

[0072] In this embodiment, the first module counts the second amount of collected device data. When the first module is in the first state, when the second amount is greater than the first threshold, the first module processes the currently collected device data into first data and stores the first data in the storage module.

[0073] In this embodiment, a first threshold and a second threshold are set in the first module, and a third threshold and a fourth threshold are set in the second module. The first to fourth thresholds are set in advance, the second threshold is greater than the first threshold, and the fourth threshold is greater than the third threshold. The first to fourth thresholds are set by relevant personnel according to actual conditions. Each time the first module collects a piece of device data, the device data is numbered in the format of 1, 2, 3..., so as to count the number of received device data. When the number of received devices is greater than the first threshold, the first module centrally processes the device data into the first data.

[0074] When the third number of unprocessed first data in the storage module is greater than the third threshold, the storage module sends a termination instruction to the first module. After receiving the termination instruction, the first module switches to the second state. In the second state, when the first number does not exceed the second threshold, the first module stops uploading the first data to the storage module.

[0075] When the first data that has not been processed in the storage module is greater than the third threshold, it indicates that the processing speed of the second module is too slow and a large amount of first data that needs to be processed is accumulated. In order to alleviate the processing pressure of the second module, the storage module sends a termination instruction to the first module to adjust the first module to the second state. In the second state, the first module will send the first data to the storage module only when it collects more device data. That is, without stopping the processing process of the first module, the upload time of the first module is increased, so that the second module can process the first data stored in the storage module.

[0076] When the fourth amount of unprocessed first data in the storage module is greater than the fourth threshold, the storage module sends a recovery instruction to the first module, the first module switches to the first state based on the recovery instruction, and the second module uses the backup module to replace itself to request the first data.

[0077] When there is a large amount of unprocessed first data in the second module, it indicates that after a period of time, the processing speed of the second module has not recovered, resulting in the storage module accumulating a large amount of first data. At this time, the second module is judged to be in an abnormal state, and the backup module is automatically used to take over the processing of the second module. At this time, the storage module sends a recovery instruction to the first module, allowing the first module to process the received device data in a normal state.

[0078] The first thread calculates the unit processing time t based on the first formula, which is: , where S is the time for the first thread to process the data to be processed, D is the size of the data to be processed, and W is the weight.

[0079] The weight can be a fixed value or a floating value. In this embodiment, a temperature sensor is provided in the module. The weight has a predetermined corresponding relationship with the temperature of the module. The higher the temperature, the larger the weight value. The weight value is between -0.2 and 0.5. For example, the weight corresponding to 30°C is 0. The higher the temperature, the easier it is for the system to be judged as busy.

[0080] The present invention also provides an intelligent data processing method system, which is used to implement the above-mentioned intelligent data processing method, and the system includes:

[0081] Terminal equipment, used to generate equipment data;

[0082] The server is used to execute the above method.

[0083] like Figure 2 As shown, the present invention also provides an intelligent data processing method and device, the system is used to implement the above-mentioned intelligent data processing method, the device includes:

[0084] A setting module, for inputting a processing strategy for device data, splitting the processing strategy into a plurality of sub-strategies, determining a first number of first modules, second modules, and third modules, combining the sub-strategies into a first number of strategy groups, and storing the strategy groups in a storage module;

[0085] The first module collects device data, which is specifically AFC data, obtains the corresponding policy group from the storage module, processes the device data into first data based on the policy group, stores the first data in the storage module, obtains storage information of the first data, and sends the storage information to the second module if the second module is in a normal state. If the second module is in an abnormal state, the second module is replaced by a spare module, and the first module sends the storage information to the spare module;

[0086] The second module obtains the first data and the corresponding policy group from the storage module according to the storage information, processes the first data into second data, and transmits the second data to the third module;

[0087] The standby module obtains the first data and the corresponding policy group from the storage module according to the storage information, processes the first data into second data, and transmits the second data to the third module;

[0088] A third module processes the second data into third data based on the policy group, updates the database based on the third data, and deletes the first data and the second data in the storage module;

[0089] The storage module is used to store the policy group, the first data and the second data.

[0090] The present invention also discloses an electronic device, which includes a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to implement the above-mentioned intelligent data processing method when executing the computer program stored in the memory.

[0091] The present invention also discloses a computer storage medium, wherein the computer storage medium stores program instructions, wherein when the program instructions are executed, the device where the computer storage medium is located is controlled to execute the above-mentioned method.

[0092] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0093] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The above embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0095] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent data processing method, characterized in that: include: Setting a processing strategy for module input device data, splitting the processing strategy into a plurality of sub-strategies, and determining a first number of a first module, a second module, and a third module; Combining the sub-strategies into the first number of strategy groups, and storing the strategy groups in a storage module; The first module collects the device data, obtains the corresponding policy group from the storage module, processes the device data into first data based on the policy group, and stores the first data in the storage module; The first module acquires storage information of the first data, and if the second module is in a normal state, sends the storage information to the second module; if the second module is in an abnormal state, the second module is replaced by a spare module, and the first module sends the storage information to the spare module; The second module or the standby module obtains the first data and the corresponding policy group from the storage module according to the storage information, and processes the first data into second data; The second module or the standby module transmits the second data to the third module, the third module processes the second data into third data based on the policy group, updates the database based on the third data, and deletes the first data and the second data in the storage module at the same time; Updating the database based on the third data comprises the following steps: The third module includes a second number of processing threads, establishes multiple databases based on the device data, calculates and obtains the standard time of each processing thread in different operating states, splits the third data into the second number of to-be-processed data, and the processing threads process the to-be-processed data respectively, defines the first processing thread that completes the processing of the to-be-processed data as the first thread, and sets it to the first type, and the first thread calculates the unit processing time based on the time of processing the to-be-processed data and the size of the to-be-processed data; Determine the running state of the third module based on the unit processing time, determine the standard time of the remaining processing threads based on the running state, calculate the unit processing time of each processing thread except the first thread at every preset time interval, and define it as the second type if the unit processing time of the processing thread is less than or equal to the corresponding standard time, otherwise define it as the third type; After the processing thread completes processing the data to be processed, it will be converted to the first type. When there are only processing threads of the first type and the third type, the corresponding database is updated based on the processing thread of the first type. After the processing thread of the third type is converted to the first type, the remaining database is updated.

2. The method according to claim 1, characterized in that Calculating the standard time includes the following steps: Obtain historical data of each processing thread, the historical data including the time required for each processing thread to complete processing the second data and the time segment for processing the second data, cluster the processing threads based on the time segment to obtain multiple clustering results, the historical data in the same clustering result have overlapping time segments, calculate the median of the unit processing time corresponding to the same processing thread in each clustering result, and use the median as the corresponding standard time.

3. The method according to claim 2, characterized in that If the historical data has overlapping time segments with the other multiple historical data, the historical data corresponding to the time segment with the largest overlapping value are divided into the same clustering result.

4. The method according to claim 1, characterized in that: The first module counts a second amount of the collected device data, and when the first module is in a first state, when the second amount is greater than a first threshold, the first module processes the currently collected device data into the first data, and stores the first data in the storage module; When a third amount of the first data that has not been processed in the storage module is greater than a third threshold, the storage module sends a stop instruction to the first module, and the first module switches to a second state after receiving the stop instruction. In the second state, when the first amount does not exceed the second threshold, the first module stops uploading the first data to the storage module; When the fourth number of unprocessed first data in the storage module is greater than a fourth threshold, the storage module sends a recovery instruction to the first module, the first module switches to the first state based on the recovery instruction, and the second module uses the backup module to replace itself to request the first data.

5. The method according to claim 1, characterized in that The first thread calculates the unit processing time t based on a first formula, wherein the first formula is: , S is the time for the first thread to process the data to be processed, D is the size of the data to be processed, and W is the weight.

6. An intelligent data processing system for implementing the method according to any one of claims 1 to 5, characterized in that: include: A terminal device, used to generate the device data; The server is used to execute the method according to any one of claims 1 to 5.

7. An intelligent data processing device, used to implement the method according to any one of claims 1 to 5, characterized in that: include: A setting module, for inputting a processing strategy of device data, splitting the processing strategy into a plurality of sub-strategies, determining a first number of first modules, second modules, and third modules, combining the sub-strategies into the first number of strategy groups, and storing the strategy groups in a storage module; A first module collects the device data, obtains the corresponding policy group from the storage module, processes the device data into first data based on the policy group, stores the first data in the storage module, obtains storage information of the first data, sends the storage information to the second module if the second module is in a normal state, and replaces the second module with a spare module if the second module is in an abnormal state, and the first module sends the storage information to the spare module; A second module, acquiring the first data and the corresponding policy group from the storage module according to the storage information, processing the first data into second data, and transmitting the second data to the third module; a standby module, which obtains the first data and the corresponding policy group from the storage module according to the storage information, processes the first data into second data, and transmits the second data to the third module; a third module, based on the policy group, processes the second data into third data, updates the database based on the third data, and deletes the first data and the second data in the storage module at the same time, wherein the third module includes a second number of processing threads, establishes multiple databases based on the device data, calculates and obtains the standard time of each processing thread under different operating conditions, splits the third data into the second number of to-be-processed data, and the processing threads process the to-be-processed data respectively, defines the first processing thread that completes the processing of the to-be-processed data as the first thread, and sets it as the first type, and the first thread calculates the unit processing time based on the time of processing the to-be-processed data and the size of the to-be-processed data The operation state of the third module is determined based on the unit processing time, and the standard time of the remaining processing threads is determined based on the operation state. At each preset time interval, the unit processing time of each processing thread except the first thread is calculated. If the unit processing time of the processing thread is less than or equal to the corresponding standard time, it is defined as the second type, otherwise it is defined as the third type. After the processing thread completes the processing of the to-be-processed data, it will be converted to the first type. When there are only processing threads of the first type and the third type, the corresponding database is updated based on the processing thread of the first type. After the processing thread of the third type is converted to the first type, the remaining databases are updated. A storage module is used to store the policy group, the first data and the second data.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store program instructions, and the processor is used to implement an intelligent data processing method described in any one of claims 1 to 5 when executing a computer program stored in the memory.

9. A computer storage medium, characterized in that The computer storage medium stores program instructions, wherein when the program instructions are executed, the device where the computer storage medium is located is controlled to execute the method according to any one of claims 1 to 5.

Images