Data processing method, system, device and storage medium

Abstract

The present application provides a data processing method, system, device and storage medium, the method comprising: a gateway obtains multiple data, classifies the multiple data according to the data format and / or the compilation language of the data, obtains M groups of data of different categories, and sends the M groups of data to a cache module; the cache module comprises M storage pipelines, the M storage pipelines receive and cache the M groups of data sent by the gateway; the computing module obtains target data from the M groups of data, the target data is one or more groups of data in the M groups of data; the computing module preprocesses the target data, obtains the data to be stored, and stores the data to be stored in a database. The method can preprocess the data before storing it in the database, and then classify and store it in the database, which relieves the pressure on the database and avoids the delay when storing the data directly in the database and the problem of high pressure on the database.

Description

Technical Field

[0001] The present application relates to the field of data processing, and more specifically, to a data processing method, system, device and storage medium in the field of data processing. Background Art

[0002] Against the backdrop of the rapid development of Internet technology, the level of science and technology is also constantly improving. Researchers are committed to creating and inventing more intelligent devices, such as robots, vacuum cleaners, smart home devices, etc. The emergence of these products can serve people more intelligently in daily life, save manpower, and thus change people's daily lives.

[0003] During operation, smart devices mainly collect different data through various sensors inside the devices. Smart devices process and analyze the collected data. Based on the processing and analysis results, users can further control the working process of smart devices so that smart devices can meet users' needs in different scenarios.

[0004] When analyzing and processing the data of smart devices, the smart devices first send the data collected by their various sensors to the gateway. The gateway receives the data reported by the smart devices, classifies the data according to the data type, and then the gateway stores the data in batches to the database. When the user needs to view certain sensor data, the processing task can be pre-configured in the database. Furthermore, the database calculates and processes the sensor data according to the processing task. Finally, the display module obtains the data after the database calculation and processing for the user to view.

[0005] In the above process, different sensors collect data at different frequencies. For sensors with a higher frequency of data collection, the amount of data collected by these sensors is relatively large. In the case of large amounts of data, when the gateway needs to send and store a large amount of data in the database, delayed storage will occur, which will cause the gateway to wait for database storage for a longer time. In addition, when a data peak occurs, the adaptability of the above process is poor, which can easily cause failures in the data processing system.

[0006] In addition, in the above process, when the gateway stores data in the database, the calculation and processing of the data mainly rely on the computing power of the database itself. If the amount of data stored in the database is large, the calculation of the database will be slow and delayed.

[0007] Therefore, in the process of processing data from smart devices, how to achieve reasonable allocation and calculation of large amounts of data to avoid data processing system failures and data processing delays has become an urgent problem that needs to be solved. Summary of the invention

[0008] The present application provides a data processing method, system, device and storage medium. The method can pre-process the data before storing it in the database and then classify and store it in the database, thereby alleviating the pressure on the database and avoiding the delay when storing the data directly in the database and the problem of high pressure on the database.

[0009] In a first aspect, a data processing method is provided, characterized in that it is applied to a system, the system comprising a gateway, a cache module, a computing module and a database, the method comprising: the gateway obtains multiple data, classifies the multiple data according to the data format and / or the compilation language of the data, obtains M groups of data of different categories, and sends the M groups of data to the cache module; the cache module comprises M storage pipelines, the M storage pipelines receive and cache the M groups of data sent by the gateway; the computing module obtains target data from the M groups of data, the target data is one or more groups of data in the M groups of data; the computing module preprocesses the target data to obtain data to be stored, and stores the data to be stored in the database.

[0010] In the above technical solution, a method is proposed for preprocessing data through a computing module before the data is stored in a database. The data is classified through a gateway to obtain multiple groups of data of different types, and the multiple groups of data are sent to a cache module. Multiple storage pipelines in the cache module receive and cache the multiple groups of data. The computing module obtains the target data from the multiple groups of data for preprocessing and calculation to obtain the data to be stored, and finally stores the data to be stored in the database. Before the data to be stored is stored in the database, the target data is preprocessed. Compared with directly storing the target data in the database, the method of processing and calculating the target data in the database can effectively reduce the pressure of storing data in the database and avoid the problem of system crash or operation interruption caused by excessive database pressure.

[0011] In combination with the first aspect, in some possible implementations, the computing module includes multiple sub-modules, and the method also includes: the computing module obtains the data computing capability of each of the multiple sub-modules, and determines a sub-module from the multiple sub-modules whose data computing capability is greater than or equal to a first preset threshold as a target sub-module; and the computing module preprocesses the target data to obtain the data to be stored, including: preprocessing the target data through the target sub-module to obtain the data to be stored.

[0012] In the above technical solution, by selecting a submodule with strong data processing capability as the target submodule and preprocessing the target data through the target submodule, the system's computing and processing capability for the target data is optimized and the system's efficiency in processing the target data is ensured.

[0013] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the method also includes: when the data computing capability of each of the multiple sub-modules is less than or equal to a second preset threshold, the computing module stops obtaining the target data from the target storage pipeline.

[0014] In the above technical solution, when the data computing capabilities of multiple sub-modules are less than the second preset threshold, it indicates that the computing power of the computing module has reached a bottleneck. At this time, the computing module will stop obtaining target data from the target storage pipeline, which can appropriately reduce the processing pressure of the computing module, avoid the computing pressure of the system caused by data peaks, and ensure the efficiency of the system in processing data.

[0015] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the computing module obtains target data from the M groups of data, including: the computing module obtains the address of each storage pipe in the M storage pipes; the computing module determines the target task according to whether there is cached data in each storage pipe in the M storage pipes, and determines the target address corresponding to the target task; the computing module determines the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe based on the target address and the address of each storage pipe in the M storage pipes, and obtains the target data from the target storage pipe.

[0016] In combination with the first aspect and the above-mentioned implementation manner, in some possible implementation manners, the database includes a cold data storage unit and a hot data storage unit, and the method also includes: the calculation module obtains the access frequency of data of each data type in the database, and divides the data to be stored into cold data and hot data according to the access frequency of data of each data type in the database and the data type of the data to be stored, wherein data whose access frequency of data of any data type in the database is greater than or equal to a third preset threshold is determined as hot data, otherwise it is determined as cold data; the data type of the data to be stored that is hot data in the data to be stored is the same as the data type of the data that is hot data in the database, and the data type of the data to be stored that is cold data in the data to be stored is the same as the data type of the data that is cold data in the database; and storing the data to be stored in the database includes: the calculation module stores the data that is hot data in the data to be stored in the hot data storage unit, and stores the data that is cold data in the data to be stored in the cold data storage unit.

[0017] In the above scheme, after the computing module processes the target data to obtain the data to be stored, it classifies the data to be stored into cold data and hot data, and further stores the data to be stored in the corresponding data storage unit in the database according to the classification results, thereby alleviating the pressure on the database when all data is stored in one database, and at the same time, it can meet user needs in a timely and fast manner when users read, thereby improving user experience.

[0018] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the system also includes a display module, and the method also includes: the display module receives a user's query request, and in response to the query request, the display module obtains and displays the stored data from the database for the user to read and view.

[0019] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the cache module includes one or more cache nodes, each of the one or more cache nodes corresponds to one or more storage pipes, and each of the one or more storage pipes is used to receive and cache any one or more groups of data among the M groups of data.

[0020] In combination with the first aspect and the above-mentioned implementation manner, in some possible implementation manners, the computing module includes a control node and a computing node, the computing module obtains the address of each storage pipe in the M storage pipes; the computing module determines the target task according to whether there is cached data in each storage pipe in the M storage pipes, and determines the target address corresponding to the target task; the computing module determines the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address and the address of each storage pipe in the M storage pipes, and obtains the target data from the target storage pipe, including: the control node obtains the M storage pipes The address of each storage pipe in the M storage pipes; according to whether there is cached data in each storage pipe in the M storage pipes, when there is cached data in the storage pipe, it is determined that there is a target task in the storage pipe; and according to the storage pipe corresponding to the target task, the target address of the storage pipe corresponding to the target task is determined from the address of each storage pipe in the M storage pipes pre-stored in the control node, and the target address is sent to the computing node, and the computing node determines the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address, and obtains the target data from the target storage pipe.

[0021] In summary, in the above scheme, before the data is stored in the database, a scheme for preprocessing and calculating the data is proposed. The data is classified by the gateway to obtain multiple groups of data of different types, and the multiple groups of data are sent to the cache module. The multiple storage pipelines in the cache module receive and cache the multiple groups of data. The computing module obtains the target data from the multiple groups of data and performs computing and processing on the target data. In the process of computing and processing the target data, by selecting a submodule with strong data processing capability as the target submodule, the target data is preprocessed by the target submodule, which ensures that the system's computing and processing capability for the target data is optimized, and the efficiency of the system in processing the target data is ensured. When the data computing capabilities of multiple submodules are less than the second preset threshold, it indicates that the computing capability of the computing module has reached a bottleneck. At this time, the computing module will stop obtaining the target data from the target storage pipeline, which can appropriately reduce the processing pressure of the computing module, avoid the computing pressure of the system caused by the data peak, and ensure the efficiency of the system in processing data. Furthermore, after pre-processing the target data to obtain the data to be stored, the computing module classifies the data to be stored into cold data and hot data, and stores the data to be stored in the corresponding data storage unit in the database according to the classification result. By classifying and storing the data to be stored, the pressure on the database caused by storing all data in one database is alleviated. At the same time, the user's needs can be met promptly and quickly when the user reads, thereby improving the user experience.

[0022] In a second aspect, a system is provided, characterized in that the system includes a gateway, a cache module, a computing module, and a database, the gateway is used to obtain multiple data, classify the multiple data according to the data format and / or the compilation language of the data, obtain M groups of data of different categories, and send the M groups of data to the cache module; the cache module includes M storage pipelines, and the M storage pipelines receive and cache the M groups of data sent by the gateway; the computing module is used to obtain target data from the M groups of data, and the target data is one or more groups of data in the M groups of data; the target data is preprocessed to obtain data to be stored, and the data to be stored is stored in the database.

[0023] In combination with the second aspect, in some possible implementations, the computing module includes multiple sub-modules, and the computing module is also used to obtain the data computing capability of each of the multiple sub-modules, and determine from the multiple sub-modules a sub-module whose data computing capability is greater than or equal to a first preset threshold as a target sub-module.

[0024] In combination with the second aspect and the above-mentioned implementation manner, in some possible implementation manners, the calculation module is specifically used to preprocess the target data through the target submodule to obtain the data to be stored.

[0025] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, when the data computing capability of each of the multiple sub-modules is less than or equal to a second preset threshold, the computing module is also used to stop obtaining the target data from the target storage pipeline.

[0026] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the computing module is specifically used to obtain the address of each storage pipe in the M storage pipes; determine the target task according to whether there is cached data in each storage pipe in the M storage pipes, and determine the target address corresponding to the target task; determine the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address and the address of each storage pipe in the M storage pipes, and obtain the target data from the target storage pipe.

[0027] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the database includes a cold data storage unit and a hot data storage unit, and the calculation module is also used to obtain the access frequency of data of each data type in the database, and divide the data to be stored into hot data or cold data according to the access frequency of data of each data type in the database and the data type of the data to be stored, wherein the data to be stored whose access frequency of data of any data type in the database is greater than or equal to a third preset threshold is determined to be hot data, otherwise it is determined to be cold data; the data type of the data to be stored that is hot data in the data to be stored is the same as the data type of the data that is hot data in the database, and the data type of the data to be stored that is cold data in the data to be stored is the same as the data type of the data that is cold data in the database.

[0028] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the computing module is specifically used to store the data to be stored that is hot data among the data to be stored in the hot data storage unit, and to store the data to be stored that is cold data among the data to be stored in the cold data storage unit.

[0029] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the system also includes a display module for receiving a user's query request, and in response to the query request, obtaining and displaying the stored data from the database for the user to read and view.

[0030] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the cache module includes one or more cache nodes, each of the one or more cache nodes corresponds to one or more storage pipes, and each of the one or more storage pipes is used to receive and cache any one or more groups of data among the M groups of data.

[0031] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the computing module includes a control node and a computing node, the control node is used to obtain the address of each storage pipe in the M storage pipes, determine whether there is cached data in each storage pipe in the M storage pipes, and when there is cached data in the storage pipe, determine that there is a target task in the storage pipe; and according to the storage pipe corresponding to the target task, determine the target address of the storage pipe corresponding to the target task from the addresses of each storage pipe in the M storage pipes pre-stored in the control node, and send the target address to the computing node; the computing node is used to determine the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address, and obtain the target data from the target storage pipe.

[0032] According to a third aspect, a data processing device is provided, characterized in that the device includes: a sending module, used to obtain multiple data, classify the multiple data according to the data format and / or the compilation language of the data, obtain M groups of data of different categories, and send the M groups of data to a cache module; a first processing module, used to receive and cache the M groups of data sent by the gateway; an acquisition module, used to obtain target data from the M groups of data; the target data is one or more groups of data in the M groups of data; and a second processing module, used to pre-process the target data to obtain data to be stored, and store the data to be stored in a database.

[0033] In combination with the third aspect, in some possible implementations, the computing module includes multiple sub-modules, and the device also includes: a third processing module, used to obtain the data computing capability of each of the multiple sub-modules, and determine from the multiple sub-modules a sub-module whose data computing capability is greater than or equal to a first preset threshold as a target sub-module.

[0034] In combination with the third aspect and the above-mentioned implementation manner, in some possible implementation manners, the second processing module is specifically used to pre-process the target data through the target sub-module to obtain the data to be stored.

[0035] In combination with the third aspect and the above-mentioned implementation methods, in some possible implementation methods, when the data computing capability of each of the multiple sub-modules is less than or equal to a second preset threshold, the device also includes: a first execution module, used to stop obtaining the target data from the target storage pipeline.

[0036] In combination with the third aspect and the above-mentioned implementation methods, in some possible implementation methods, the acquisition module is specifically used to obtain the address of each storage pipe in the M storage pipes; determine the target task according to whether there is cached data in each storage pipe in the M storage pipes, and determine the target address corresponding to the target task; according to the target address and the address of each storage pipe in the M storage pipes, determine the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe, and obtain the target data from the target storage pipe.

[0037] In combination with the third aspect and the above-mentioned implementation methods, in some possible implementation methods, the database includes a cold data storage unit and a hot data storage unit, and the device also includes: a fourth processing module, used to obtain the access frequency of data of each data type in the database, and according to the access frequency of data of each data type in the database and the data type of the data to be stored, the data to be stored is divided into hot data or cold data, wherein data whose access frequency of data of any data type in the database is greater than or equal to a third preset threshold is determined as hot data, otherwise it is determined as cold data; the data type of the data to be stored that is hot data in the data to be stored is the same as the data type of the data that is hot data in the database, and the data type of the data to be stored that is cold data in the data to be stored is the same as the data type of the data that is cold data in the database.

[0038] In combination with the third aspect and the above-mentioned implementation methods, in some possible implementation methods, the second processing module is specifically used to store the data to be stored that is hot data among the data to be stored in the hot data storage unit, and to store the data to be stored that is cold data among the data to be stored in the cold data storage unit.

[0039] In combination with the third aspect and the above-mentioned implementation methods, in some possible implementation methods, the device also includes: a second execution module, which is used to receive a user's query request, and in response to the query request, obtain and display the stored data from the database for the user to read and view.

[0040] In combination with the third aspect and the above-mentioned implementation methods, in some possible implementation methods, the cache module includes one or more cache nodes, each of the one or more cache nodes corresponds to one or more storage pipes, and each of the one or more storage pipes is used to receive and cache any one or more groups of data in the M groups of data.

[0041] In combination with the third aspect and the above-mentioned implementation methods, in some possible implementation methods, the computing module includes a control node and a computing node, and the acquisition module also includes a sending unit and an acquisition unit, the sending unit is used to obtain the address of each storage pipe in the M storage pipes, determine whether there is cached data in each storage pipe in the M storage pipes, and when there is cached data in the storage pipe, determine that there is a target task in the storage pipe; and according to the storage pipe corresponding to the target task, determine the target address of the storage pipe corresponding to the target task from the addresses of each storage pipe in the M storage pipes pre-stored in the control node, and send the target address to the acquisition unit; the acquisition unit is used to determine the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address, and obtain the target data from the target storage pipe.

[0042] In a fourth aspect, a device is provided, comprising a memory and a processor, wherein the memory is used to store instructions, and the processor is used to call and execute the instructions from the memory, so that the device executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0043] In a fifth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.

[0044] In a sixth aspect, a computer-readable medium is provided, wherein the computer-readable medium stores a computer program code, and when the computer program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a structural diagram of a system provided in an embodiment of the present application;

[0046] Figure 2 is a schematic flow chart of a data processing method provided in an embodiment of the present application;

[0047] Figure 3 It is a structural diagram of a data processing system provided in an embodiment of the present application;

[0048] Figure 4 It is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0049] Figure 5 It is a structural schematic diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0051] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0052] Figure 1 It is a structural diagram of a system provided in an embodiment of the present application.

[0053] For example, Figure 1 As shown, the system 100 includes:

[0054] The gateway 101 is used to obtain multiple data, classify the multiple data according to the data format and / or the compilation language of the data, obtain M groups of data of different categories, and send the M groups of data to the cache module;

[0055] The cache module 102 includes M storage pipes, and the M storage pipes receive and cache the M groups of data sent by the gateway;

[0056] The calculation module 103 is used to obtain target data from the M groups of data, where the target data is one or more groups of data in the M groups of data; preprocess the target data to obtain data to be stored, and store the data to be stored in the database 104.

[0057] In one possible implementation, the computing module 103 includes multiple sub-modules, and the computing module 103 is also used to obtain the data computing capability of each of the multiple sub-modules, and determine a sub-module from the multiple sub-modules whose data computing capability is greater than or equal to a first preset threshold as a target sub-module; through the target sub-module, the target data is preprocessed to obtain the data to be stored.

[0058] In a possible implementation, when the data computing capability of each of the multiple submodules is less than or equal to a second preset threshold, the computing module 103 is further configured to stop acquiring the target data from the storage pipeline.

[0059] In one possible implementation, the calculation module 103 is specifically used to obtain the address of each storage pipe in the M storage pipes; determine the target task according to whether there is cached data in each storage pipe in the M storage pipes, and determine the target address corresponding to the target task; determine the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address and the address of each storage pipe in the M storage pipes, and obtain the target data from the target storage pipe.

[0060] In a possible implementation, the database 104 includes a cold data storage unit and a hot data storage unit, and the calculation module 103 is further used to obtain the access frequency of data of each data type in the database 104, and classify the data to be stored into hot data or cold data according to the access frequency of data of each data type in the database 104 and the data type of the data to be stored, wherein data of any data type in the database 104 whose access frequency is greater than or equal to a third preset threshold is determined as hot data, otherwise it is determined as cold data; the data type of the data to be stored that is hot data in the data to be stored is the same as the data type of the data that is hot data in the database 104, and the data type of the data to be stored that is cold data in the data to be stored is the same as the data type of the data that is cold data in the database 104.

[0061] In a possible implementation, the calculation module 103 is specifically configured to store the data to be stored that is hot data in the hot data storage unit, and store the data to be stored that is cold data in the cold data storage unit.

[0062] Optionally, the system 100 further includes a display module for receiving a user's query request, and in response to the query request, acquiring and displaying the stored data from the database 104 for the user to read and view.

[0063] In one possible implementation, the cache module 102 includes one or more cache nodes, each of the one or more cache nodes corresponds to one or more storage pipes, and each of the one or more storage pipes is used to receive and cache any one or more groups of data among the M groups of data.

[0064] In one possible implementation, the computing module 103 includes a control node and a computing node. The control node is used to obtain the address of each storage pipe in the M storage pipes, determine whether there is cached data in each storage pipe in the M storage pipes, and when there is cached data in the storage pipe, determine that there is a target task in the storage pipe; and according to the storage pipe corresponding to the target task, determine the target address of the storage pipe corresponding to the target task from the addresses of each storage pipe in the M storage pipes pre-stored in the control node, and send the target address to the computing node; the computing node is used to determine the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address, and obtain the target data from the target storage pipe.

[0065] Figure 2 It is a schematic flow chart of a data processing method provided in an embodiment of the present application.

[0066] For example, Figure 2 As shown, the method 200 includes:

[0067] 201 , the gateway obtains multiple data, classifies the multiple data according to the data format and / or the compilation language of the data, obtains M groups of data of different categories, and sends the M groups of data to the cache module.

[0068] It should be understood that the multiple data obtained by the gateway are collected by the sensors of the smart device. Various types of sensors are integrated inside the smart device, such as image sensors, speed sensors, acceleration sensors, temperature sensors, etc. Among them, the image sensor can collect images of the environment around the smart device, the speed sensor can collect the current speed of the smart device, and the acceleration sensor can collect the acceleration of the current movement of the smart device. During the operation of the smart device, the above-mentioned multiple sensors can collect multiple data related to the smart device, and report the multiple data to the gateway. The gateway receives the multiple data, classifies the multiple data according to the data format and / or the compilation language of the data, and can obtain M groups of data of different categories. The gateway sends all the M groups of data to the cache module, wherein the M groups of data include at least one or more groups of data.

[0069] Figure 3 It is a structural diagram of a data processing system provided in an embodiment of the present application.

[0070] For example, Figure 3As shown, the smart device 301 reports all the multiple data collected by the sensor to the gateway 101. After receiving the multiple data reported by the smart device 301, the gateway 101 classifies the multiple data, wherein "□, △, ☆" respectively represent the multiple data collected by different types of sensors. After classifying the multiple data, the gateway 101 can obtain M groups of different data, and the gateway 101 sends the M groups of different data to the cache module 102.

[0071] It should be understood that when the gateway classifies multiple data, for sensors with the same function, the formats of the collected data may be different due to different production specifications of manufacturers or sensor parts.

[0072] Optionally, the number of gateways may be one, and the gateway may store all the acquired multiple data inside the gateway, and classify the multiple data inside the gateway to obtain M groups of data.

[0073] Optionally, when there are sensors with a relatively high collection frequency inside the smart device, the amount of data collected by these sensors is relatively large. In the case of large amounts of data, the number of gateways can be set to multiple to reduce the pressure on the gateway to obtain data.

[0074] It should be understood that when there are multiple gateways, there is not a one-to-one correspondence between the number of gateways and the number of sensors. For example, the acquisition frequency of the image sensor is relatively high, so the amount of data collected is large, and it needs to be stored separately by the first gateway. For the velocity sensor and the acceleration sensor, the acquisition frequency is relatively low, so the amount of data obtained is relatively small, and the data collected by these two sensors can be stored together in the second gateway.

[0075] 202. The cache module includes M storage pipelines, and the M storage pipelines receive and cache M groups of data sent by the gateway.

[0076] The gateway sends the M groups of data obtained after classification to the cache module, and the cache module receives the M groups of data. The cache module includes multiple storage pipelines. The gateway sends the M groups of data obtained after classification to different storage pipelines, and the storage pipelines are used to receive and cache the data sent by the gateway.

[0077] The M storage pipelines include more than one storage pipeline. In addition, the cache module includes one or more cache nodes, each of the one or more cache nodes includes one or more storage pipelines, and each of the one or more storage pipelines is used to receive and cache any one or more groups of data in the M groups of data.

[0078] For example, Figure 3As shown, in the cache module 102, a cache node 1021 is set, and multiple storage pipes 10211 are set under each cache node 1021. When the gateway 101 sends M groups of data to the cache module 102, the M storage pipes 10211 in the cache module 102 are responsible for receiving and caching the M groups of data, and each storage pipe 10211 is responsible for receiving and caching any one or more groups of data in the M groups of data.

[0079] Optionally, when the cache module includes multiple nodes, some cache nodes among the multiple nodes are determined as primary cache nodes, and other cache nodes in the cache module except the primary cache node are determined as backup cache nodes. It should be noted that each primary cache node has only one backup cache node corresponding to it, and the backup cache node is responsible for obtaining all data in the primary cache node and receiving and caching the same data as the primary cache node.

[0080] Optionally, each storage pipeline can be set with multiple replica storage pipelines, and each storage pipeline is in a master-slave relationship with the multiple replica storage pipelines. When the storage pipeline stores any one or more groups of data among the M groups of data, the multiple replica storage pipelines will also synchronously obtain the same data as the primary storage pipeline.

[0081] In the above technical solution, by setting a backup cache node, the backup cache node is enabled to synchronously obtain data that is exactly the same as the primary cache node, and by setting multiple replica storage pipelines for a storage pipeline, the replica storage pipeline is enabled to synchronously obtain data that is exactly the same as the primary storage pipeline. This can avoid sudden interruption of the data processing system when the primary cache node or the primary storage pipeline fails or fails, thereby effectively ensuring the efficiency of the system in processing data.

[0082] 203, the calculation module obtains target data from the M groups of data, where the target data is one or more groups of data in the M groups of data.

[0083] Specifically, when the computing module obtains one or more groups of data as target data from M groups of data, the computing module includes the following steps:

[0084] (1) The computing module obtains the address of each storage pipeline in the M storage pipelines;

[0085] (2) The computing module determines the target task according to whether there is cached data in each of the M storage pipes, and determines the target address corresponding to the target task;

[0086] (3) The calculation module determines the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address and the address of each storage pipe in the M storage pipes, and obtains the target data from the target storage pipe.

[0087] It should be understood that when the gateway sends M groups of data to the cache module, each group of data corresponds to an identifier. When a storage pipe in the cache module receives and caches one or more groups of data in the M groups of data, it will simultaneously obtain the identifiers of the one or more groups of data. The type of data in the M groups of data specifically stored in the storage pipe can be determined through the identifier. At the same time, each group of data corresponds to an address in the cache module, and the computing module can obtain the address of each storage pipe in the M storage pipes.

[0088] It should also be understood that the M storage pipelines and the M groups of data are not in a one-to-one correspondence, and one storage pipeline among the M storage pipelines can simultaneously receive and cache one or more groups of data among the M groups of data, so there may be one or more storage pipelines among the M storage pipelines that do not receive and cache one or more groups of data among the M groups of data. Therefore, the computing module can determine the target task and the target address of the target task by whether there is cached data in each storage pipeline among the M storage pipelines.

[0089] Furthermore, the computing module is divided into two parts: a control node and a computing node. The control node and the computing node are in a subordinate relationship, and multiple computing nodes can be set under one control node. The computing module obtains the address of each storage pipeline in the M storage pipelines; the computing module determines the target task according to whether there is cached data in each storage pipeline in the M storage pipelines, and determines the target address corresponding to the target task; the computing module determines the storage pipeline corresponding to the target address from the M storage pipelines as the target storage pipeline according to the target address and the address of each storage pipeline in the M storage pipelines, and obtains the target data from the target storage pipeline, which can be specifically expressed as:

[0090] (1) The control node obtains the address of each storage pipeline in the M storage pipelines, determines whether each storage pipeline in the M storage pipelines has cached data, and when there is cached data in the storage pipeline, determines that the storage pipeline has a target task, and determines the target address of the storage pipeline corresponding to the target task from the addresses of each storage pipeline in the M storage pipelines pre-stored in the control node according to the storage pipeline corresponding to the target task, and sends the target address to the computing node;

[0091] (2) The computing node determines the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address, and obtains the target data from the target storage pipe.

[0092] Specifically, the control node determines the target task according to whether each storage pipe in the M storage pipes has cached data, and after obtaining the target address corresponding to the target task according to the target task, determines the target address of the storage pipe corresponding to the target task according to the address of each storage pipe in the M storage pipes stored in advance, and creates multiple task processes according to the different storage pipes corresponding to the target tasks. Each of the multiple task processes corresponds to the data of a storage pipe of the target task and the target address of the storage pipe. At the same time, each task process can parse a target task, and generate multiple sub-task processes in multiple computing nodes under the control node according to the parsing results.

[0093] For example, Figure 3 As shown, a control node 1031 and a computing node 1032 are provided in the computing module 103, wherein the control node 1031 determines the target task according to whether each storage pipe in the M storage pipes has cached data, and after obtaining the target address corresponding to the target task according to the target task, determines the target address of the storage pipe corresponding to the target task according to the address of each storage pipe in the M storage pipes stored in advance, and creates multiple task processes according to the different storage pipes corresponding to the targets, each task process 10311 in the multiple subtask processes corresponds to the data of a storage pipe of the target task and the target address of the storage pipe, and each task process can parse a target task, and generate multiple subtask processes 10321 in the multiple computing nodes under the control node according to the parsing result.

[0094] Each task process can send the target address of the corresponding storage pipe to the subtask process in the computing node. One computing node corresponds to one subtask process of the task process, and data can be transmitted between multiple subtask processes. The multiple subtask processes in the computing node are used to determine the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address of the storage pipe corresponding to each task process, and obtain the target data from the target storage pipe.

[0095] It should be understood that the multiple subtask processes include "acquisition subtask process" and "computation subtask process". Among them, the "acquisition subtask process" is used to accept the target address of the target task sent by the task process in the control node, determine the target storage pipeline corresponding to the target address according to the target address, and obtain the target data from the target storage pipeline, and transmit the target data to the "computation subtask process"; the "computation subtask process" is responsible for receiving the target data sent by the "acquisition subtask process" and performing calculation processing on the target data. The sum of the calculation results of multiple "computation subtask processes" corresponds to the processing calculation result of the target data of a target task, and each task process is mainly responsible for monitoring the calculation processing status of multiple "computation subtask processes" under the task process.

[0096] Furthermore, in the computing node, each "computing subtask process" can be regarded as a submodule. When multiple submodules are processing and calculating the target data, the task process of the control node can obtain the data computing capability of each of the multiple submodules. The task process determines the submodule with a data computing capability greater than or equal to the first preset threshold from the multiple submodules as the target submodule; the target data is processed and calculated through the target submodule. In other words, when there is a "computing subtask process" with a data processing capability less than the first preset threshold, the task process in the control node will allocate the computing processing task of the target data corresponding to the "computing subtask process" with a data processing capability less than the first preset threshold to other target submodules with a data computing capability greater than or equal to the first preset threshold.

[0097] Furthermore, when the data computing capability of each of the multiple submodules is less than or equal to the second preset threshold, it indicates that the processing speed of the multiple subtask processes for the target data has slowed down, and the target data may be accumulated in the multiple submodules. At this time, the task process responsible for monitoring in the control node will generate a target instruction, and the target instruction is used to stop the "acquisition subtask process" from continuing to obtain the target data from the target storage pipeline. The task process sends the target instruction to the "acquisition subtask process" in the computing node, and after receiving the target instruction, the "acquisition subtask process" stops continuing to obtain the target data from the target storage pipeline.

[0098] In the above scheme, by selecting a submodule with strong data processing capability as the target submodule, the target submodule performs preprocessing on the target data, thereby ensuring that the system optimizes the computing processing capability of the target data and the efficiency of the system in processing the target data. When the data computing capabilities of multiple submodules are all less than the second preset threshold, it indicates that the computing capabilities of multiple submodules have reached a bottleneck. At this time, the computing module will stop acquiring the target data from the target storage pipeline, which can appropriately reduce the processing pressure of the computing module, avoid the computing pressure of the system caused by the data peak, and ensure the efficiency of the system in processing data.

[0099] Optionally, multiple backup computing nodes can be set for a computing node. The computing node that actually performs computing processing on the target data is determined as the main computing node, and the other computing nodes are all backup computing nodes. The backup nodes will realize synchronous data transmission during the process of the main computing node calculating the target data.

[0100] Optionally, the number of control nodes may be multiple. When the number of control nodes is multiple, the multiple control nodes are in a master-slave relationship, that is, one of the multiple control nodes is a master control node. During data processing, only the master control node interacts with other modules in the system, and the remaining control nodes are standby control nodes. When the master control node transmits data, the remaining control nodes also transmit data synchronously, serving as standby control nodes of the master control node.

[0101] In the above technical solution, by backing up the computing nodes and control nodes, the backup control node obtains exactly the same data as the main control node, and the backup computing node synchronously obtains all the data of the main computing node, thereby solving the problem of suspension of the data processing system due to the failure of a computing node and / or a control node, and ensuring the efficient use of the data processing system.

[0102] 204, the computing module pre-processes the target data to obtain data to be stored, and stores the data to be stored in the database.

[0103] After the "computing subtask process" in the computing node completes the calculation and processing of the target data, the data to be stored is obtained. The computing module can determine the specific type of the data to be stored and store the data to be stored in the corresponding database according to the determined type.

[0104] Specifically, the database includes a cold data storage unit and a hot data storage unit. After obtaining the data to be stored, when storing the data to be stored in the database, the specific steps are as follows:

[0105] (1) The calculation module obtains the access frequency of each data type in the database;

[0106] (2) the calculation module classifies the data to be stored into hot data or cold data according to the access frequency of data of each data type in the database and the data type of the data to be stored, wherein data of any data type in the database whose access frequency is greater than or equal to a third preset threshold is determined as hot data, otherwise it is determined as cold data; the data type of the data to be stored that is hot data is the same as the data type of the data in the database that is hot data; the data type of the data to be stored that is cold data is the same as the data type of the data in the database that is cold data;

[0107] (3) The computing module stores the data to be stored that is hot data in the hot data storage unit, and stores the data to be stored that is cold data in the cold data storage unit.

[0108] For example, Figure 3 As shown, when the computing module 103 stores the data to be stored in the database 104, the subtask process 10321 in the computing node 1032 can directly connect to the hot data storage unit 1041 and the cold data storage unit 1042 in the database 104, thereby transferring the data to be stored to different types of data storage units according to the data type of the data to be stored outputted from the computing module 103, thereby completing the storage of different types of data to be stored.

[0109] In the above scheme, by classifying and storing the preprocessed data, the classified management of the data to be stored can be achieved, and the storage management of the data is more efficient and faster. At the same time, it can also alleviate the storage pressure on the database when all data is stored in one database.

[0110] Furthermore, after the data to be stored is stored in the database, the display module receives a query request from the user. In response to the query request, the display module obtains and displays the stored data from the database for the user to read and view.

[0111] For example, Figure 3 As shown, the display module 302 obtains corresponding stored data from the corresponding hot data storage unit 1041 or cold data storage unit 1042 in the database 104 according to the viewing and reading requirements of the user, and displays the obtained stored data.

[0112] In summary, in the above scheme, before the data is stored in the database, a scheme for preprocessing and calculating the data is proposed, the data is classified by the gateway to obtain multiple groups of data, the multiple groups of data are sent to the cache module, the multiple storage pipelines in the cache module receive and cache the multiple groups of data sent by the gateway, the calculation module obtains the target data, and performs calculation processing on the target data. In the process of calculating and processing the target data. By selecting a sub-module with strong data processing capability as the target sub-module, the target data is pre-processed by the target sub-module, which ensures that the system's calculation processing capability for the target data is optimized, and the efficiency of the system in processing the target data is ensured. When the data calculation capabilities of multiple sub-modules are less than the second preset threshold, it indicates that the calculation capability of the calculation module has reached a bottleneck. At this time, the calculation module will stop obtaining the target data from the target storage pipeline, which can appropriately reduce the processing pressure of the calculation module, avoid the calculation pressure of the system caused by the data peak, and ensure the efficiency of the system in processing data. Furthermore, after pre-processing the target data to obtain the data to be stored, the computing module classifies the data to be stored into cold data and hot data, and stores the data to be stored in the corresponding data storage unit in the database according to the classification result. By classifying and storing the data to be stored, the pressure on the database caused by storing all data in one database is alleviated. At the same time, the user's needs can be met promptly and quickly when the user reads, thereby improving the user experience.

[0113] Figure 4 It is a structural diagram of a data processing device provided in an embodiment of the present application.

[0114] For example, Figure 4 As shown, the device 400 includes:

[0115] The sending module 401 is used to obtain multiple data, classify the multiple data according to the data format and / or the compilation language of the data, obtain M groups of data of different categories, and send the M groups of data to the cache module;

[0116] A first processing module 402, configured to receive and cache the M groups of data sent by the gateway;

[0117] An acquisition module 403 is used to acquire target data from the M groups of data, where the target data is one or more groups of data in the M groups of data;

[0118] The second processing module 404 is used to pre-process the target data to obtain data to be stored, and store the data to be stored in the database.

[0119] Optionally, when the computing module includes multiple sub-modules, the device also includes: a third processing module, used to obtain the data computing capability of each of the multiple sub-modules, and determine from the multiple sub-modules a sub-module whose data computing capability is greater than or equal to a first preset threshold as a target sub-module.

[0120] In a possible implementation, the second processing module 404 is specifically configured to pre-process the target data through the target submodule to obtain the data to be stored.

[0121] Optionally, when the data computing capability of each of the multiple sub-modules is less than or equal to a second preset threshold, the device further includes: a first execution module, configured to stop acquiring the target data from the target storage pipeline.

[0122] In a possible implementation, the acquisition module 403 is specifically used to obtain the address of each storage pipe in the M storage pipes; determine the target task according to whether there is cached data in each storage pipe in the M storage pipes, and determine the target address corresponding to the target task; determine the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address and the address of each storage pipe in the M storage pipes, and obtain the target data from the target storage pipe.

[0123] Optionally, when the database includes a cold data storage unit and a hot data storage unit, the device also includes: a fourth processing module, used to obtain the access frequency of data of each data type in the database, and classify the data to be stored into hot data or cold data according to the access frequency of data of each data type in the database and the data type of the data to be stored, wherein data whose access frequency of data of any data type in the database is greater than or equal to a third preset threshold is determined as hot data, otherwise it is determined as cold data; the data type of the data to be stored that is hot data in the data to be stored is the same as the data type of the data that is hot data in the database, and the data type of the data to be stored that is cold data in the data to be stored is the same as the data type of the data that is cold data in the database.

[0124] In a possible implementation, the second processing module 404 is specifically configured to store the data to be stored that is hot data in the hot data storage unit, and store the data to be stored that is cold data in the cold data storage unit.

[0125] Optionally, the device further includes: a second execution module, configured to receive a query request from a user, and in response to the query request, obtain and display the stored data from the database for the user to read and view.

[0126] In one possible implementation, the cache module includes one or more cache nodes, each of the one or more cache nodes corresponds to one or more storage pipes, and each of the one or more storage pipes is used to receive and cache any one or more groups of data among the M groups of data.

[0127] In a possible implementation, the computing module includes a control node and a computing node, and the acquisition module 403 also includes a sending unit and an acquisition unit, the sending unit is used to obtain the address of each storage pipe in the M storage pipes, determine whether there is cached data in each storage pipe in the M storage pipes, and when there is cached data in the storage pipe, determine that there is a target task in the storage pipe; and according to the storage pipe corresponding to the target task, determine the target address of the storage pipe corresponding to the target task from the addresses of each storage pipe in the M storage pipes pre-stored in the control node, and send the target address to the acquisition unit; the acquisition unit is used to determine the storage pipe corresponding to the target address from the M storage pipes as the target storage pipe according to the target address, and acquire the target data from the target storage pipe.

[0128] Figure 5 It is a structural schematic diagram of a device provided in an embodiment of the present application.

[0129] For example, Figure 5 As shown, the device 500 includes: a memory 501 and a processor 502, wherein the memory 501 stores an instruction 5011, and the processor 502 is used to call and execute the instruction 5011 to perform a data processing method.

[0130] In this embodiment, the functional modules of the device can be divided according to the above method example. For example, each functional module can be corresponded, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0131] In the case of dividing each functional module according to each function, the device may include: a sending module, a first processing module, an acquisition module, a second processing module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0132] The device provided in this embodiment is used to execute the above-mentioned data processing method, and thus can achieve the same effect as the above-mentioned implementation method.

[0133] In the case of an integrated unit, the device may include a processing module and a storage module. The processing module may be used to control and manage the actions of the device. The storage module may be used to support the device to execute mutual program codes and data.

[0134] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits disclosed in the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0135] This embodiment further provides a computer-readable storage medium, in which a computer program code is stored. When the computer program code is executed on a computer, the computer executes the above-mentioned related method steps to implement a data processing method in the above-mentioned embodiment.

[0136] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement a data processing method in the above-mentioned embodiment.

[0137] In addition, the device provided in the embodiments of the present application may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store instructions, and when the device is running, the processor may call and execute instructions so that the chip executes a data processing method in the above-mentioned embodiments.

[0138] Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0139] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0140] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: Applied to a system, the system includes a gateway, a cache module, a computing module and a database, the method includes: The gateway acquires a plurality of data, classifies the plurality of data according to the data format and / or the compilation language of the data, obtains M groups of data of different categories, and sends the M groups of data to the cache module; The cache module includes M storage pipes, and the M storage pipes receive and cache the M groups of data sent by the gateway; The calculation module obtains target data from the M groups of data, wherein the target data is one or more groups of data in the M groups of data, wherein the calculation module includes multiple submodules; The calculation module obtains the data calculation capability of each of the multiple submodules, and determines a submodule whose data calculation capability is greater than or equal to a first preset threshold from the multiple submodules as a target submodule; Preprocessing the target data through the target submodule to obtain data to be stored, and storing the data to be stored in the database; When the data computing capability of each of the multiple sub-modules is less than or equal to a second preset threshold, the computing module stops acquiring the target data from the target storage pipeline.

2. The method according to claim 1, characterized in that The calculation module obtains target data from the M groups of data, including: The calculation module obtains the address of each storage pipe in the M storage pipes; The calculation module determines a target task according to whether there is cached data in each of the M storage pipes, and determines a target address corresponding to the target task; The calculation module determines the storage pipe corresponding to the target address as the target storage pipe from the M storage pipes according to the target address and the address of each storage pipe in the M storage pipes, and obtains the target data from the target storage pipe.

3. The method according to claim 1, characterized in that: The database includes a cold data storage unit and a hot data storage unit, and the method further includes: The calculation module obtains the access frequency of data of each data type in the database, and divides the data to be stored into hot data and cold data according to the access frequency of data of each data type in the database and the data type of the data to be stored, wherein data whose access frequency of data of any data type in the database is greater than or equal to a third preset threshold is determined as hot data, otherwise it is determined as cold data; the data type of the data to be stored that is hot data among the data to be stored is the same as the data type of the data that is hot data in the database, and the data type of the data to be stored that is cold data among the data to be stored is the same as the data type of the data that is cold data in the database; and storing the data to be stored in the database includes: The calculation module stores the data to be stored that is hot data among the data to be stored in the hot data storage unit, and stores the data to be stored that is cold data among the data to be stored in the cold data storage unit.

4. The method according to claim 1, characterized in that: The system further includes a display module, and the method further includes: The display module receives a query request from a user, and in response to the query request, the display module obtains and displays the stored data from the database for the user to read and view.

5. The method according to claim 1, characterized in that: The cache module includes one or more cache nodes, each of the one or more cache nodes corresponds to one or more storage pipelines, and each of the storage pipelines is used to receive and cache any one or more groups of data in the M groups of data.

6. The method according to claim 2, characterized in that The computing module includes a control node and a computing node, and the computing module obtains the address of each storage pipe in the M storage pipes; the computing module determines a target task according to whether there is cached data in each storage pipe in the M storage pipes, and determines a target address corresponding to the target task; the computing module determines a storage pipe corresponding to the target address from the M storage pipes as a target storage pipe according to the target address and the address of each storage pipe in the M storage pipes, and obtains the target data from the target storage pipe, including: The control node obtains the address of each storage pipeline in the M storage pipelines; determines whether there is cached data in each storage pipeline in the M storage pipelines, and when there is cached data in the storage pipeline, determines whether there is a target task in the storage pipeline; and determines the target address of the storage pipeline corresponding to the target task from the address of each storage pipeline in the M storage pipelines pre-stored in the control node according to the storage pipeline corresponding to the target task, and sends the target address to the computing node; The computing node determines, according to the target address, a storage pipe corresponding to the target address from the M storage pipes as a target storage pipe, and obtains the target data from the target storage pipe.

7. A system, characterized in that: The system includes a gateway, a cache module, a computing module, and a database, wherein the computing module includes multiple submodules. The gateway is used to obtain multiple data, classify the multiple data according to the data format and / or the compilation language of the data, obtain M groups of data of different categories, and send the M groups of data to the cache module; The cache module includes M storage pipes, and the M storage pipes receive and cache the M groups of data sent by the gateway; The calculation module is used to obtain target data from the M groups of data; obtain the data calculation capability of each of the multiple submodules, and determine a submodule whose data calculation capability is greater than or equal to a first preset threshold from the multiple submodules as a target submodule, preprocess the target data through the target submodule to obtain the data to be stored, and store the data to be stored in the database; when the data calculation capability of each of the multiple submodules is less than or equal to a second preset threshold, the calculation module stops obtaining the target data from the target storage pipeline.

8. A device, characterized in that: The device comprises: A memory for storing instructions; A processor, configured to call and execute the instructions from the memory, so that the device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 6 is implemented.

Images