Data processing method, device and equipment, computer storage medium and vehicle

Abstract

The invention provides a data processing method, device and equipment, a computer storage medium and a vehicle, and relates to the technical field of big data processing. The method comprises the steps that first requests of M pieces of callback information in one-to-one correspondence with M pieces of data are sent to a storage server, a time sequence database is deployed on the storage server, and M is a positive integer; the M pieces of callback information sent by the storage server based on the time sequence database are received, the M pieces of callback information are determined by the time sequence database based on time sequence index information, and the time sequence index information comprises at least one of an equipment list and a time interval; and distributing the M pieces of callback information to each distributed computing node in a distributed computing node cluster, so that each distributed computing node respectively executes target data operation. According to the embodiment of the invention, data processing can be realized more efficiently and economically.

Description

Technical Field

[0001] The present application belongs to the field of big data processing technology, and in particular, relates to a data processing method, device, equipment, computer storage medium and vehicle. Background Art

[0002] With the continuous development of informatization and the advent of the technological era, being able to efficiently process and utilize the increasing amount of massive data generated in the process of informatization is of great significance to information construction and data application. At present, in the process of data storage and subsequent data processing applications, storage servers usually rely on serverless computing functions to implement file storage partition splitting. Specifically, the storage server monitors the file upload events in the object storage through the serverless computing function, and then splits the data files reported by each device into different storage partitions according to certain rules based on the device-side identifier (such as device code). In this way, based on the above storage partitioning strategy, a data processing task can be split into multiple tasks for parallel processing, and each task only needs to serve and process the data files in the corresponding storage partition.

[0003] However, since the above-mentioned solution relies on the construction of serverless computing functions, the technical solution is relatively complex, and therefore often requires additional data link construction costs. In addition, the data file monitoring path that the serverless computing function relies on is actually highly coupled with the data file upload path, and needs to be adjusted in time with the data file reporting path on the device side, which is not convenient for expansion. When performing data analysis and calculation and related data statistics on the stored data, the specific file data under all file paths in a single storage partition in the above-mentioned solution needs to be recursively scanned, and the statistical and data processing performance and efficiency are relatively low. In view of the above, the industry is still in urgent need of a new data processing solution to more efficiently and economically realize data processing operations such as data storage, query, filtering, analysis and statistics. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device, computer storage medium and vehicle, which can more efficiently and economically implement data processing operations such as data loading and storage.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the data processing method comprising:

[0006] Send a first request for M callback information corresponding to M data one by one to a storage server, where a time series database is deployed on the storage server, and M is a positive integer;

[0007] Receive M callback information sent by the storage server based on the time series database, where the M callback information is determined by the time series database based on the time series index information, and the time series index information includes at least one of a device list and a time interval;

[0008] The M callback information is distributed to each distributed computing node in the distributed computing node cluster, so that each distributed computing node executes the target data operation respectively.

[0009] In some possible implementations, the M callback information include device codes and data collection times corresponding to the M callback information one by one;

[0010] Distribute the M callback information to each distributed computing node in the distributed computing node cluster, including:

[0011] According to the device codes and data collection times corresponding to the M callback information, the M callback information are distributed to each distributed computing node.

[0012] In some possible implementations, the M callback information also includes storage paths of M data corresponding one-to-one to the M callback information; after receiving the M callback information sent by the storage server based on the time series database, the data processing method further includes:

[0013] According to the storage paths of the N data corresponding to the N callback information, the N data corresponding to the N callback information are obtained from the object storage service end.

[0014] Analyze and calculate N data to obtain the target result;

[0015] The M callback information includes N callback information, where N is a positive integer less than or equal to M.

[0016] In a second aspect, an embodiment of the present application provides a data processing method, the data processing method comprising:

[0017] Receive a first request from the distributed computing node cluster for M callback information;

[0018] By querying the time series index information in the time series database, M callback information is determined, where the time series index information includes at least one of a device list and a time interval; the time series database stores M callback information, the M callback information corresponds to the M data one by one, and M is a positive integer;

[0019] Send M callback messages to the distributed computing node cluster.

[0020] In some possible implementations, the M callback information include device codes and data collection times corresponding to the M callback information one by one.

[0021] In some possible implementations, before the storage server receives a first request from the distributed computing node cluster for M callback information, the data processing method further includes:

[0022] Receive M callback messages sent by the application server through the consumer program instance;

[0023] According to the device codes and data collection times corresponding to the M callback information, the M callback information is distributedly stored in the time series database;

[0024] Establish timing index information corresponding to the M callback information.

[0025] In some possible implementations, the M callback information further include storage paths of M data corresponding one-to-one to the M callback information.

[0026] In a third aspect, an embodiment of the present application provides a data processing method, the data processing method comprising:

[0027] The consumer program instance consumes M callback information from the message queue. The consumer program instance is deployed on the application server. The M callback information corresponds to the M data one by one, and M is a positive integer.

[0028] The consumed M callback information is sent to the storage server, so that the storage server performs distributed storage on the M callback information in the time series database, and the time series database is deployed on the storage server.

[0029] In some possible implementations, sending the consumed M callback information to the storage server includes:

[0030] According to the data format requirements supported by the time series database, the M callback information after consumption is structured respectively;

[0031] The structured M callback information is sent to the storage server.

[0032] In a fourth aspect, an embodiment of the present application provides a data processing method, the data processing method comprising:

[0033] Store M data uploaded by at least one device, where M is a positive integer;

[0034] Generate M callback information corresponding to the M data one by one;

[0035] Synchronize the M callback information to the message queue, so that the application server consumes the M callback information from the message queue through the consumer program instance, and sends the consumed M callback information to the storage server, so that the storage server performs distributed storage of the M callback information in the time series database; wherein the storage server is deployed with the time series database, and the application server is deployed with the consumer program instance.

[0036] In a fifth aspect, an embodiment of the present application provides a data processing device, the data processing device comprising:

[0037] A first sending module is used to send a first request for M callback information corresponding to M data one by one to a storage server, where a time series database is deployed on the storage server, and M is a positive integer;

[0038] A first receiving module is used to receive M callback information sent by the storage server based on the time series database, the time series database determines the M callback information based on the time series index information, and the time series index information includes at least one of a device list and a time interval;

[0039] The first allocation module is used to allocate M callback information to each distributed computing node in the distributed computing node cluster, so that each distributed computing node executes the target data operation respectively.

[0040] In a sixth aspect, an embodiment of the present application provides a data processing device, the data processing device comprising:

[0041] A second receiving module is used to receive a first request from the distributed computing node cluster for M callback information;

[0042] A first determination module is used to determine M callback information by querying the time series index information in the time series database, where the time series index information includes at least one of a device list and a time interval; the time series database stores M callback information, the M callback information corresponds to M data one by one, and M is a positive integer;

[0043] The second sending module is used to send M callback information to the distributed computing node cluster.

[0044] In a seventh aspect, an embodiment of the present application provides a data processing device, the data processing device comprising:

[0045] The first consumption module is used to consume M callback information from the message queue through the consumption program instance, where the M callback information corresponds to the M data one by one, and M is a positive integer;

[0046] The third sending module is used to send the consumed M callback information to the storage server, so that the storage server performs distributed storage on the M callback information in the time series database, and the time series database is deployed on the storage server.

[0047] In an eighth aspect, an embodiment of the present application provides a data processing device, the data processing device comprising:

[0048] A first storage module is used to store M data uploaded by at least one device, where M is a positive integer;

[0049] The first generating module is used to generate M callback information corresponding to the M data one by one;

[0050] The first synchronization module is used to synchronize the M callback information to the message queue, so that the application server consumes the M callback information from the message queue through the consumer program instance, and sends the consumed M callback information to the storage server, so that the storage server performs distributed storage of the M callback information in the time series database;

[0051] Among them, a time series database is deployed on the storage server, and a consumer program instance is deployed on the application server.

[0052] In a ninth aspect, an embodiment of the present application provides a data processing device, the data processing device comprising:

[0053] a processor and a memory storing computer program instructions;

[0054] When the processor executes the computer program instructions, it implements the data processing method provided in any one of the above-mentioned embodiments of the present application.

[0055] In the tenth aspect, an embodiment of the present application provides a computer storage medium, on which computer readable storage medium is stored computer program instructions, and when the computer program instructions are executed by a processor, a data processing method as provided in any one of the above-mentioned embodiments of the present application is implemented.

[0056] In the eleventh aspect, an embodiment of the present application provides a vehicle, which includes at least one of the following: the data processing device provided in the above-mentioned embodiment of the device of the present application; the data processing equipment provided in the above-mentioned embodiment of the present application; the computer-readable storage medium provided in the above-mentioned embodiment of the present application.

[0057] The data processing method, device, equipment, computer storage medium and vehicle of the embodiment of the present application receive and store the data uploaded by each device through the object storage service end, and synchronize the callback information corresponding to the data generated after storage to the message queue. The consumer program instance deployed on the application server can consume the callback information from the message queue and send it to the time series database in the storage server. The time series database performs distributed storage on the received callback information and establishes the time series index information corresponding to the callback information. In this way, when the distributed computing node cluster needs to perform data loading analysis, by requesting the callback information from the storage server, the time series database in the storage server queries the requested callback information based on the established time series index information and sends it to the distributed computing node cluster. After receiving the callback information sent by the storage server based on the time series database, the distributed computing node cluster splits the callback information to different computing nodes so that each computing node performs corresponding data processing according to the allocated callback information.

[0058] Compared with the prior art, the data processing method, device, equipment, computer storage medium and vehicle provided by the embodiment of the present application, on the one hand, do not need to rely on the construction of serverless computing functions for data storage and partitioning, which not only reduces the implementation cost of the solution, but also avoids the problem of high coupling between the data file monitoring path and the data file upload path that the original serverless computing function relies on. On the other hand, since the actual data is only stored on the object storage service side, and the data processing links involved in the message queue, consumer program instance and time series database are only lightweight callback information processing, the efficiency of data information processing is greatly improved. In addition, it stores callback information through a time series database, and can make full use of the time series database to establish a time series index for callback information. Therefore, in the subsequent data analysis and processing process of the distributed computing node cluster, the required callback information can be quickly queried through the time series index without recursively scanning the specific file data under all file paths, and the data processing performance and efficiency are greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is one of the flowcharts of the data processing method provided in one embodiment of the present application;

[0060] Figure 2 This is the second flow chart of the data processing method provided by an embodiment of the present application;

[0061] Figure 3 This is a third flowchart of a data processing method provided in an embodiment of the present application;

[0062] Figure 4 This is a fourth flowchart of a data processing method provided in an embodiment of the present application;

[0063] Figure 5 This is one of the flowcharts of a scenario implementation of a data processing method provided by an embodiment of the present application;

[0064] Figure 6 This is a second flow chart of a scenario implementation of a data processing method provided by an embodiment of the present application;

[0065] Figure 7 It is one of the structural schematic diagrams of a data processing device provided in one embodiment of the present application;

[0066] Figure 8 This is a second structural diagram of a data processing device provided by an embodiment of the present application;

[0067] Fig. 9 This is a third structural diagram of a data processing device provided in an embodiment of the present application;

[0068] Fig.10 This is a fourth structural diagram of a data processing device provided in an embodiment of the present application;

[0069] Fig.11 It is a structural diagram of a data processing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0071] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0072] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0073] In view of the above, in order to solve the problems of the prior art, the embodiments of the present application provide a data processing method, device, equipment, computer storage medium and vehicle. The data processing method provided by the embodiments of the present application is first introduced below.

[0074] Figure 1 One of the flow charts of the data processing method provided by an embodiment of the present application is shown. Figure 1 The data processing method shown can be applied to a distributed computing node cluster. Figure 1 As shown, the data processing method provided in the embodiment of the present application includes the following steps:

[0075] S110, sending a first request for M callback information corresponding to M data one by one to a storage server, where a time series database is deployed on the storage server, and M is a positive integer;

[0076] S120, receiving M callback information sent by the storage server based on the time series database, where the M callback information is determined by the time series database based on the time series index information, and the time series index information includes at least one of a device list and a time interval;

[0077] S130, distributing the M callback information to each distributed computing node in the distributed computing node cluster, so that each distributed computing node executes the target data operation respectively.

[0078] A data processing method provided by an embodiment of the present application includes a distributed computing node cluster sending a request for callback information to a storage server, and then receiving callback information sent by the storage server based on a time series database, the callback information is determined by the time series database based on time series index information, and finally the received callback information is split to different computing nodes in the cluster, so that each computing node performs data parsing calculations according to the allocated callback information. A data processing method of an embodiment of the present application does not need to recursively scan all file paths in the object storage locally, but completes the rapid loading and acquisition of callback information of data required in the data calculation process based on the time series index information established by the time series database, so that data processing performance and efficiency are greatly improved.

[0079] The specific implementation of the above steps 110 to 130 is described in detail below.

[0080] In S110, when implemented specifically, the distributed computing node cluster may establish communication with the storage server in advance. When data processing is required, the distributed computing node cluster sends a first request for M callback information corresponding to the M data to the storage server, so as to request the storage server for the M callback information. A time series database is specifically deployed on the storage server, and the M callback information may be pre-stored in the time series database.

[0081] The first request may be in the form of code, character string, or text, etc., and this application does not impose any restrictions on this.

[0082] And, the first request may specifically include the relevant information of the M callback information. For example, if the M callback information is stored in the time partition corresponding to a certain day in the time series database, the first request may specifically include the time period corresponding to the date.

[0083] The above callback information can be equivalent to the metadata information of the data file, and specifically can include the device number, file type information, storage path, file size, partition, storage domain and other information corresponding to the actual data file. This application does not impose any specific restrictions on this.

[0084] In S120, when implemented specifically, the M callback information is determined by the time series database based on the time series index information, which can be understood as follows: the time series database in the storage server can quickly determine the M callback information by querying the time series index information pre-established in the database, and the time series index information includes at least one of the device list and the time interval. In this way, the distributed computing node cluster receives the M callback information sent by the storage server based on the time series database, thereby obtaining the callback information of the data required in the data processing task.

[0085] Exemplarily, after the timing database in the storage server obtains the first request for M callback information, according to the established timing index information, for example, the required callback information can be located by partition index according to the time interval (15 minutes, 30 minutes, hourly level), thereby realizing the rapid determination of the above-mentioned M callback information.

[0086] Alternatively, the M callback information may be determined more quickly and accurately based on a device list in a time series database and an index combination of time partitions, and this application does not impose any specific restrictions on this.

[0087] In S130, during specific implementation, after the distributed computing node cluster obtains the M callback information, the M callback information is distributed to each distributed computing node in the distributed computing node cluster, so that each distributed computing node executes the target data operation respectively.

[0088] The above-mentioned target data operations may specifically be operations such as data analysis, filtering and data statistics, and this application does not impose strict restrictions on this.

[0089] When performing specific callback information allocation, multiple computing nodes in the distributed computing node cluster may be sorted and numbered, so as to allocate callback information to each computing node in a round-robin manner. Alternatively, in some other embodiments, the M callback information may be allocated to the corresponding computing nodes according to different weights based on the computing performance and capacity of each computing node, and this application does not impose specific restrictions on this.

[0090] In some possible implementations, in order to more reasonably implement the allocation and processing of the M callback information, the M callback information may include device codes and data collection times corresponding to the M callback information one by one;

[0091] The above-mentioned allocating the M callback information to each distributed computing node in the distributed computing node cluster may include:

[0092] According to the device codes and data collection times corresponding to the M callback information, the M callback information are distributed to each distributed computing node.

[0093] It should be noted that, assuming that the target callback information is any one of the M callback information, the target callback information includes its own device code and data collection time. In the actual data storage process, the data corresponding to the target callback information is often uploaded by a device to the object storage service for data storage.

[0094] Based on this, the device code in the target callback information can be the device code corresponding to the device end. The data collection time in the target callback information can be the time when the device end stores the data corresponding to the target callback information to the object storage service end, and this application does not impose strict restrictions on this.

[0095] For the above-mentioned allocation of the M callback information to each distributed computing node according to the device codes and data collection times corresponding one to one to the M callback information, exemplarily, it can be: after the distributed computing node cluster obtains the above-mentioned M callback information, it is assumed that the M callback information includes callback information with device code A and in the partition from six to eight o'clock, and callback information with device code B and in the partition from five to eight o'clock.

[0096] At this time, the callback information whose device code is A and is in the partition from 6 o'clock to 7 o'clock is assigned to computing node 1, the callback information whose device code is A and is in the partition from 7 o'clock to 8 o'clock is assigned to computing node 2, the callback information whose device code is B and is in the partition from 5 o'clock to 6 o'clock is assigned to computing node 3, and so on. Based on the device codes and data collection times corresponding to the M callback information one by one, the task of M callback information can be split.

[0097] It should be noted that the above is only an example and does not play a substantial role in limiting how to allocate callback information according to device code and data collection time in this application. In other implementations, other methods can be used to achieve the division of callback information according to device code and data collection time.

[0098] It should be added that, in some other embodiments, the callback information may include only the device code or only the data collection time. The M callback information is distributed to each distributed computing node in the distributed computing node cluster, which may be specifically: the M callback information is distributed to each distributed computing node according to the device code or data collection time corresponding to the M callback information, and this application does not impose any specific restrictions on this.

[0099] In some possible implementations, considering the actual data processing scenarios at this stage, the M callback information may further include storage paths of M data corresponding to the M callback information one by one; after receiving the M callback information sent by the storage server based on the time series database, the data processing method further includes:

[0100] According to the storage paths of the N data corresponding to the N callback information, the N data corresponding to the N callback information are obtained from the object storage service end.

[0101] Analyze and calculate N data to obtain the target result;

[0102] The M callback information includes N callback information, where N is a positive integer less than or equal to M.

[0103] Specifically, each of the M callback information contains the storage path of the corresponding data in the object storage service. In this way, after the distributed computing node cluster receives the M callback information sent by the storage server based on the time series database, it can further select the currently required N callback information from the M callback information, and then obtain the N data corresponding to the N callback information from the object storage service according to the storage path of the data included in the N callback information.

[0104] In this way, after obtaining the actual N data, the N data can be processed by analytical operations, comprehensive analysis, etc. according to business requirements, so as to obtain the target result corresponding to the N data. The target result may be determined according to the specific data processing operation performed, and this application will not elaborate on this in detail.

[0105] It should be noted that in some other embodiments, the above-mentioned target results may be further persisted in a data warehouse for storage to cope with changing data requirements and scenarios, so as to facilitate more comprehensive and extensive analysis and application of data.

[0106] It should be added that, in some other feasible implementations, combined with the actual work processing process of the distributed computing node cluster, in the above embodiment, the behavior of obtaining N data from the object storage server according to the storage path, and parsing and calculating the N data to obtain the target result can also be specifically performed by a target distributed computing node under the distributed computing node cluster.

[0107] For example, the above distributed computing node cluster will distribute M callback information to each distributed computing node. Among them, the above N callback information is distributed to the target distributed computing node. In this way, in the specific implementation, the target distributed computing node can obtain N data corresponding to the N allocated callback information from the object storage service end according to the storage path of the data included in the N allocated callback information.

[0108] After acquiring N data, the target distributed computing node may perform data processing such as analytical operations on the N data to obtain a target result. The target result depends specifically on the data processing operation performed, and this application will not elaborate on this in detail.

[0109] See below Figure 2 , Figure 2 FIG2 shows a second flow chart of a data processing method provided by an embodiment of the present application. Figure 2 The data processing method shown can be applied to a storage server, on which a time series database can be deployed. Figure 2 As shown, the data processing method provided in the embodiment of the present application includes the following steps:

[0110] S210, receiving a first request from a distributed computing node cluster for M callback information;

[0111] S220, determining M callback information by querying the timing index information in the timing database, where the timing index information includes at least one of a device list and a time interval; the timing database stores the M callback information, the M callback information corresponds to the M data one by one, and M is a positive integer;

[0112] S230, sending M callback information to the distributed computing node cluster.

[0113] A data processing method provided by an embodiment of the present application comprises a storage server receiving a callback information request from a distributed computing node cluster, and then determining the required callback information from the timing database based on the timing index information established in the timing database in the storage server, and sending the determined callback information to the distributed computing node cluster. A data processing method according to an embodiment of the present application does not need to recursively scan the specific file data under all file paths in the object storage locally, but instead completes the rapid loading and acquisition of the callback information of the data required in the data calculation and processing process based on the timing index information established in the timing database, so that the data processing performance and efficiency are greatly improved.

[0114] The specific implementation of the above steps 210 to 230 is described in detail below.

[0115] In S210, when implemented specifically, the distributed computing node cluster may establish communication with the storage server in advance. When the distributed computing node cluster needs to process data, the distributed computing node cluster sends a first request for M callback information corresponding to the M data to the storage server, so as to request the above M callback information from the storage server. Based on the pre-established communication, the storage server receives the first request for the M callback information from the distributed computing node cluster.

[0116] A time series database is deployed on the storage server. The time series database stores M callback information, and the M callback information corresponds to the M data one by one.

[0117] The first request may be in the form of code, character string, or text, etc., and this application does not impose any restrictions on this.

[0118] And, the first request may specifically include the relevant information of the M callback information. For example, if the M callback information is stored in the time partition corresponding to a certain day in the time series database, the first request may specifically include the time period corresponding to the date.

[0119] The above callback information can be equivalent to the metadata information of the data file, and specifically can include the device number, file type information, storage path, file size, partition, storage domain and other information corresponding to the actual data file. This application does not impose any specific restrictions on this.

[0120] In S220, during specific implementation, the time series database in the storage server can quickly determine the M callback information by querying the time series index information pre-established in the database, where the time series index information includes at least one of a device list and a time interval.

[0121] It should be understood that, for the sake of brevity, the specific implementation of this step can refer to the corresponding description in the aforementioned embodiment, and this application will not elaborate on it here.

[0122] In S230 , in specific implementation, the M callback information may be sent to the distributed computing node cluster through pre-established communication, a corresponding communication protocol or a communication channel.

[0123] In some possible implementations, in order to more reasonably implement the allocation and processing of the M callback information, the M callback information may include device codes and data collection times corresponding to the M callback information one by one.

[0124] It should be noted that in the actual data storage process, the data corresponding to the callback information is often uploaded from a certain device to the object storage service for data storage.

[0125] Based on this, the device code in the callback information can be the device code corresponding to the device end that uploaded the data of the callback information. The data collection time in the callback information can be the time when the device end stores the data corresponding to the callback information to the object storage service end, and this application does not impose strict restrictions on this.

[0126] In some possible implementations, before the storage server receives the first request of the distributed computing node cluster for M callback information, the data processing method may further include:

[0127] Receive M callback messages sent by the application server through the consumer program instance;

[0128] According to the device codes and data collection times corresponding to the M callback information, the M callback information is distributedly stored in the time series database;

[0129] Establish timing index information corresponding to the M callback information.

[0130] In specific implementation, a consumer program instance may be deployed on the application server. The consumer program instance deployed on the application server subscribes to and consumes the M callback information synchronized in the message queue, and sends the consumed M callback information to the storage server, so as to perform distributed storage on the time series database.

[0131] It should be noted that the above-mentioned consumption behavior of the M callback information can be specifically understood as an action of obtaining the M callback information, and this application does not impose strict restrictions on this.

[0132] The timing database in the storage server receives the above-mentioned M callback information based on the communication method pre-established between the consumer program instance on the application server, and distributes the information according to the device code and data collection time included in the callback information, and establishes timing index information corresponding to the above-mentioned M callback information.

[0133] The above-mentioned timing index information may specifically include at least one of a device list and a time interval, and the present application does not impose any specific limitation on this.

[0134] In some possible implementations, in order to facilitate subsequent computing nodes to more reasonably implement analytical calculations on N data corresponding one-to-one to the N callback information, the above-mentioned M callback information may also include storage paths of the M data corresponding one-to-one to the M callback information.

[0135] The above-mentioned M data may be pre-selected and stored in the object storage service client. The storage path may specifically be a specific storage location of the data in the object storage service client, and this application does not impose any specific restrictions on this.

[0136] In this way, after sending M callback information to the distributed computing node cluster, each distributed computing node in the distributed computing node cluster can obtain the actual data file corresponding to the callback information according to the storage path included in the callback information, thereby realizing data parsing and calculation operations on the obtained actual data file.

[0137] See below Figure 3 , Figure 3 FIG3 shows a flow chart of a data processing method provided by an embodiment of the present application. Figure 3 The data processing method shown can be applied to an application server, on which a consumer program instance can be deployed. Figure 3 As shown, the data processing method provided in the embodiment of the present application includes the following steps:

[0138] S310, consuming M callback information from the message queue through the consumer program instance, the M callback information corresponds to the M data one by one, and M is a positive integer;

[0139] S320, sending the consumed M callback information to the storage server, so that the storage server performs distributed storage on the M callback information in a time series database, and a time series database is deployed on the storage server.

[0140] A data processing method provided in an embodiment of the present application consumes callback information from a message queue through a consumer program instance, and then sends the consumed callback information to a storage server, so that the callback information can be distributedly stored in a time series database deployed in the storage server. A data processing method in an embodiment of the present application consumes callback information based on a message queue, and sends the consumed callback information to a time series database for distributed storage. It does not need to rely on the construction of serverless computing functions for data storage and partitioning, which not only reduces the implementation cost of the solution, but also avoids the problem of high coupling between the data file monitoring path and the data file upload path that the original serverless computing function relies on, and is also conducive to improving the subsequent parsing and processing performance and efficiency of different computing nodes for data.

[0141] The specific implementation of the above steps 310 to 320 is described in detail below.

[0142] In S310 , during specific implementation, the consumer program instance deployed in the application server may establish communication with the message queue in advance and monitor information upload events in the message queue.

[0143] In this way, after the new callback information is synchronized in the message queue, the application server consumes the M callback information synchronized in the message queue through the consumer program instance, wherein the M callback information corresponds to the M data one by one, and M is a positive integer.

[0144] It should be noted that the above-mentioned message queue can specifically be Apache Kafka, MQ, Apache Pulsar, etc. This application does not make strict provisions on the specific technology selection of the message queue.

[0145] Furthermore, the above-mentioned consumption behavior of the M callback information can be specifically understood as an action of obtaining the M callback information, and this application does not impose strict restrictions on this.

[0146] In S320, during the specific implementation, the consumer program instance on the application server sends M callback information to the storage server based on the pre-established communication method between the consumer program instance in the message application server and the time series database in the storage server, so that the storage server performs distributed storage of the M callback information in the time series database.

[0147] In some possible implementations, considering the format requirements of the time series database for the data entering the database, in order to more reasonably store the subsequent M callback information in the time series database, the above-mentioned sending of the consumed M callback information to the storage server may specifically include:

[0148] According to the data format requirements supported by the time series database, the M callback information after consumption is structured respectively;

[0149] The structured M callback information is sent to the storage server.

[0150] In specific implementation, the data format requirements supported by the time series database may be json, csv, etc. After consuming the M callback information from the message queue, the M callback information consumed is first structured, that is, the data format is converted, so that the data format of the M callback information after the structured processing meets the requirements of the time series database.

[0151] In some possible implementations, the above-mentioned consumer program instance can also use a developer tool kit (Software Development Kit, SDK) to send the M callback information after structured processing to a time series database to achieve fast writing of the M callback information after structured processing in the time series database.

[0152] See below Figure 4 , Figure 4 FIG4 shows a flow chart of a data processing method provided by an embodiment of the present application. Figure 4 The data processing method shown can be applied to the object storage service end. Figure 4 As shown, the data processing method provided in the embodiment of the present application includes the following steps:

[0153] S410, storing M data uploaded by at least one device, where M is a positive integer;

[0154] S420, generating M callback information corresponding to the M data one by one;

[0155] S430, synchronize the M callback information to the message queue, so that the application server consumes the M callback information from the message queue through the consumer program instance, and sends the consumed M callback information to the storage server, so that the storage server performs distributed storage of the M callback information in the time series database; wherein, a time series database is deployed on the storage server, and a consumer program instance is deployed on the application server.

[0156] A data processing method provided by an embodiment of the present application receives and stores data uploaded by each device through an object storage service, and synchronizes the callback information corresponding to the data generated after storage to a message queue, so that the consumer program instance deployed on the application server can consume the callback information from the message queue, and send the consumed callback information to the time series database in the storage server, so that the callback information is distributedly stored by the time series database. A data processing method of an embodiment of the present application does not need to rely on the construction of serverless computing functions for data storage and partitioning, which not only reduces the implementation cost of the solution, but also avoids the problem of high coupling between the data file monitoring path and the data file upload path that the original serverless computing function relies on. And because the data processing links involved in the message queue, the consumer program instance and the time series database during the data storage process are only lightweight callback information processing, the efficiency of data information processing is also greatly improved. In addition, since it uses a time series database to store callback information in a distributed manner, it is also beneficial to improve the subsequent parsing and processing performance and efficiency of different computing nodes for data.

[0157] The specific implementation of the above steps 410 to 430 is described in detail below.

[0158] In S410, during specific implementation, at least one device uploads data to the object storage service, and the object storage service stores M data uploaded by the at least one device.

[0159] The above data may specifically be a CAN file or other data files, etc., and this application does not impose any specific restrictions on this.

[0160] In S420, during specific implementation, after the M data are stored in the object storage service, M callback information corresponding to the M data will be generated.

[0161] The above callback information can be equivalent to the metadata information of the data file, and specifically can include the device number, file type information, storage path, file size, partition, storage domain and other information corresponding to the actual data file. This application does not impose any specific restrictions on this.

[0162] In S430, during the specific implementation, the object storage service synchronizes the M callback information generated to the message queue, so that the application server consumes the M callback information from the message queue through the consumer program instance, and sends the consumed M callback information to the storage server, so that the storage server performs distributed storage of the M callback information in the time series database.

[0163] Among them, a time series database is deployed on the above storage server, and a consumer program instance is deployed on the above application server.

[0164] In some possible implementations, in order to more reasonably implement the allocation and processing of the M callback information, the M callback information may include device codes and data collection times corresponding to the M callback information one by one.

[0165] It should be added that, in some embodiments, the above-mentioned M callback information may also only include device codes corresponding to the M callback information one by one, or only include data collection times corresponding to the M callback information one by one, and this application does not make specific restrictions on this.

[0166] In order to facilitate understanding of the data processing method provided in the above embodiment, the above method is described below using a specific scenario embodiment. Figure 5 and Figure 6 , Figure 5 This is one of the flowcharts of a scenario implementation of a data processing method provided by an embodiment of the present application; Figure 6 This is the second flowchart of a scenario implementation of the data processing method provided in one embodiment of the present application.

[0167] like Figure 5 and Figure 6 As shown, the scenario embodiment may specifically include the following steps:

[0168] Step 1: at least one device, including device 1, device 2, device 3 and device 4, uploads data to the object storage (object storage service). The uploaded data may be a can file, etc., which is not specifically limited in this application.

[0169] Step 2: The object storage service receives and stores the data uploaded by each device, and generates callback information of the data, i.e., object storage metadata. The callback information may be equivalent to metadata information of the data stored by the corresponding storage service, and may specifically include information such as the device number, file type information, storage path, file size, partition, and storage domain corresponding to the actual data file, and this application does not impose any specific restrictions on this.

[0170] Step 3: The object storage service synchronizes the callback information of the generated data to the message queue. Communication can be established in advance between the object storage service and the message queue to achieve synchronization of the callback information.

[0171] Step 4: The consumer program instance (meta consumer program) in the application server subscribes to and consumes the synchronized callback information in the message queue. In specific implementation, the consumer program instance deployed in the application server can establish communication with the message queue in advance and listen to the information upload event in the message queue.

[0172] In this way, after the new callback information is synchronized to the message queue, the application server can consume the synchronized callback information in the message queue in a timely manner through the consumer program instance.

[0173] In step 5, the consumer program instance in the application server sends the consumed callback information to the time series database, which is deployed in the storage server.

[0174] Specifically, after consuming the above callback information, the consumer program instance in the application server can first structure the callback information according to the data format requirements (json, csv format, etc.) supported by the time series database, and then send the structured callback information to the time series database through the SDK corresponding to the time series database.

[0175] Step 6: After receiving the callback information, the time series database performs partitioned distributed storage according to the device code and data collection time in the callback information. The time series index switch is turned on at the table level in the time series database, and the time series index information corresponding to the callback information is established in real time. The time series index information may specifically include at least one of the device list and the time partition, and this application does not impose specific restrictions on this.

[0176] Step seven: when data processing is required for the callback information, a data synchronization task is first performed. Specifically, the distributed computing node cluster sends a request for the callback information to the time series database in the storage server.

[0177] Step eight, the time series database in the storage server quickly queries and determines the above callback information (target file list) based on the time series index information pre-established in the database according to the time interval or / and time interval (15 minutes, 30 minutes, hour level), and sends the determined callback information to the distributed computing node cluster.

[0178] Step 9: Divide subtasks by device and time slice. Specifically, according to the device code and data collection time included in each callback information, each callback information is allocated to each distributed computing node to split the data processing task into multiple subtasks, so that each distributed computing node performs parallel local data parsing calculations according to the callback information allocated to it, thereby accelerating the data task processing efficiency.

[0179] Step 10, the executor (distributed computing node) performs parsing and result storage tasks in parallel. Specifically, each of the above distributed computing nodes parses and calculates the data corresponding to the allocated callback information based on the callback information allocated to each node, obtains the target result, and persists the target result to the data warehouse for storage to cope with changing data requirements and scenarios. The above target data operations can specifically be operations such as data analysis, filtering and data statistics, and this application does not impose strict restrictions on this.

[0180] It should be added that, in some more specific embodiments, each of the M callback information may also include the storage path of the corresponding data in the object storage service. In this way, after the distributed computing node cluster distributes the M callback information to each distributed computing node, each distributed computing node obtains the corresponding data from the object storage service according to the storage path of the data included in the distributed callback information, and performs data processing such as parsing calculations on the actual data obtained to obtain the above target result, which is determined specifically according to the data processing operation performed, and this application will not elaborate on this in detail.

[0181] In the embodiment of this scenario, the object storage service is used to receive and store the data uploaded by each device, and the callback information corresponding to the data generated after storage is synchronized to the message queue. The consumer program instance deployed on the application server can consume the callback information from the message queue and send it to the time series database in the storage server. The time series database performs distributed storage on the received callback information and establishes the time series index information corresponding to the callback information. In this way, when the distributed computing node cluster needs to perform data loading analysis, by requesting the callback information from the storage server, the time series database in the storage server queries the requested callback information based on the established time series index information and sends it to the distributed computing node cluster. After receiving the callback information sent by the storage server based on the time series database, the distributed computing node cluster splits the callback information to different computing nodes so that each computing node performs corresponding data processing according to the allocated callback information.

[0182] Compared with the prior art, the data processing method provided by the embodiment of the present application, on the one hand, does not need to rely on the construction of serverless computing functions for data storage and partitioning, which not only reduces the implementation cost of the solution, but also avoids the problem of high coupling between the data file monitoring path and the data file upload path that the original serverless computing function relies on. On the other hand, since the actual data is only stored on the object storage service side, and the data processing links involved in the message queue, consumer program instance and time series database are only lightweight callback information processing, the efficiency of data information processing is greatly improved. In addition, it stores callback information through a time series database, and can make full use of the time series database to establish a time series index for callback information. Therefore, in the subsequent data analysis and processing process of the distributed computing node cluster, the required callback information can be quickly queried through the time series index without recursively scanning the specific file data under all file paths, and the data processing performance and efficiency are greatly improved.

[0183] Based on the data processing method provided in the above embodiment, the present application also provides a data processing device corresponding to the above data processing method. Figure 7 The data processing device is introduced in detail.

[0184] Figure 7 is one of the structural diagrams of a data processing device provided in one embodiment of the present application, Figure 7 The data processing apparatus shown can be applied to a distributed computing node cluster. Figure 7 The data processing device 700 shown comprises:

[0185] A first sending module 710 is used to send a first request for M callback information corresponding to M data one by one to a storage server, where a time series database is deployed on the storage server, and M is a positive integer;

[0186] A first receiving module 720 is configured to receive M callback information sent by the storage server based on the time series database, where the M callback information is determined by the time series database based on the time series index information, and the time series index information includes at least one of a device list and a time interval;

[0187] The first allocation module 730 is used to allocate the M callback information to each distributed computing node in the distributed computing node cluster so that each distributed computing node performs the target data operation respectively.

[0188] A data processing device provided by an embodiment of the present application includes a distributed computing node cluster sending a request for callback information to a storage server, and then receiving callback information sent by the storage server based on a time series database, the callback information is determined by the time series database based on time series index information, and finally the received callback information is split to different computing nodes in the cluster, so that each computing node performs data parsing calculations according to the allocated callback information. A data processing device according to an embodiment of the present application does not need to recursively scan the specific file data under all file paths in the object storage locally, but completes the rapid loading and acquisition of the callback information of the data required in the data calculation and processing process based on the time series index information established by the time series database, so that the data processing performance and efficiency are greatly improved.

[0189] In some possible implementations, the M callback information may include device codes and data collection times corresponding to the M callback information one by one;

[0190] The first allocation module 730 allocates the M callback information to each distributed computing node in the distributed computing node cluster, which can be specifically used for:

[0191] According to the device codes and data collection times corresponding to the M callback information, the M callback information are distributed to each distributed computing node.

[0192] In some possible implementations, the M callback information also includes storage paths of M data corresponding one-to-one to the M callback information; after receiving the M callback information sent by the storage server based on the time series database, the data processing device may further include:

[0193] The data acquisition module can be used to acquire N data corresponding to the N callback information from the object storage service end according to the storage path of the N data corresponding to the N callback information.

[0194] The data analysis module can be used to analyze and calculate N data to obtain the target result;

[0195] The M callback information includes N callback information, where N is a positive integer less than or equal to M.

[0196] Based on the data processing method provided in the above embodiment, the present application also provides a data processing device corresponding to the above data processing method. Figure 8 The data processing device is introduced in detail.

[0197] Figure 8 This is a second structural diagram of a data processing device provided in an embodiment of the present application. Figure 8The data processing device shown can be applied to a storage server, on which a time series database can be deployed. Figure 8 The data processing device 800 shown comprises:

[0198] The second receiving module 810 is used to receive a first request from the distributed computing node cluster for M callback information;

[0199] A first determination module 820 is used to determine M callback information by querying the time series index information in the time series database, where the time series index information includes at least one of a device list and a time interval; the time series database stores M callback information, the M callback information corresponds to the M data one by one, and M is a positive integer;

[0200] The second sending module 830 is used to send M callback information to the distributed computing node cluster.

[0201] A data processing device provided by an embodiment of the present application receives a callback information request from a distributed computing node cluster by a storage server, determines the required callback information from the time series database based on the time series index information established in the time series database in the storage server, and sends the determined callback information to the distributed computing node cluster. A data processing device according to an embodiment of the present application does not need to recursively scan the specific file data under all file paths in the object storage locally, but instead completes the rapid loading and acquisition of the callback information of the data required in the data calculation and processing process based on the time series index information established in the time series database, so that the data processing performance and efficiency are greatly improved.

[0202] In some possible implementations, the M pieces of callback information may include device codes and data collection times corresponding to the M pieces of callback information one by one.

[0203] In some possible implementations, before the storage server receives the first request of the distributed computing node cluster for M callback information, the data processing device may further include:

[0204] The third receiving module may be used to receive M callback information sent by the application server through the consumer program instance;

[0205] The second storage module can be used to perform distributed storage of the M callback information in a time series database according to the device codes and data collection times corresponding to the M callback information one by one;

[0206] The first establishing module can be used to establish timing index information corresponding to the M callback information.

[0207] In some possible implementations, the M pieces of callback information may further include storage paths of M pieces of data corresponding one-to-one to the M pieces of callback information.

[0208] Based on the data processing method provided in the above embodiment, the present application also provides a data processing device corresponding to the above data processing method. Fig. 9 The data processing device is introduced in detail.

[0209] Fig. 9 This is a third structural diagram of a data processing device provided in an embodiment of the present application. Fig. 9 The data processing apparatus shown may be applied to an application server, on which a consumer program instance may be deployed. Fig. 9 The data processing device 900 shown comprises:

[0210] The first consumption module 910 is used to consume M callback information from the message queue through the consumption program instance, where the M callback information corresponds to the M data one by one, and M is a positive integer;

[0211] The third sending module 920 is used to send the consumed M callback information to the storage server, so that the storage server performs distributed storage on the M callback information in the time series database, and the time series database is deployed on the storage server.

[0212] A data processing device provided in an embodiment of the present application consumes callback information from a message queue through a consumer program instance, and then sends the consumed callback information to a storage server, so that the callback information can be distributedly stored in a time series database deployed in the storage server. A data processing device in an embodiment of the present application consumes callback information based on a message queue, and sends the consumed callback information to a time series database for distributed storage. It does not need to rely on the construction of serverless computing functions for data storage and partitioning, which not only reduces the implementation cost of the solution, but also avoids the problem of high coupling between the data file monitoring path and the data file upload path that the original serverless computing function relies on, and is also conducive to improving the subsequent parsing and processing performance and efficiency of different computing nodes for data.

[0213] In some possible implementations, the third sending module 910 sending the consumed M callback information to the storage server may include:

[0214] The first processing submodule can be used to perform structured processing on the M callback information after consumption according to the data format requirements supported by the time series database;

[0215] The first sending submodule may be used to send the M callback information after structured processing to the storage server.

[0216] Based on the data processing method provided in the above embodiment, the present application also provides a data processing device corresponding to the above data processing method. Fig.10 The data processing device is introduced in detail.

[0217] Fig.10 This is a fourth structural diagram of a data processing device provided in an embodiment of the present application. Fig.10 The data processing device shown can be applied to an object storage service client. Fig.10 The data processing device 1000 shown comprises:

[0218] The first storage module 1010 is used to store M data uploaded by at least one device, where M is a positive integer;

[0219] The first generating module 1020 is used to generate M callback information corresponding to the M data one by one;

[0220] The first synchronization module 1030 is used to synchronize the M callback information to the message queue, so that the application server consumes the M callback information from the message queue through the consumer program instance, and sends the consumed M callback information to the storage server, so that the storage server performs distributed storage of the M callback information in the time series database; wherein the storage server is deployed with a time series database, and the application server is deployed with a consumer program instance.

[0221] A data processing device provided in an embodiment of the present application receives and stores data uploaded by each device end through an object storage service end, and synchronizes the callback information corresponding to the data generated after storage to a message queue, so that the consumer program instance deployed on the application server can consume the callback information from the message queue, and send the consumed callback information to the time series database in the storage server, so that the callback information is distributedly stored by the time series database. A data processing device in an embodiment of the present application does not need to rely on the construction of serverless computing functions for data storage and partitioning, which not only reduces the implementation cost of the solution, but also avoids the problem of high coupling between the data file monitoring path and the data file upload path that the original serverless computing function relies on. And because the data processing links involved in the message queue, the consumer program instance and the time series database during the data storage process are only lightweight callback information processing, the efficiency of data information processing is also greatly improved. In addition, since it uses a time series database to store callback information in a distributed manner, it is also beneficial to improve the subsequent parsing and processing performance and efficiency of different computing nodes for data.

[0222] In some possible implementations, the M pieces of callback information may include device codes and data collection times corresponding to the M pieces of callback information one by one.

[0223] Fig.11 It is a structural diagram of a data processing device provided in one embodiment of the present application.

[0224] The data processing device may include a processor 1101 and a memory 1102 storing computer program instructions.

[0225] Specifically, the processor 1101 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0226] The memory 1102 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 1102 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 1102 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 1102 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 1102 is a non-volatile solid-state memory.

[0227] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0228] The processor 1101 implements any one of the data processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 1102 .

[0229] In one example, the data processing device may further include a communication interface 1103 and a bus 1110. Fig.11 As shown, the processor 1101, the memory 1102, and the communication interface 1103 are connected via a bus 1110 and communicate with each other.

[0230] The communication interface 1103 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0231] Bus 1110 includes hardware, software or both, and the parts of data processing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 1110 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0232] The data processing device executes the data processing method in the embodiment of the present application, thereby realizing Figure 1 Describe the data processing methods.

[0233] In addition, in combination with the data processing method in the above embodiments, the present application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the data processing methods in the above embodiments is implemented.

[0234] Furthermore, an embodiment of the present application further provides a vehicle, which includes at least one of the following: the data processing device provided in the above-mentioned embodiment of the present application; the data processing equipment provided in the above-mentioned embodiment of the present application; and the computer-readable storage medium provided in the above-mentioned embodiment of the present application.

[0235] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0236] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0237] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0238] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0239] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A data processing method, characterized in that: The method comprises: Sending a first request for M callback information corresponding to M data one by one to a storage server, where a time series database is deployed on the storage server, and M is a positive integer; receiving the M callback information sent by the storage server based on the time series database, where the M callback information is determined by the time series database based on time series index information, and the time series index information includes at least one of a device list and a time interval; The M callback information are distributed to each distributed computing node in the distributed computing node cluster, so that each distributed computing node executes the target data operation respectively.

2. The method according to claim 1, characterized in that The M callback information include device codes and data collection times corresponding to the M callback information one by one; The distributing the M callback information to each distributed computing node in the distributed computing node cluster includes: According to the device codes and data collection times corresponding to the M callback information one by one, the M callback information are distributed to the respective distributed computing nodes.

3. The method according to claim 1, characterized in that The M callback information also includes storage paths of M data corresponding to the M callback information one by one; After receiving the M callback information sent by the storage server based on the time series database, the method further includes: According to the storage paths of the N data corresponding to the N callback information one by one, obtain the N data corresponding to the N callback information one by one from the object storage service end, Analyze and calculate the N data to obtain the target result; The M callback information includes the N callback information, where N is a positive integer less than or equal to M.

4. A data processing method, characterized in that: The method comprises: Receive a first request from the distributed computing node cluster for M callback information; The M callback information is determined by querying the timing index information in the timing database, where the timing index information includes at least one of a device list and a time interval; the timing database stores the M callback information, the M callback information corresponds to the M data one by one, and M is a positive integer; The M callback information are sent to the distributed computing node cluster.

5. The method according to claim 4, characterized in that The M callback information include device codes and data collection times corresponding to the M callback information one by one.

6. The method according to claim 5, characterized in that Before the storage server receives a first request from the distributed computing node cluster for M callback information, the method further includes: Receiving the M callback information sent by the application server through the consumer program instance; Distributed storage of the M callback information in the time series database is performed according to the device codes and data collection times corresponding to the M callback information one by one; Establish timing index information corresponding to the M callback information.

7. The method according to claim 4, characterized in that The M callback information also include storage paths of M data corresponding one-to-one to the M callback information.

8. A data processing method, characterized in that: The method comprises: Consume M callback information from the message queue through the consumer program instance, where the M callback information corresponds to M data one by one, and M is a positive integer; The consumed M callback information are sent to a storage server, so that the storage server performs distributed storage on the M callback information in a time series database, and the time series database is deployed on the storage server.

9. The method according to claim 8, characterized in that The sending of the consumed M callback information to the storage server includes: According to the data format requirements supported by the time series database, the M callback information after consumption is structured respectively; The M callback information after the structured processing is sent to the storage server.

10. A data processing method, characterized in that: The method comprises: Store M data uploaded by at least one device, where M is a positive integer; Generate M callback information corresponding to the M data one by one; Synchronize the M callback information into a message queue, so that the application server consumes the M callback information from the message queue through a consumer program instance, and sends the consumed M callback information to a storage server, so that the storage server performs distributed storage of the M callback information in a time series database; The time series database is deployed on the storage server, and the consumer program instance is deployed on the application server.

11. A data processing device, characterized in that: The device comprises: A first sending module is used to send a first request for M callback information corresponding to M data one by one to a storage server, where a time series database is deployed on the storage server, and M is a positive integer; A first receiving module is configured to receive the M callback information sent by the storage server based on the time series database, where the M callback information is determined by the time series database based on time series index information, and the time series index information includes at least one of a device list and a time interval; The first allocation module is used to allocate the M callback information to each distributed computing node in the distributed computing node cluster, so that each of the distributed computing nodes executes the target data operation respectively.

12. A data processing device, characterized in that: The device comprises: A second receiving module is used to receive a first request from the distributed computing node cluster for M callback information; A first determining module is configured to determine the M callback information by querying the timing index information in the timing database, where the timing index information includes at least one of a device list and a time interval; the timing database stores the M callback information, the M callback information corresponds to the M data one by one, and M is a positive integer; The second sending module is used to send the M callback information to the distributed computing node cluster.

13. A data processing device, characterized in that: The device comprises: A first consumption module is used to consume M callback information from a message queue through a consumption program instance, wherein the M callback information corresponds to M data one by one, and M is a positive integer; The third sending module is used to send the consumed M callback information to the storage server, so that the storage server performs distributed storage on the M callback information in a time series database, and the time series database is deployed on the storage server.

14. A data processing device, characterized in that: The device comprises: A first storage module is used to store M data uploaded by at least one device, where M is a positive integer; A first generating module, used to generate M callback information corresponding to the M data one by one; A first synchronization module is used to synchronize the M callback information to a message queue, so that the application server consumes the M callback information from the message queue through a consumer program instance, and sends the consumed M callback information to a storage server, so that the storage server performs distributed storage of the M callback information in a time series database; The time series database is deployed on the storage server, and the consumer program instance is deployed on the application server.

15. A data processing device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the data processing method according to any one of claims 1 to 10 is implemented.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.

17. A vehicle, characterized in that: The vehicle comprises at least one of the following: a data processing apparatus as claimed in any one of claims 11 to 14; a data processing device as claimed in claim 15; and a computer-readable storage medium as claimed in claim 16.

Images