Data processing system, method, electronic device and storage medium

Abstract

The present application provides a data processing system, method, electronic device, and storage medium. The data processing system includes: an acquisition module, configured to obtain first vehicle data and send first metadata of the first vehicle data to a first message queue; a processing module, configured to obtain the first metadata in the first message queue, obtain the first vehicle data, process the first vehicle data through a first workflow engine to obtain second vehicle data, and send second metadata of the second vehicle data to a second message queue; a storage module, configured to obtain the second metadata in the second message queue, obtain the second vehicle data through a second workflow engine, and store a target file in a file storage; a data verification and compensation module, configured to perform data integrity verification and compensation on the first vehicle data, the second vehicle data, and the target file. This solution improves the timeliness of the data processing system while ensuring the stability and high availability of the system.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular, to a data processing system, method, electronic device, and storage medium. Background Art

[0002] In recent years, the development demand for autonomous driving technology in the automotive field has been increasing day by day. The stability, efficiency, and convenience of the acquisition, processing, and storage of automotive data are the prerequisite guarantees for the development of automotive autonomous driving technology. With the explosion of new energy vehicle autonomous driving, it is an increasingly urgent technology for each automotive manufacturing enterprise to conveniently and effectively collect, process, and store automotive data.

[0003] Currently, to solve the above problems, a common method is to use multiple modules to process automotive data. For example, the links such as acquisition, processing, and storage are respectively integrated into modules, and the automotive data is processed sequentially through each module, which improves the convenience of data acquisition, processing, and storage.

[0004] However, since the modules in the existing methods are all discrete modules, the time delay between each module is large. When data loss occurs, each module needs to process the lost data separately, resulting in a T+1 mechanism between system modules and poor timeliness.

[0005] Content of the Application

[0006] In view of this, embodiments of the present application provide a data processing system, method, electronic device, and storage medium to solve the problem of poor timeliness of the data processing system.

[0007] According to a first aspect of the embodiments of the present application, a data processing system is provided, including: an acquisition module, configured to obtain first automotive data and send first metadata of the first automotive data to a first message queue, where the first automotive data includes data generated during the autonomous driving of a vehicle; a processing module, configured to obtain the first metadata in the first message queue, obtain the first automotive data according to the first metadata, process the first automotive data through a first workflow engine to obtain second automotive data, and after determining that the second automotive data meets a preset data processing requirement, send second metadata of the second automotive data to a second message queue; a storage module, configured to obtain the second metadata in the second message queue, obtain the second automotive data according to the second metadata through a second workflow engine, convert the second automotive data into a target file, and store the target file in a file storage; a data verification and compensation module, configured to perform data integrity verification on the first automotive data, the second automotive data, and the target file, and compensate for the missing first automotive data, second automotive data, and target file.

[0008] According to a second aspect of the embodiments of the present application, a data processing method is provided, including: obtaining first vehicle data, and sending first metadata of the first vehicle data to a first message queue, where the first vehicle data includes data generated during the automatic driving of a vehicle; obtaining the first metadata in the first message queue, obtaining the first vehicle data according to the first metadata, processing the first vehicle data through a first workflow engine to obtain second vehicle data, and after determining that the second vehicle data meets a preset data processing requirement, sending second metadata of the second vehicle data to a second message queue; obtaining the second metadata in the second message queue, obtaining the second vehicle data according to the second metadata through a second workflow engine, converting the second vehicle data into a target file, and storing the target file in a file storage; performing data integrity verification on the first vehicle data, the second vehicle data, and the target file, and compensating for the missing first vehicle data, second vehicle data, and target file.

[0009] According to a third aspect of the embodiments of the present application, an electronic device is provided, including: a processor, a communication interface, a memory, and a communication bus, where the processor, the memory, and the communication interface complete communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the data processing method described in the second aspect of the embodiment.

[0010] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, it implements the data processing method described in the second aspect of the embodiment.

[0011] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, and the computer instructions instruct a computing device to perform operations corresponding to the data processing method described in the second aspect of the embodiment.

[0012] With the above technical solutions, a data collection module, a processing module, a storage module, and a data verification and compensation module are provided in the data processing system. By using a flow engine and distributed high-performance processing and storage technologies, the stability and high availability of the system are ensured. Through the data verification and compensation module, an integrated setting of the data processing system is realized, and the process and data from data collection to the final storage result are overall controlled, improving the timeliness of the data processing system. Description of the Drawings

[0013] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0014] Figure 1 It is a schematic diagram of a data processing system provided by an embodiment of the present application;

[0015] Figure 2 It is a schematic diagram of a data verification and compensation module provided by an embodiment of the present application

[0016] Figure 3 It is a schematic diagram of a data processing system including a numbering device provided by another embodiment of the present application;

[0017] Figure 4 It is a schematic diagram of a data processing system including a data management module and a data display module provided by another embodiment of the present application;

[0018] Figure 5 It is a schematic diagram of a data processing method provided by an embodiment of the present application;

[0019] Figure 6 It is a schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some, rather than all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0021] Data processing system

[0022] Figure 1 It is a schematic diagram of a data processing system provided by an embodiment, as Figure 1 shown, the data processing system 100 includes: a collection module 101, a processing module 102, a storage module 103, and a data verification and compensation module 104.

[0023] The collection module 101 is used to obtain first vehicle data and send the first metadata of the first vehicle data to the first message queue.

[0024] The first automotive data includes, but is not limited to, the event buried point structured data of various chips on the vehicle and the picture and video data related to autonomous driving technology. The first metadata is the data describing the first automotive data, such as relationships, fields, constraints, etc.

[0025] The processing module 102 is used to obtain the first metadata in the first message queue, obtain the first automotive data according to the first metadata, process the first automotive data through the first workflow engine to obtain the second automotive data, and after determining that the second automotive data meets the preset data processing requirements, send the second metadata of the second automotive data to the second message queue.

[0026] The first metadata in the first message queue serves as an input source for the processing module 102 to obtain the first automotive data according to the first metadata. There is at least one first workflow engine. Different first workflow engines will receive the first automotive data respectively according to different requirements (such as positioning, calculating driving trajectories, etc.) and process the first automotive data according to the preset data processing requirements to obtain the second automotive data processed as needed.

[0027] The storage module 103 is used to obtain the second metadata in the second message queue, obtain the second automotive data according to the second metadata through the second workflow engine, convert the second automotive data into a target file, and store the target file in the file storage.

[0028] The second metadata in the second message queue serves as an input source for the storage module 103 to obtain the second automotive data according to the second metadata. The second metadata is the data describing the second automotive data. There is at least one second workflow engine. Different second workflow engines will convert the second automotive data into a target file respectively according to different types of data in the second automotive data. The target file is a binary file, and the file storage can be a memory such as ROM (Read-Only Memory) or RAM (Random Access Memory).

[0029] The data verification and compensation module 104 is used to verify the data integrity of the first automotive data, the second automotive data, and the target file, and compensate for the missing first automotive data, second automotive data, and target file.

[0030] The data verification and compensation module 104 will regularly perform integrity verification on the first automotive data, second automotive data, and target file obtained by the module. When data loss is found in the first automotive data, the data verification and compensation module 104 will control the acquisition module 101 to obtain the first automotive data again. When data loss is found in the second automotive data, the data verification and compensation module 104 will control the processing module 102 to make the first workflow engine process the first automotive data again to re-obtain the second automotive data. When data loss is found in the target file, the data verification and compensation module 104 will control the storage module 103 to make the second workflow engine convert the second automotive data into the target file again.

[0031] In the embodiment of the present application, an acquisition module 101, a processing module 102, a storage module 103, and a data verification and compensation module 104 are provided in the data processing system 100. By using the workflow engine and distributed high-performance processing and storage technologies, the stability and high availability of the system are ensured. Through the data verification and compensation module 104, an integrated setting of the data processing system 100 is realized, and the process and results from data acquisition to the final storage result are overall controlled, improving the timeliness of the data processing system 100.

[0032] In a feasible manner, the acquisition module 101 can also convert the structured data in the first automotive data into unstructured data, and store the unstructured data in the first automotive data and the converted unstructured data into the first object storage system.

[0033] The first automotive data includes at least one piece of structured data and / or at least one piece of unstructured data. Structured data is data logically expressed and implemented by a two-dimensional table structure, such as name, age, gender, etc., and is mainly stored and managed through a distributed relational database. Unstructured data has an irregular or incomplete data structure and no predefined data model, such as various reports, pictures, audio, and video information, etc., and generally stores unstructured data through files, blocks, objects, etc. In order to store the first automotive data in the first object storage system, it is necessary to convert the structured data in the first automotive data into unstructured data, and store the unstructured data in the first automotive data and the converted unstructured data into the first object storage system.

[0034] The processing module 102 can also store the second automotive data obtained after the first workflow engine processes the first automotive data according to the preset data processing requirements into the second object storage system.

[0035] Since the first vehicle data obtained by the processing module 102 is from the first message queue, and the first vehicle data in the first message queue consists of at least one structured data and / or at least one unstructured data, the second vehicle data obtained after the first workflow engine processes the first vehicle data also consists of at least one structured data and / or unstructured data. In order to store the second vehicle data into the object storage system, the structured data in the second vehicle data is converted into unstructured data, and the unstructured data in the second vehicle data and the converted unstructured data are stored in the second object storage system.

[0036] In the embodiment of the present application, both the acquisition module 101 and the processing module 102 can process structured data and store the unified vehicle data. All object storage systems are distributed storage systems, and the vehicle data is dispersed on multiple nodes. Thus, problems such as the performance bottleneck of the data server are solved, and the scalability of the data management architecture is improved.

[0037] In a feasible manner, the acquisition module 101 can also generate an acquisition log according to the acquisition operation, the unstructured data in the first vehicle data, and the converted unstructured data.

[0038] After the acquisition module 101 completes the acquisition operation, it can preprocess the first vehicle data, such as cleaning, format sorting, filtering out dirty data, etc. After the preprocessing, an acquisition log is generated according to the processed first vehicle data and the acquisition operation record. The acquisition log can be used for data verification and can trace back to each specific operation and the corresponding data.

[0039] The processing module 102 can also generate a processing log according to the processing operation on the first vehicle data and the second vehicle data.

[0040] After the processing module 102 completes the processing operation, it can also preprocess the second vehicle data and generate a processing log according to the processed second vehicle data and the processing operation record on the first vehicle data. The processing log can also be used for data verification and can trace back to each specific operation and the corresponding data.

[0041] In the embodiment of the present application, the acquisition module 101 and the processing module 102 can generate corresponding logs to record the operations performed by the modules and the corresponding data. For the automated processing mechanism, generating corresponding flow logs can be used for system recovery when the system fails, thereby ensuring the security of the system.

[0042] Figure 2 It is a schematic diagram of the data verification compensation module 104 provided by an embodiment of the present application, as Figure 2As shown in the figure, the data verification module 104 may include: a data verification sub-module 201 and a data verification sub-module 202.

[0043] The data verification sub-module 201 may be used to verify the first vehicle data according to the collection log, verify the second vehicle data according to the processing log, verify the target file according to the storage log through the third workflow engine, and generate a verification log based on the verification operations on the first vehicle data, the second vehicle data, and the target file.

[0044] The data verification sub-module 201 periodically obtains the collection log, the processing log, and the storage log. There is at least one third workflow engine, and different third workflow engines respectively verify the collection log, the processing log, and the storage log. After the third workflow engine finishes verifying the collection log, the data verification sub-module 201 generates a collection module verification log according to the verification operation record and the verification data. After the third workflow engine finishes verifying the processing log, the data verification sub-module 201 generates a processing module verification log according to the verification operation record and the verification data. After the third workflow engine finishes verifying the storage log, the data verification sub-module 201 generates a storage module verification log according to the verification operation record and the verification data. The verification log includes the collection module verification log, the processing module verification log, and the storage module verification log.

[0045] The data compensation sub-module 202 may be used to compensate the first vehicle data according to the collection log when the data verification sub-module 201 determines that the first vehicle data is missing, compensate the second vehicle data according to the processing log when the data verification sub-module 201 determines that the second vehicle data is missing, and compensate the target file according to the storage log when the data verification sub-module 201 determines that the target file is missing.

[0046] After the data verification sub-module 201 determines that the first vehicle data in the acquisition log is missing, the data compensation sub-module 202 calls the acquisition log to determine the missing data, and controls the acquisition module 101 to collect the missing data again, and compensates the missing data with the collected data. When the data verification sub-module 201 determines that the second vehicle data in the processing log is missing, the data compensation sub-module 202 calls the processing log to determine the missing data, and controls the processing module 102 to re-obtain the first vehicle data according to the first metadata in the first message queue, and enables the first workflow engine to process the first metadata again, and compensates the missing data with the second vehicle data after the re-processing. When the data verification sub-module 201 determines that the target file data in the storage log is missing, the data compensation sub-module 202 calls the storage log to determine the missing data, and controls the storage module 103 to re-obtain the second vehicle data according to the second metadata in the second message queue, and enables the second workflow engine to convert the second vehicle data into a target file again, and compensates the missing data with the target file after the re-conversion.

[0047] In the embodiment of the present application, through the data verification sub-module 201, the logs generated by each module can be verified, ensuring the integrity of the data. After the data verification sub-module 201 confirms data loss, the data compensation sub-module 202 can retrieve the log to confirm the missing data, improving the compensation efficiency. By controlling each module to re-execute the corresponding operations, the data compensation sub-module 202 can find the missing data and automatically compensate for the missing data, improving the processing efficiency of the system.

[0048] In a feasible manner, the data compensation sub-module 202 can also be used to determine whether the weight of the missing first vehicle data, second vehicle data, or target file is greater than a preset weight threshold after the missing first vehicle data, second vehicle data, or target file has not been obtained within a preset compensation time threshold. If so, the acquisition module 101, the processing module 102, and the storage module 103 are suspended and an error message is generated.

[0049] The compensation time threshold is the maximum duration for the data compensation sub-module 202 to execute the compensation task when it is preset. When the duration of the compensation task is within the compensation time threshold, the data compensation sub-module 202 can repeatedly execute the compensation for the first vehicle data, the second vehicle data, or the lost data of the target file until the lost data compensation is successful or the duration of the compensation task exceeds the compensation time threshold. The weight threshold is the maximum weight value when the loss of vehicle data does not affect the system operation. When the duration of the compensation task exceeds the compensation time threshold, it is necessary to determine whether the system continues to operate according to whether the weight of the missing first vehicle data, the second vehicle data, or the target file is greater than the preset weight threshold. When the weight of the missing data is greater than the preset weight threshold, the missing data is more important to the system. Therefore, it is necessary to suspend the system operation and generate an error message, and manually compensate for the lost data. After the lost data compensation is completed, the system operation is restored.

[0050] In the embodiment of the present application, by setting the compensation time threshold, it is possible to prevent the compensation task from affecting the system operation when the lost data is not found for a long time, thereby improving the security of the system. By setting the weight threshold and assigning weights to all vehicle data according to the importance of the data to the system, when the importance of the lost data is relatively high, the system operation can be suspended immediately and the system operation can be restored after the lost data compensation is completed, further improving the security of the system.

[0051] In a feasible manner, the data compensation sub-module 202 can also be used to determine whether the weight of the missing first vehicle data, the second vehicle data, or the target file is greater than the preset weight threshold after the missing first vehicle data, the second vehicle data, or the target file is not obtained within the preset compensation time threshold. If not, an error message is generated.

[0052] When the weight of the missing data is less than the preset weight threshold, the missing data is less important to the system. Therefore, it is not necessary to suspend the system operation, and only an error message is generated to prompt the loss of the data.

[0053] In the embodiment of the present application, by setting the weight threshold and assigning weights to all vehicle data according to the importance of the data to the system, it is possible not to affect the continuous operation of the system when the data with relatively low importance is lost, thereby improving the operation efficiency of the system.

[0054] Figure 3 It is a schematic diagram of a data processing system including a numbering device provided by an embodiment of the present application. As Figure 3 shown, the data processing system 100 may include: a numbering device 301.

[0055] The numbering device 301 is used to generate corresponding identification information for each piece of first automotive data, where different pieces of first automotive data correspond to different identification information.

[0056] Before the acquisition module 101 acquires the first automotive data, the numbering device 301 generates a unique number, such as a combination of numbers and English letters, and assigns the numbers to different data in the first automotive data one by one.

[0057] The storage module 103 is used to perform duplicate removal on the target file according to the identification information of the first automotive data corresponding to the target file, and store the target file after duplicate removal in the file storage.

[0058] Since there may be data processing errors resulting in duplicate data during the process of the second workflow engine processing the second automotive data and sending the metadata of the second automotive data to the second message queue, before the storage module 103 writes the target file into the file storage, the unique number assigned to the data by the numbering device 301 can be used to perform duplicate removal on the target file, and the data with duplicate unique numbers is deleted until there is only one, completing the duplicate removal of the target file.

[0059] In the embodiment of the present application, the storage module 103 can perform duplicate removal on the duplicate automotive data through the unique number assigned to the automotive data by the numbering device 301, ensuring the uniqueness of the data stored in the storage module 103, and thus improving the effectiveness of the system.

[0060] Figure 4 is a schematic diagram of a data processing system including a data management module and a data display module provided by an embodiment of the present application, as Figure 4 shown, the data processing system 100 may include a data management module 401 and a data display module 402.

[0061] The data management module 401 is used to monitor the acquisition module 101, the processing module 102, and the storage module 103, obtain at least one of the task execution progress information, the storage resource space distribution information, the system resource occupancy information, and the gateway notification record information, and perform permission management and configuration management on the acquisition module 101, the processing module 102, and the storage module 103.

[0062] When the acquisition module 101 acquires the first vehicle data and the processing module 102 processes the second vehicle data, task execution progress information will be generated respectively. When the storage module 103 stores relevant data, the relevant data will occupy storage resource space, and the storage module 103 will correspondingly generate storage resource space distribution information, which shows the internal resource usage of the file storage. The acquisition module 101, the processing module 102, and the storage module 103 will occupy system resources during operation and generate system resource occupancy information at the same time. After the storage module 103 completes the storage operation, it will notify the gateway of the record information, indicating that the gateway system has completed the processing of the data. The data management module 401 can monitor the acquisition module 101, the processing module 102, and the storage module 103 by obtaining the above-mentioned task execution progress information, storage resource space distribution information, system resource occupancy information, and gateway notification record information.

[0063] The data display module 402 is used to display at least one of the task execution progress information, storage resource space distribution information, system resource occupancy information, and gateway notification record information.

[0064] After the data management module 401 obtains the task execution progress information, storage resource space distribution information, system resource occupancy information, and gateway notification record information, the data display module 402 will display different information to the user respectively.

[0065] In the embodiment of the present application, through the data management module 401, the operation progress of the acquisition module 101, the processing module 102, and the storage module 103 can be monitored, and corresponding permission management can be performed during external calls or manipulations, ensuring the security of the system. Through the data display module 402, information such as task execution progress information, storage resource space distribution information, system resource occupancy information, and gateway notification record information can be displayed to the user, improving the user experience.

[0066] Data processing method

[0067] Figure 5 is a schematic diagram of a data processing method provided by an embodiment of the present application. As Figure 5 shown, the data processing method in this embodiment includes:

[0068] Step 501: Obtain the first vehicle data and send the first metadata of the first vehicle data to the first message queue, where the first vehicle data includes data generated during the vehicle's autonomous driving process.

[0069] Step 502: Obtain the first metadata in the first message queue, obtain the first vehicle data according to the first metadata, process the first vehicle data through the first workflow engine to obtain the second vehicle data, and after determining that the second vehicle data meets the preset data processing requirements, send the second metadata of the second vehicle data to the second message queue.

[0070] Step 503: Obtain the second metadata in the second message queue, obtain the second vehicle data according to the second metadata through the second workflow engine, convert the second vehicle data into a target file, and store the target file in the file storage.

[0071] Step 504: Perform data integrity verification on the first vehicle data, the second vehicle data, and the target file, and compensate for the missing first vehicle data, second vehicle data, and target file.

[0072] In the embodiments of the present application, by using at least one first workflow engine to process different types of first vehicle data respectively, and using at least one second workflow engine to process different types of second vehicle data respectively, the processing efficiency of the system can be improved. By obtaining, processing, and storing vehicle data, the integrity of data processing is ensured. By performing data integrity verification on the first vehicle data, the second vehicle data, and the target file, possible missing data can be discovered. By compensating for the missing data, the integrity of the data can be ensured.

[0073] It should be noted that the interactions between the above method embodiments and the respective modules in the foregoing device embodiments are based on the same inventive concept. For specific content, reference can be made to the descriptions in the foregoing method embodiments, and details will not be elaborated here.

[0074] Electronic device

[0075] In this embodiment, an electronic device 600 is provided, as Figure 6 shown. The electronic device 600 may include: a processor 601, a communication interface 602, a memory 603, and a communication bus 604. Among them:

[0076] The processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0077] The communication interface 602 is used to communicate with other electronic devices or servers.

[0078] The processor 601 is used to execute the program 605, and specifically can execute the relevant steps in the foregoing method embodiments of motion control.

[0079] Specifically, the program 605 may include program code that includes computer operation instructions.

[0080] The processor 601 may be a CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. One or more processors included in the intelligent device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0081] The memory 603 is used to store the program 605. The memory 603 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

[0082] The program 605 may specifically be used to cause the processor 601 to execute the data processing method in the foregoing embodiments.

[0083] For the specific implementation of each step in the program 605, reference may be made to the corresponding steps and descriptions in the corresponding units in the foregoing data processing method embodiments, which will not be elaborated here. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, which will not be elaborated here.

[0084] Through the electronic device of the embodiments of the present application, by using at least one first workflow engine to process different types of first vehicle data respectively, and using at least one second workflow engine to process different types of second vehicle data respectively, the processing efficiency of the system can be improved. By acquiring, processing, and storing vehicle data, the integrity of data processing is ensured. By performing data integrity verification on the first vehicle data, the second vehicle data, and the target file, possible missing data can be discovered. By compensating for the missing data, the integrity of the data can be ensured.

[0085] Computer storage medium

[0086] In this embodiment, a computer-readable storage medium is provided, storing instructions for causing a machine to execute the data processing method as described herein. Specifically, a system or device equipped with a storage medium may be provided, on which software program code for implementing the functions of any one of the foregoing embodiments is stored, and causing the computer (or CPU or MPU) of the system or device to read and execute the program code stored in the storage medium.

[0087] In this case, the program code read from the storage medium itself can implement the functions of any one of the above-described embodiments. Therefore, the program code and the storage medium storing the program code constitute a part of the present application.

[0088] Examples of the storage medium for providing the program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer via a communication network.

[0089] Computer program product

[0090] In this embodiment, a computer program product is provided, including computer instructions that direct a computing device to perform any corresponding operation in the above-described multiple method embodiments.

[0091] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of a component / step can be combined into a new component / step to achieve the purpose of the embodiments of the present application.

[0092] The above method according to the embodiments of the present application can be implemented in hardware, firmware, or be implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or be implemented as computer code that is originally stored in a remote recording medium or a non-transitory machine-readable medium and downloaded via a network and will be stored in a local recording medium. Thus, the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (such as RAM, ROM, flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for implementing the method shown herein.

[0093] It should be noted that in the description of the present application, the terms "first" and "second" are only used for convenience in describing different components or names, and cannot be understood as indicating or implying a sequential relationship, relative importance, or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features.

[0094] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0095] It should be noted that although the specific embodiments of this application have been described in detail in conjunction with the accompanying drawings, it should not be construed as a limitation on the scope of protection of this application. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of this application.

[0096] The examples of the embodiments of this application are intended to briefly illustrate the technical features of the embodiments of this application, so that those skilled in the art can intuitively understand the technical features of the embodiments of this application and are not used as an improper limitation of the embodiments of this application.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing system (100), comprising: a collection module (101) for acquiring first vehicle data and sending first metadata of the first vehicle data to a first message queue, wherein the first vehicle data includes data generated during the automatic driving of a vehicle; a processing module (102) for acquiring the first metadata in the first message queue, obtaining the first vehicle data according to the first metadata, processing the first vehicle data through a first workflow engine to obtain second vehicle data, and after determining that the second vehicle data meets a preset data processing requirement, sending second metadata of the second vehicle data to a second message queue; a storage module (103) for acquiring the second metadata in the second message queue, obtaining the second vehicle data according to the second metadata through a second workflow engine, converting the second vehicle data into a target file, and storing the target file in a file storage; a data verification and compensation module (104) for performing data integrity verification on the first vehicle data, the second vehicle data, and the target file, and compensating for the missing first vehicle data, second vehicle data, and target file.

2. The system according to claim 1, wherein, the collection module (101) is configured to convert structured data in the first vehicle data into unstructured data, and store the unstructured data in the first vehicle data and the converted unstructured data in a first object storage system; the processing module (102) is configured to store the second vehicle data in a second object storage system.

3. The system according to claim 2, wherein, the collection module (101) generates a collection log according to a collection operation, unstructured data in the first vehicle data, and the converted unstructured data; the processing module (102) generates a processing log according to a processing operation on the first vehicle data and the second vehicle data; the storage module (103) is configured to generate a storage log according to a storage operation on the second vehicle data and the second vehicle data, and send a message notification to a gateway.

4. The system according to claim 3, wherein, the data verification and compensation module (104) includes: a data verification sub-module (201) for verifying the first vehicle data according to the collection log through a third workflow engine, verifying the second vehicle data according to the processing log, verifying the target file according to the storage log, and generating a verification log according to verification operations on the first vehicle data, the second vehicle data, and the target file; A data compensation sub-module (202) is configured to, when the data verification sub-module (201) determines that the first vehicle data is missing, compensate the first vehicle data according to the acquisition log, and when the data verification sub-module (201) determines that the second vehicle data is missing, compensate the second vehicle data according to the processing log, and when the data verification sub-module (201) determines that the target file is missing, compensate the target file according to the storage log.

5. The system according to claim 4, wherein, the data compensation sub-module (202) is configured to generate a data missing log according to the missing first vehicle data, the second vehicle data or the target file, and the compensation operation if the missing first vehicle data, the second vehicle data or the target file is not obtained within a preset compensation time threshold.

6. The system according to claim 5, wherein, the data compensation sub-module (202) is configured to, after the missing first vehicle data, the second vehicle data or the target file is not obtained within a preset compensation time threshold, determine whether the weight of the missing first vehicle data, the second vehicle data or the target file is greater than a preset weight threshold. If so, the acquisition module (101), the processing module (102) and the storage module (103) are suspended from running, and an error message is generated.

7. The system according to claim 1, wherein, the system further includes: a numbering device (301); the numbering device (301) is configured to generate corresponding identification information for each of the first vehicle data, wherein different first vehicle data correspond to different identification information; the storage module (103) is configured to perform duplicate removal processing on the target file according to the identification information of the first vehicle data corresponding to the target file, and store the target file after duplicate removal processing in the file storage.

8. The system according to claim 1, wherein, the system further includes: a data management module (401) configured to monitor the acquisition module (101), the processing module (102) and the storage module (103) to obtain at least one of task execution progress information, storage resource space distribution information, system resource occupancy information and gateway notification record information, and perform permission management and configuration management on the acquisition module (101), the processing module (102) and the storage module (103); a data display module (402) configured to display at least one of the task execution progress information, the storage resource space distribution information, the system resource occupancy information and the gateway notification record information.

9. A data processing method, including: acquiring first vehicle data, and sending first metadata of the first vehicle data to a first message queue, wherein the first vehicle data includes data generated during the vehicle's autonomous driving process; Obtain the first metadata in the first message queue, obtain the first vehicle data according to the first metadata, process the first vehicle data through a first workflow engine to obtain second vehicle data, and after determining that the second vehicle data meets the preset data processing requirements, send the second metadata of the second vehicle data to a second message queue; Obtain the second metadata in the second message queue, obtain the second vehicle data according to the second metadata through a second workflow engine, convert the second vehicle data into a target file, and store the target file in a file storage; Perform data integrity verification on the first vehicle data, the second vehicle data, and the target file, and compensate for the missing first vehicle data, second vehicle data, and target file.

10. An electronic device (600), comprising: a processor (601), a communication interface (602), a memory (603), and a communication bus (604), and the processor (601), the memory (603), and the communication interface (602) complete mutual communication through the communication bus (604); The memory (603) is used to store at least one executable instruction, and the executable instruction causes the processor (601) to perform operations corresponding to the data processing method described in claim 9.

11. A computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the data processing method described in claim 9.

12. A computer program product, comprising computer instructions, and the computer instructions instruct a computing device to perform operations corresponding to the data processing method described in claim 9.

Images