A data processing method, device, equipment and medium

Abstract

The present invention discloses a data processing method, device, equipment and medium, belonging to the field of data processing technology. The method comprises: obtaining the vehicle data sent by the data sending end to the message queue, and determining the data base amount of the vehicle data; in response to the real-time consumption operation of the vehicle data by the data processing end, determining the consumption data and the data consumption amount of the consumption data; in response to the storage operation of the consumption data by the data landing end, determining the data storage amount of the stored data; and performing data consistency verification according to the data base amount, the data consumption amount and the data storage amount. Through the above technical scheme, the accuracy and integrity of the data during transmission and processing can be guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, equipment and medium. Background Art

[0002] With the widespread use of big data applications, streaming data processing has gradually become an important means of processing massive real-time data. However, in actual application scenarios, streaming data processing has many problems, such as data loss, duplication, time series and other consistency issues, which have brought many troubles to business applications. For example, in the field of connected vehicles, consistency issues may lead to incorrect judgments and delayed alarms of vehicle failures and abnormalities in vehicle health status detection; in vehicle safety and assisted driving systems, consistency issues may lead to loss or errors in the sensor data processing process, which may directly lead to abnormal judgments of the vehicle's surrounding environment and road conditions, thereby affecting the vehicle's decision-making and control, and endangering personal and road safety. Therefore, it is very important to ensure the accuracy and completeness of data during transmission and processing. Summary of the invention

[0003] The present invention provides a data processing method, device, equipment and medium to ensure the consistency of full-link data and determine the accuracy and integrity of data during transmission and processing.

[0004] According to one aspect of the present invention, there is provided a data processing method, the method comprising:

[0005] Acquire the vehicle computer data sent by the data sending end to the message queue, and determine the data base amount of the vehicle computer data;

[0006] In response to the real-time consumption operation of the vehicle data by the data processing end, determining the consumption data and the data consumption amount of the consumption data;

[0007] In response to a storage operation of the consumption data by the data landing end, determining a data storage capacity of the stored data;

[0008] A data consistency check is performed according to the data reference amount, the data consumption amount and the data storage amount.

[0009] According to another aspect of the present invention, there is provided a data processing device, the device comprising:

[0010] A data acquisition module, used to acquire the vehicle data sent by the data sending end to the message queue, and determine the data reference amount of the vehicle data;

[0011] A consumption data determination module, configured to determine consumption data and a data consumption amount of the consumption data in response to a real-time consumption operation of the vehicle data by the data processing end;

[0012] A data storage capacity determination module, configured to determine the data storage capacity of the stored data in response to a storage operation of the consumption data by the data landing end;

[0013] The data verification module is used to perform data consistency verification based on the data reference amount, the data consumption amount and the data storage amount.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data processing method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention obtains the vehicle data sent by the data sending end to the message queue, and determines the data base amount of the vehicle data, and then responds to the real-time consumption operation of the vehicle data by the data processing end to determine the consumption data and the data consumption amount of the consumption data, and then responds to the storage operation of the consumption data by the data landing end to determine the data storage amount of the stored data, and finally performs data consistency verification based on the data base amount, data consumption amount and data storage amount. The above technical solution monitors and verifies the consistency of data in real time, so as to ensure the accuracy and integrity of data during transmission and processing.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a data processing method provided according to Embodiment 1 of the present invention;

[0023] Figure 2 is a flow chart of a data processing method provided according to Embodiment 2 of the present invention;

[0024] Figure 3 is a structural schematic diagram of a data processing device provided according to Embodiment 3 of the present invention;

[0025] Figure 4 It is a structural schematic diagram of an electronic device for implementing the data processing method of an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In addition, it should be noted that the collection, storage, use, processing, transmission, provision and disclosure of vehicle data involved in the technical solution of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0029] Embodiment 1

[0030] Figure 1is a flow chart of a data processing method provided according to the first embodiment of the present invention. This embodiment is applicable to the situation of how to ensure data consistency during the streaming data processing in the field of vehicle networking. The method can be executed by a data processing device, which can be implemented in the form of hardware and / or software and can be integrated into an electronic device that carries the data processing function, such as a big data platform in a server. Figure 1 As shown, the data processing method of this embodiment may include:

[0031] S110, obtaining the vehicle computer data sent by the data sending end to the message queue, and determining the data reference amount of the vehicle computer data.

[0032] In this embodiment, the data sending end refers to the data provider of the vehicle computer data. The so-called vehicle computer data refers to the data related to vehicle computer connection in the field of vehicle connection. The so-called data reference volume refers to the data volume of the vehicle computer data uploaded to the big data platform, which is used for the consistency judgment in the subsequent vehicle computer data processing process.

[0033] Specifically, the data sending end can upload the vehicle computer data to the message queue of the big data platform based on the 4G / 5G signal through the Restful interface, real-time data integration tools (such as Oracle Golden Gate), and the vehicle computer module, and upload the data transmission volume of the vehicle computer data at the same time; accordingly, the big data platform obtains the vehicle computer data in the message queue, and stores the uploaded data transmission volume as the data baseline of the vehicle computer data.

[0034] Furthermore, some tool applications cannot count the data transmission volume of the vehicle data. In this case, the data baseline volume of the vehicle data can be determined by relying on the actual data reception volume in the message queue. Specifically, the data baseline volume of the vehicle data can be determined based on the starting data offset and ending data offset in the message queue.

[0035] The start data offset is the offset mark when the data reception starts, and the end data offset is the offset mark when the data reception is completed.

[0036] Specifically, the difference between the end data offset and the start data offset in the message queue may be used as the data reference quantity of the vehicle computer data.

[0037] It can be understood that by determining the data baseline of the vehicle computer data through the data offset in the message queue, the data baseline can be accurately determined when the data transmission amount cannot be accurately obtained, thereby laying the foundation for subsequent data consistency verification.

[0038] S120 , in response to the real-time consumption operation of the vehicle computer data by the data processing end, determining the consumption data and the data consumption amount of the consumption data.

[0039] In this embodiment, the data processing end refers to the end that consumes the vehicle data, and can process and analyze the vehicle data in real time. The so-called consumption data refers to the vehicle data consumed by the data processing end. The so-called data consumption volume refers to the data volume of the consumption data. The real-time consumption operation refers to the related operation of processing the vehicle data.

[0040] Specifically, the data processing end can perform real-time processing and analysis on the received data based on mainstream real-time processing technology stacks, such as Apache Flink, FlinkSQL, and Spark streaming. During the data processing process, the data consumption of the consumed data is counted and recorded in real time.

[0041] S130. Determine the data storage capacity of the stored data in response to the storage operation of the consumption data by the data landing end.

[0042] In this embodiment, the data landing end refers to an end for storing consumption data. The so-called storage data refers to the data for storing consumption data, that is, the landing data. The so-called data storage capacity refers to the amount of stored data.

[0043] Specifically, the data landing end can store consumption data in the storage system. Common storage systems include Hive and HBase. One optional method is to use Hive's partitioning function to partition and store consumption data by date when writing consumption data into the Hive table. After each day, count the amount of data in each partition to obtain the data storage capacity of the stored data. Another optional method is to write consumption data into HBase. You can add counters to the HBase column cluster or introduce data version management to count the amount of stored data, thereby determining the data storage capacity of the stored data.

[0044] S140: Perform data consistency check based on the data baseline, data consumption and data storage capacity.

[0045] An optional method is to determine whether the consumption data in the data processing end is consistent with the vehicle computer data based on the data baseline and data consumption; and to determine whether the storage data in the data landing end is consistent with the vehicle computer data based on the data baseline and data storage capacity.

[0046] Specifically, if the data base amount is consistent with the data consumption amount, it is determined that the consumption data in the data processing end is consistent with the vehicle computer data. If the data base amount is consistent with the data storage amount, it is determined that the storage data in the data landing end is consistent with the vehicle computer data.

[0047] It can be understood that by comparing the amount of data during the transmission and processing of data with the data reference amount, the integrity of the data during the transmission and processing can be ensured.

[0048] Furthermore, if the consumption data is inconsistent with the vehicle data, and / or the storage data is inconsistent with the vehicle data, the data processing end and / or the data storage end is instructed to perform a data reprocessing operation. In an optional manner, if the consumption data is inconsistent with the vehicle data, the data processing end is instructed to perform a data reprocessing operation, that is, to consume the vehicle data again. If the storage data is inconsistent with the vehicle data, the data storage end is instructed to reprocess the consumption data process, that is, to store the consumption data again.

[0049] It is understandable that during the data transmission and processing process, adding data reprocessing operations can enable the data processing end or the data landing end to reprocess the data, thereby ensuring whether the data transmission and processing process is really abnormal.

[0050] If, after data reprocessing, the consumption data is inconsistent with the vehicle computer data, and / or the storage data is inconsistent with the vehicle computer data, an alarm message is generated to instruct relevant personnel to handle the exception.

[0051] It is understandable that after data reprocessing, if the consumption data and storage data are still inconsistent with the vehicle computer data, an alarm will be issued to the relevant personnel.

[0052] The technical solution of the embodiment of the present invention obtains the vehicle data sent by the data sending end to the message queue, and determines the data base amount of the vehicle data, and then responds to the real-time consumption operation of the vehicle data by the data processing end to determine the consumption data and the data consumption amount of the consumption data, and then responds to the storage operation of the consumption data by the data landing end to determine the data storage amount of the stored data, and finally performs data consistency verification based on the data base amount, data consumption amount and data storage amount. The above technical solution monitors and verifies the consistency of data in real time, so as to ensure the accuracy and integrity of data during transmission and processing.

[0053] Embodiment 2

[0054] Figure 2 1 is a flow chart of a data processing method provided according to Embodiment 2 of the present invention. Based on the above embodiment, this embodiment monitors the full link data and provides an optional solution. Figure 2 As shown, the data processing method of this embodiment may include:

[0055] S210, obtaining the vehicle computer data sent by the data sending end to the message queue, and determining the data reference amount of the vehicle computer data.

[0056] S220 , in response to the real-time consumption operation of the vehicle data by the data processing end, determine the consumption data and the data consumption amount of the consumption data.

[0057] S230. Determine the data storage capacity of the stored data in response to the storage operation of the consumption data by the data landing end.

[0058] S240: Perform data consistency check based on the data baseline, data consumption and data storage capacity.

[0059] S250: Real-time data monitoring of the message queue, data processing end, and data landing end.

[0060] In this embodiment, each link in the entire data processing link can be monitored in real time.

[0061] An optional method is to monitor the interruption of the message queue according to the data receiving frequency of the message queue; monitor the task processing status of the data processing end according to the application program interface API status of the data processing end; monitor the consumption delay of the data processing end according to the data consumption rate of the data processing end; and perform data quality verification on the storage data at the data landing end; the data quality verification includes at least one of the following: data integrity verification, data accuracy verification, data legitimacy verification and data anomaly detection.

[0062] The data receiving frequency refers to the frequency at which the message queue receives data, such as receiving data every 15 seconds or once an hour. The so-called data consumption rate refers to the rate at which the processing end consumes data.

[0063] Optionally, the data receiving frequency of the message queue is compared with the preset receiving frequency. If they are consistent, it means that the message queue receives data normally. If they are inconsistent, it means that the message queue is disconnected, that is, the data cannot be received accurately.

[0064] Optionally, the application program interface API status of the data processing end is obtained, and the task processing status of the data processing end is monitored according to whether the API interface status is interrupted, in progress, or normal.

[0065] Optionally, the data consumption rate of the data processing end is compared with the preset consumption rate. If they are consistent, it is determined that there is no consumption delay when the data processing end processes the task in real time. If they are inconsistent, it is determined that there is a consumption delay when the data processing end processes the task in real time.

[0066] Optionally, the quality of the stored data at the landing end can be checked. For example, the integrity of the stored data can be checked based on whether the data volume is consistent. In another example, the accuracy of the stored data can be checked based on business logic (such as whether the voltage and current are within the normal range). In another example, the legality of the stored data can be checked by checking whether the stored data is uploaded encrypted and meets the preset defined indicators. In another example, data anomalies can be detected by checking whether the stored data is empty or within the preset range.

[0067] It is understandable that monitoring is performed at each node of data processing to ensure the accuracy of data in the data processing link.

[0068] The technical solution provided by the embodiment of the present invention obtains the vehicle data sent by the data sending end to the message queue, and determines the data baseline of the vehicle data. Then, in response to the real-time consumption operation of the vehicle data by the data processing end, the consumption data and the data consumption of the consumption data are determined. In response to the storage operation of the consumption data by the data landing end, the data storage volume of the stored data is determined, and then data consistency verification is performed based on the data baseline, data consumption and data storage. At the same time, the message queue, data processing end and data landing end are monitored in real time. The above technical solution can detect and handle anomalies in a timely manner by monitoring the entire link data processing process, thereby ensuring the accuracy, integrity and consistency of the data in the entire processing process.

[0069] Embodiment 3

[0070] Figure 3 : is a structural diagram of a data processing device provided according to Embodiment 3 of the present invention. This embodiment can be applied to the situation of how to ensure data consistency during the streaming data processing in the field of vehicle networking. The device can be implemented in the form of hardware and / or software, and can be integrated into an electronic device that carries the data processing function, such as a big data platform in a server. Figure 3 As shown, the data processing device of this embodiment may include:

[0071] The data acquisition module 310 is used to acquire the vehicle computer data sent by the data sending end to the message queue and determine the data reference amount of the vehicle computer data;

[0072] The consumption data determination module 320 is used to determine the consumption data and the data consumption amount of the consumption data in response to the real-time consumption operation of the vehicle data by the data processing end;

[0073] A data storage capacity determination module 330, for determining the data storage capacity of the stored data in response to the storage operation of the consumption data by the data landing end;

[0074] The data verification module 340 is used to perform data consistency verification based on the data reference amount, data consumption amount and data storage amount.

[0075] The technical solution of the embodiment of the present invention obtains the vehicle data sent by the data sending end to the message queue, and determines the data base amount of the vehicle data, and then responds to the real-time consumption operation of the vehicle data by the data processing end to determine the consumption data and the data consumption amount of the consumption data, and then responds to the storage operation of the consumption data by the data landing end to determine the data storage amount of the stored data, and finally performs data consistency verification based on the data base amount, data consumption amount and data storage amount. The above technical solution monitors and verifies the consistency of data in real time, so as to ensure the accuracy and integrity of data during transmission and processing.

[0076] Optionally, the data acquisition module 310 is specifically used for:

[0077] The data base amount of the vehicle computer data is determined according to the starting data offset and the ending data offset in the message queue.

[0078] Optionally, the data verification module 340 is used to:

[0079] According to the data baseline and data consumption, determine whether the consumption data in the data processing end is consistent with the vehicle computer data;

[0080] Based on the data baseline and data storage volume, determine whether the data stored in the data landing end is consistent with the vehicle computer data.

[0081] Optionally, the data verification module 340 further includes:

[0082] The data reprocessing unit is used to instruct the data processing end and / or the data storage end to perform a data reprocessing operation if the consumption data is inconsistent with the vehicle computer data and / or the storage data is inconsistent with the vehicle computer data.

[0083] Optionally, the data verification module 340 further includes:

[0084] The data alarm unit is used to generate an alarm message to instruct relevant personnel to handle the exception if the consumption data is inconsistent with the vehicle data after data reprocessing, and / or the storage data is inconsistent with the vehicle data.

[0085] Optionally, the device further comprises:

[0086] The data monitoring module is used to monitor the message queue, data processing end and data landing end in real time.

[0087] Optionally, the data monitoring module is specifically used for:

[0088] Monitor the interruption of the message queue according to the frequency of receiving data in the message queue;

[0089] Monitor the task processing status of the data processing end according to the API status of the data processing end;

[0090] Monitor the consumption delay of the data processing end according to the data consumption rate of the data processing end;

[0091] Perform data quality verification on the stored data at the data landing end; data quality verification includes at least one of the following: data integrity verification, data accuracy verification, data legitimacy verification and data anomaly detection.

[0092] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0093] Embodiment 4

[0094] Figure 4 is a schematic diagram of the structure of an electronic device for implementing the data processing method according to an embodiment of the present invention; Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0095] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0096] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0097] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data processing method.

[0098] In some embodiments, the data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0099] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0101] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0102] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0103] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0104] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0105] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0106] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method, It is characterized in that include: Acquire the vehicle computer data sent by the data sending end to the message queue, and determine the data base amount of the vehicle computer data; The data sending end uploads the vehicle data to the message queue of the big data platform through the Restful interface, real-time data integration tool, and vehicle module based on the 4G / 5G signal, and uploads the data transmission volume of the vehicle data at the same time. Correspondingly, the big data platform obtains the vehicle data in the message queue and stores the uploaded data transmission volume as the data benchmark of the vehicle data. Wherein, the tool application that is unable to count the data transmission amount of the vehicle data relies on the actual data reception amount in the message queue to determine the data baseline amount of the vehicle data; In response to the real-time consumption operation of the vehicle data by the data processing end, determining the consumption data and the data consumption amount of the consumption data; The data processing end is a real-time processing technology stack that performs real-time processing and analysis on the received data; In response to a storage operation of the consumption data by the data landing end, determining a data storage capacity of the stored data; Performing a data consistency check according to the data reference amount, the data consumption amount, and the data storage amount; The vehicle data refers to the data related to vehicle connectivity in the field of vehicle connectivity; the data benchmark refers to the data volume of the vehicle data uploaded to the big data platform; The data landing end refers to an end used to store the consumption data; the stored data refers to the data storing the consumption data, i.e., the landing data; the data storage capacity refers to the data volume of the stored data; The method further comprises: performing real-time data monitoring on the message queue, the data processing end and the data landing end; The real-time monitoring of the message queue, the data processing end, and the data landing end includes: Performing interruption monitoring on the message queue according to a data receiving frequency of the message queue receiving data; Monitoring the task processing status of the data processing end according to the application program interface API status of the data processing end; Monitoring the consumption delay of the data processing end according to the data consumption rate of the data processing end; Performing data quality verification on the stored data at the data landing end; the data quality verification includes at least one of the following: data integrity verification, data accuracy verification, data legitimacy verification and data anomaly detection; The step of monitoring the message queue for interruption according to the frequency of receiving data received by the message queue includes: The data receiving frequency of the message queue receiving data is compared with the preset receiving frequency. If they are consistent, it means that the message queue receives data normally. If they are inconsistent, it means that the message queue is disconnected and cannot accurately receive data. The monitoring of the consumption delay of the data processing end according to the data consumption rate of the data processing end includes: Comparing the data consumption rate of the data processing end with the preset consumption rate, if they are consistent, it is determined that there is no consumption delay when the data processing end processes the task in real time; if they are inconsistent, it is determined that there is a consumption delay when the data processing end processes the task in real time; The data quality verification of the stored data at the data landing end includes: Based on whether the data amounts are consistent, performing integrity check on the stored data; Verifying the accuracy of the stored data based on whether the voltage and current are within a normal range; Check the legality of the stored data by checking whether the stored data is uploaded encrypted and whether it meets the preset definition indicators; Check whether the stored data is a null value or exceeds a preset range, and perform data anomaly detection on the stored data.

2. The method according to claim 1, It is characterized in that The step of determining the data reference amount of the vehicle computer data includes: The data reference amount of the vehicle computer data is determined according to the start data offset and the end data offset in the message queue.

3. The method according to claim 1, It is characterized in that The performing of data consistency check according to the data reference amount, the data consumption amount and the data storage amount includes: Determining whether the consumption data in the data processing terminal is consistent with the vehicle computer data according to the data reference amount and the data consumption amount; According to the data reference amount and the data storage amount, determine whether the data stored in the data landing terminal is consistent with the vehicle computer data.

4. The method according to claim 3, It is characterized in that Also includes: If the consumption data is inconsistent with the vehicle computer data, instructing the data processing end to perform a data reprocessing operation; If the stored data is inconsistent with the vehicle computer data, the data landing end is instructed to perform a data reprocessing operation.

5. The method according to claim 4, It is characterized in that Also includes: If, after data reprocessing, the consumption data is inconsistent with the vehicle data, and / or the storage data is inconsistent with the vehicle data, an alarm message is generated to instruct relevant personnel to handle the exception.

6. A data processing device, It is characterized in that include: A data acquisition module, used to acquire the vehicle data sent by the data sending end to the message queue, and determine the data reference amount of the vehicle data; A consumption data determination module, configured to determine consumption data and a data consumption amount of the consumption data in response to a real-time consumption operation of the vehicle data by the data processing end; A data storage capacity determination module, configured to determine the data storage capacity of the stored data in response to a storage operation of the consumption data by the data landing end; A data verification module, used for performing data consistency verification according to the data reference amount, the data consumption amount and the data storage amount; The vehicle data refers to the data related to vehicle connectivity in the field of vehicle connectivity; the data benchmark refers to the data volume of the vehicle data uploaded to the big data platform; The data landing end refers to an end used to store the consumption data; the stored data refers to the data storing the consumption data, i.e., the landing data; the data storage capacity refers to the data volume of the stored data; A data monitoring module, used to monitor the message queue, the data processing end and the data landing end in real time; The data monitoring module is specifically used to monitor the interruption of the message queue according to the data receiving frequency of the message queue receiving data; monitor the task processing status of the data processing end according to the application program interface API status of the data processing end; monitor the consumption delay of the data processing end according to the data consumption rate of the data processing end; perform data quality verification on the storage data of the data landing end; the data quality verification includes at least one of the following: data integrity verification, data accuracy verification, data legitimacy verification and data anomaly detection; The step of monitoring the message queue for interruption according to the frequency of receiving data received by the message queue includes: The data receiving frequency of the message queue receiving data is compared with the preset receiving frequency. If they are consistent, it means that the message queue receives data normally. If they are inconsistent, it means that the message queue is disconnected and cannot accurately receive data. The monitoring of the consumption delay of the data processing end according to the data consumption rate of the data processing end includes: Comparing the data consumption rate of the data processing end with the preset consumption rate, if they are consistent, it is determined that there is no consumption delay when the data processing end processes the task in real time; if they are inconsistent, it is determined that there is a consumption delay when the data processing end processes the task in real time; The data quality verification of the stored data at the data landing end includes: Based on whether the data amounts are consistent, performing integrity check on the stored data; Verifying the accuracy of the stored data based on whether the voltage and current are within a normal range; Check the legality of the stored data by checking whether the stored data is uploaded encrypted and whether it meets the preset definition indicators; Check whether the stored data is a null value or exceeds a preset range, and perform data anomaly detection on the stored data.

7. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method according to any one of claims 1 to 5 when executed.

Images