Mass data processing method, system and equipment and storage medium

Abstract

The invention discloses a mass data processing method, system and device and a storage medium, and relates to the technical field of Internet of Vehicles. According to the specific technical scheme, firstly, a vehicle end uploads vehicle data to an MQTT Broker cluster, wherein the vehicle data includes but is not limited to real-time data of unique identification, travel and driving of a vehicle; the MQTT Broker cluster receives the vehicle data according to the MQTT theme corresponding to the vehicle data, filters the vehicle data, and sends the filtered vehicle data to a Kafka message queue according to a corresponding theme mapping relation; and finally, the Kafka message queue sends the vehicle data to the platform end according to the Kafka theme subscribed by the platform end. According to the method, the data processing requirement in a mass data scene in the field of the Internet of Vehicles can be met, and the stability, timeliness and high availability of message forwarding are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle networking, and in particular to a method, system, device and storage medium for processing massive data. Background Art

[0002] The Internet of Vehicles uses moving vehicles as information perception objects, and with the help of new-generation information and communication technologies, it realizes network connections between vehicles, people, roads, and service platforms, improves the overall intelligent driving level of vehicles, and provides users with safe, comfortable, intelligent, and efficient driving experience and transportation services, while improving traffic operation efficiency and enhancing the intelligence level of social transportation services.

[0003] Currently in the Internet of Vehicles industry, when the vehicle uploads massive amounts of data to the cloud, the existing technical solutions for data stream processing from data reception to data consumption mainly focus on the business processing level, including using a rule engine, sending data through a single Kafka topic, or managing it through self-developed services. Specifically, when using a rule engine for data stream processing, users or developers define complex business logic in the form of business rules to convert, process, and forward data streams. The core of their focus is on the business level, reducing the amount of data while processing the business. Sending a single Kafka topic means that a Kafka topic is used by default, all received messages are forwarded to it, and then the downstream service completes the topic classification and processing. Self-developed services handle the message forwarding process from the receiving end to the consumer end by writing a separate service for message forwarding.

[0004] However, these technical solutions still have some problems: the solution using rule engines is generally large in size and resource consumption, and lacks flexibility, because rule engines are mainly used for business processing; sending to a single Kafka topic will cause the consumer to receive a large number of data sets that are irrelevant to the target data, which will not only increase the complexity of subsequent business, but also cause unnecessary data pressure; developing services to handle data forwarding by oneself is costly and will cause unnecessary resource consumption. Summary of the invention

[0005] In view of the above problems, the present invention invents a method, system, device and storage medium for processing massive data. The present invention utilizes Erlang, which is good at processing a large number of concurrent connections; MQTT, which is good at providing low-latency and high-reliability data transmission capabilities for massive IoT devices; and Kafka, which is good at efficiently processing the collected data streams in massive amounts, with high fault tolerance and queue storage, to meet the data processing needs in the massive data scenarios in the field of Internet of Vehicles and ensure the stability, timeliness and high availability of message forwarding. The technical solution is as follows: According to a first aspect of an embodiment of the present invention, a method for processing massive data is provided, the method comprising: The vehicle uploads vehicle data to the MQTT Broker cluster, where the vehicle data includes but is not limited to the vehicle's unique identification, itinerary, and real-time driving data; The MQTT Broker cluster receives the vehicle data according to the MQTT topic corresponding to the vehicle data, and filters the vehicle data; The MQTT Broker cluster sends the filtered vehicle data to the Kafka message queue according to the corresponding topic mapping relationship; The Kafka message queue sends the vehicle data to the platform according to the Kafka topic subscribed by the platform.

[0006] The method for processing massive data provided by the embodiment of the present invention firstly uploads vehicle data to the MQTT Broker cluster, and the vehicle data includes but is not limited to the unique identification of the vehicle, the itinerary, and the real-time driving data; then the MQTT Broker cluster receives the vehicle data according to the MQTT topic corresponding to the vehicle data, and filters the vehicle data; then the filtered vehicle data is sent to the Kafka message queue according to the corresponding topic mapping relationship; finally, the Kafka message queue sends the vehicle data to the platform according to the Kafka topic subscribed by the platform. The method of the present invention can cope with the data processing requirements in the massive data scenario in the field of Internet of Vehicles, and ensure the stability, timeliness, and high availability of message forwarding.

[0007] As a further solution of the present invention: before the MQTT Broker cluster receives the vehicle data according to the MQTT topic corresponding to the vehicle data, the method further includes: The MQTT Broker receives the user's bridge plug-in, which is a program with Kafka topic mapping and message filtering functions.

[0008] The present invention can be directly integrated with the MQTT Broker through a bridge plug-in without requiring other platforms or software, thereby achieving low coupling.

[0009] As a further solution of the present invention: filtering the vehicle data comprises the following steps: The MQTT Broker cluster obtains the MQTT message type corresponding to the vehicle data, and determines whether the message type is a preset message type to be intercepted; If so, the vehicle data is intercepted.

[0010] The present invention intercepts preset message types that need to be intercepted, thereby avoiding the transmission of data that does not need to be processed downstream, reducing additional resource consumption, shortening the data processing link, and reducing the risk of causing failures.

[0011] As a further solution of the present invention: the preset message types to be intercepted include MQTT hook functions and preset data streams.

[0012] The present invention improves the data flow speed by setting the preset message types to be intercepted.

[0013] As a further solution of the present invention: the MQTT Broker cluster sends the filtered vehicle data to the Kafka message queue according to the corresponding topic mapping relationship, including: The MQTT Broker identifies the vehicle unique identifier corresponding to the vehicle data, and obtains the partition identifier by taking the partition number of the Kafka topic of the mapped Kafka message queue modulo the hash value of the vehicle unique identifier; The topic mapping relationship of the vehicle data in the configuration file of the plug-in is read, and the vehicle data received by the topic specified by the MQTT Broker is forwarded to the Kafka message queue according to the partition identifier.

[0014] The present invention distributes the massive messages reported by vehicles through the MQTT protocol to the Kafka message queue according to custom rules to support the processing of subsequent business, ensuring the stability, timeliness and high availability of data flow.

[0015] According to a second aspect of an embodiment of the present invention, there is provided a massive data processing system, comprising: a vehicle side, an MQTTBroker cluster, a Kafka message queue, and a platform side; The vehicle end is used to upload vehicle data to the MQTT Broker cluster, where the vehicle data includes but is not limited to the vehicle's unique identification, itinerary, and real-time driving data; The MQTT Broker cluster is used to receive the vehicle data according to the MQTT topic corresponding to the vehicle data, filter the vehicle data, and then send the filtered vehicle data to the Kafka message queue according to the corresponding topic mapping relationship; The Kafka message queue is used to send the vehicle data to the platform end according to the Kafka topic subscribed by the platform end; The platform end is used to send the subscribed topic content to the Kafka message queue and receive the vehicle data sent by the Kafka message queue.

[0016] The massive data processing system provided by the embodiment of the present invention includes a vehicle side, an MQTT Broker cluster, a Kafka message queue, and a platform side; the vehicle side uploads vehicle data to the MQTT Broker cluster, and the vehicle data includes but is not limited to the unique identification of the vehicle, the itinerary, and the real-time driving data; the MQTT Broker cluster receives the vehicle data according to the MQTT topic corresponding to the vehicle data, and filters the vehicle data, and then sends the filtered vehicle data to the Kafka message queue according to the corresponding topic mapping relationship; the Kafka message queue sends the vehicle data to the platform side according to the Kafka topic subscribed by the platform side; the platform side sends the subscribed topic content to the Kafka message queue, and receives the vehicle data sent by the Kafka message queue. The system of the present invention can cope with the data processing needs in the massive data scenario in the field of Internet of Vehicles, and ensure the stability, timeliness, and high availability of message forwarding.

[0017] As a further solution of the present invention: the MQTT Broker is configured with a bridge plug-in, and the plug-in is a program with Kafka topic mapping and message filtering functions.

[0018] As a further solution of the present invention: the MQTT Broker cluster is specifically used to obtain the MQTT message type corresponding to the vehicle data, and determine whether the message type is a preset message type to be intercepted; when the determination result is yes, the vehicle data is intercepted.

[0019] According to a third aspect of an embodiment of the present invention, a massive data processing device is provided, the massive data processing device comprising a processor and a memory, the memory storing at least one computer instruction, the instruction being loaded and executed by the processor to implement the steps performed in any of the above-mentioned massive data processing methods.

[0020] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein at least one computer instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the steps performed in any of the above-mentioned methods for processing massive data.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0023] Figure 1is a flow chart of a method for processing massive data provided by an embodiment of the present invention; Figure 2 It is a business system structure diagram of a method for processing massive data provided by an embodiment of the present invention; Figure 3 It is a business system timing diagram of a method for processing massive data provided by an embodiment of the present invention; Figure 4 It is a structural diagram of a mass data processing system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention.

[0025] The embodiment of the present invention provides a method for processing massive data. Figure 1 As shown, the mass data processing method includes the following steps: Step 101: The vehicle uploads vehicle data to the MQTT Broker cluster. The vehicle data includes but is not limited to the vehicle's unique identification, itinerary, and real-time driving data. like Figure 2~3 As shown, in this embodiment, the vehicle end, as a message producer, reports vehicle data to the message proxy server MQTT Broker through the message queuing telemetry transport (MQTT) protocol at a high frequency (generally once a second) according to different MQTT topics based on business needs. In addition, real-time driving data may include time, date, weather and other data; the unique identifier of the vehicle is the ClientId (terminal unique identifier) ​​of the vehicle terminal.

[0026] Step 102, the MQTT Broker cluster receives the vehicle data according to the MQTT topic corresponding to the vehicle data, and filters the vehicle data; In this embodiment, when the message proxy server MQTT Broker receives a message from the vehicle side, it reads the MQTT message type and topic. Since MQTT can receive business messages as well as terminal status messages, and terminal status messages do not require processing by downstream services, they need to be filtered before forwarding the data to Kafka.

[0027] In one embodiment, before the MQTT Broker cluster receives the vehicle data according to the MQTT topic corresponding to the vehicle data, the method further includes: MQTT Broker receives the user's bridge plug-in, which is a program with Kafka topic mapping and message filtering functions.

[0028] In this embodiment, the bridge plug-in is configured at the MQTT Broker. In the specific implementation, EMQX is used as the MQTT Broker, and the plug-in is implemented using the Erlang language. The plug-in has Kafka topic mapping and message filtering functions, and is integrated and deployed with the MQTT Broker. The data bridge plug-in is integrated with the MQTT Broker in the form of a plug-in and decoupled from the business. Through this configuration method, the present invention does not require other platforms or software, and can be directly integrated with the MQTT Broker to achieve low coupling.

[0029] In actual use, as a programming language designed specifically for concurrent performance, Erlang uses lightweight processes and message passing models to achieve efficient concurrent programming. This makes Erlang perform well in handling large-scale concurrent tasks, so Erlang is more suitable for handling the message forwarding task here.

[0030] In one embodiment, filtering vehicle data includes the following steps: The MQTT Broker cluster obtains the MQTT message type corresponding to the vehicle data, and determines whether the message type is a preset message type to be intercepted; If so, the vehicle data is intercepted.

[0031] In one embodiment, the preset message types to be intercepted include MQTT hook functions and preset data streams.

[0032] Specifically, the MQTT hook functions that do not need to be processed and the downstream data streams that do not need to be processed are filtered according to business rules to shield the forwarding of the above data during the bridging process.

[0033] Step 103: The MQTT Broker cluster sends the filtered vehicle data to the Kafka message queue according to the corresponding topic mapping relationship.

[0034] In one embodiment, the MQTT Broker cluster sends the filtered vehicle data to the Kafka message queue according to the corresponding topic mapping relationship, including: The MQTT Broker identifies the unique vehicle ID corresponding to the vehicle data, and obtains the partition ID by taking the modulus of the partition number of the Kafka topic of the mapped Kafka message queue according to the hash value of the unique vehicle ID. Read the topic mapping relationship of vehicle data in the configuration file of the plug-in, and forward the vehicle data received by the topic specified by the MQTT Broker to the Kafka message queue according to the partition identifier.

[0035] In actual use, the vehicle data is forwarded to the specified topic and partition of Kafka according to the configured mapping relationship. In fact, it is implemented by calling the brod:produce_cb / 6 function of the brod library. Specifically, it reads the message topic mapping relationship in the configuration file, and forwards the message received by the specified topic of the MQTT Broker to Kafka; identifies the ClientId of the reporting vehicle terminal, and uses the hash value of the ClientId to modulo the number of partitions under the mapped Kafka topic to obtain the partition identifier, ensuring that the calculation results of different ClientIds are in the same partition.

[0036] Step 104, the Kafka message queue sends the vehicle data to the platform end according to the Kafka topic subscribed by the platform end; In actual use, the platform, as a message consumer, first subscribes to the Kafka topic information to the Kafka message queue. The Kafka message queue then sends the corresponding vehicle data to the platform based on the topic information for consumption by the platform.

[0037] The method for processing massive data provided by the embodiment of the present invention firstly uploads vehicle data to the MQTT Broker cluster, and the vehicle data includes but is not limited to the unique identification of the vehicle, the itinerary, and the real-time driving data; then the MQTT Broker cluster receives the vehicle data according to the MQTT topic corresponding to the vehicle data, and filters the vehicle data; then the filtered vehicle data is sent to the Kafka message queue according to the corresponding topic mapping relationship; finally, the Kafka message queue sends the vehicle data to the platform according to the Kafka topic subscribed by the platform. The method of the present invention can cope with the data processing requirements in the massive data scenario in the field of Internet of Vehicles, and ensure the stability, timeliness, and high availability of message forwarding.

[0038] In addition, during the message flow process, the present invention can horizontally expand the system structure according to the specific business scenario and data volume, and can increase the number of MQTT Brokers and Kafka Brokers. At the same time, the QoS of the message can be configured according to the needs, making full use of the characteristics of MQTT and Kafka to provide control over the data flow process in specific scenarios.

[0039] Based on the above Figure 1 The mass data processing method described in the corresponding embodiment is as follows: the system embodiment of the present invention can be used to execute the method embodiment of the present invention.

[0040] The embodiment of the present invention provides a massive data processing system, such as Figure 4 As shown, the system includes a vehicle end 201, an MQTTBroker cluster 202, a Kafka message queue 203, and a platform end 204; The vehicle end 201 is used to upload vehicle data to the MQTT Broker cluster 202. The vehicle data includes but is not limited to the unique identification, itinerary, and real-time driving data of the vehicle; The MQTT Broker cluster 202 is used to receive the vehicle data according to the MQTT topic corresponding to the vehicle data, filter the vehicle data, and then send the filtered vehicle data to the Kafka message queue 203 according to the corresponding topic mapping relationship; Kafka message queue 203, used to send vehicle data to platform end 204 according to the Kafka topic subscribed by platform end 204; The platform end 204 is used to send the subscribed topic content to the Kafka message queue 203 and receive the vehicle data sent by the Kafka message queue 203.

[0041] The massive data processing system provided by the embodiment of the present invention includes a vehicle end 201, an MQTT Broker cluster 202, a Kafka message queue 203, and a platform end 204; the vehicle end 201 uploads vehicle data to the MQTT Broker cluster 202, and the vehicle data includes but is not limited to the unique identification of the vehicle, the itinerary, and the real-time driving data; the MQTT Broker cluster 202 receives the vehicle data according to the MQTT topic corresponding to the vehicle data, filters the vehicle data, and then sends the filtered vehicle data to the Kafka message queue 203 according to the corresponding topic mapping relationship; the Kafka message queue 203 sends the vehicle data to the platform end 204 according to the Kafka topic subscribed by the platform end 204; the platform end 204 sends the subscribed topic content to the Kafka message queue 203, and receives the vehicle data sent by the Kafka message queue 203. The system of the present invention can cope with the data processing requirements in the massive data scenario in the field of Internet of Vehicles, and ensure the stability, timeliness and high availability of message forwarding.

[0042] In one embodiment, a bridge plug-in is configured at the MQTT Broker, and the plug-in is a program with Kafka topic mapping and message filtering functions.

[0043] The present invention can be directly integrated with the MQTT Broker by setting a bridge plug-in, without the need for other platforms or software, thus achieving low coupling.

[0044] In one embodiment, the MQTT Broker cluster 202 is specifically used to obtain the MQTT message type corresponding to the vehicle data, and determine whether the message type is a preset message type to be intercepted; when the determination result is yes, the vehicle data is intercepted.

[0045] The present invention intercepts preset message types that need to be intercepted, thereby avoiding the transmission of data that does not need to be processed downstream, reducing additional resource consumption, shortening the data processing link, and reducing the risk of causing failures.

[0046] In one embodiment, the preset message types to be intercepted include MQTT hook functions and preset data streams.

[0047] The present invention improves the data flow speed by setting the preset message types to be intercepted.

[0048] In one embodiment, the MQTT Broker is specifically used to identify the vehicle unique identifier corresponding to the vehicle data, modulo the partition number of the Kafka topic of the mapped Kafka message queue 203 according to the hash value of the vehicle unique identifier to obtain the partition identifier, read the topic mapping relationship of the vehicle data in the configuration file of the plug-in, and forward the vehicle data received by the topic specified by the MQTT Broker to the Kafka message queue 203 according to the partition identifier.

[0049] The present invention distributes the massive messages reported by vehicles through the MQTT protocol to the Kafka message queue according to custom rules to support the processing of subsequent business, ensuring the stability, timeliness and high availability of data flow.

[0050] Based on the above Figure 1 In accordance with the method for processing massive data described in the embodiment, another embodiment of the present invention further provides a massive data processing device, the massive data processing device comprising a processor and a memory, the memory storing at least one computer instruction, the instruction being loaded and executed by the processor to implement the above Figure 1 The massive data processing method described in the corresponding embodiment.

[0051] Based on the above Figure 1 In accordance with the method for processing massive amounts of data described in the embodiments, the embodiments of the present invention further provide a computer-readable storage medium. For example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, or an optical data storage device. The storage medium stores at least one computer instruction for executing the above-mentioned Figure 1The method for processing massive amounts of data described in the corresponding embodiments will not be described in detail here.

[0052] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the disclosure disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0053] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for processing massive data, characterized in that: The method comprises: The vehicle uploads vehicle data to the MQTT Broker cluster, where the vehicle data includes but is not limited to the vehicle's unique identification, itinerary, and real-time driving data; The MQTT Broker cluster receives the vehicle data according to the MQTT topic corresponding to the vehicle data, and filters the vehicle data; The MQTT Broker cluster sends the filtered vehicle data to the Kafka message queue according to the corresponding topic mapping relationship; The Kafka message queue sends the vehicle data to the platform according to the Kafka topic subscribed by the platform.

2. The method for processing massive data according to claim 1, characterized in that: Before the MQTT Broker cluster receives the vehicle data according to the MQTT topic corresponding to the vehicle data, the method further includes: The MQTT Broker receives the user's bridge plug-in, which is a program with Kafka topic mapping and message filtering functions.

3. The method for processing massive data according to claim 1, characterized in that: The filtering of the vehicle data comprises the following steps: The MQTT Broker cluster obtains the MQTT message type corresponding to the vehicle data, and determines whether the message type is a preset message type to be intercepted; If so, the vehicle data is intercepted.

4. The method for processing massive data according to claim 3, characterized in that: The preset message types to be intercepted include MQTT hook functions and preset data streams.

5. The method for processing massive data according to claim 2, characterized in that: The MQTT Broker cluster sends the filtered vehicle data to the Kafka message queue according to the corresponding topic mapping relationship, including: The MQTT Broker identifies the vehicle unique identifier corresponding to the vehicle data, and obtains the partition identifier by taking the modulus of the partition number of the Kafka topic of the mapped Kafka message queue according to the hash value of the vehicle unique identifier; The topic mapping relationship of the vehicle data in the configuration file of the plug-in is read, and the vehicle data received by the topic specified by the MQTT Broker is forwarded to the Kafka message queue according to the partition identifier.

6. A mass data processing system, characterized in that: include: Vehicle side, MQTT Broker cluster, Kafka message queue, and platform side; The vehicle end is used to upload vehicle data to the MQTT Broker cluster, where the vehicle data includes but is not limited to the vehicle's unique identification, itinerary, and real-time driving data; The MQTT Broker cluster is used to receive the vehicle data according to the MQTT topic corresponding to the vehicle data, filter the vehicle data, and then send the filtered vehicle data to the Kafka message queue according to the corresponding topic mapping relationship; The Kafka message queue is used to send the vehicle data to the platform end according to the Kafka topic subscribed by the platform end; The platform end is used to send the subscribed topic content to the Kafka message queue and receive the vehicle data sent by the Kafka message queue.

7. The mass data processing system according to claim 6, characterized in that: The MQTT Broker is configured with a bridge plug-in, which is a program with Kafka topic mapping and message filtering functions.

8. The mass data processing system according to claim 6, characterized in that: The MQTT Broker cluster is specifically used to obtain the MQTT message type corresponding to the vehicle data, and determine whether the message type is a preset message type to be intercepted; when the determination result is yes, the vehicle data is intercepted.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the mass data processing method according to any one of claims 1 to 5 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the mass data processing method according to any one of claims 1 to 5 are implemented.

Images