Big data analysis system and vehicle

Abstract

The invention discloses a big data analysis system and a vehicle. The system comprises a data query management module, an update queue module and a calculation module, the data query management module is used for outputting a calculation model update instruction under the condition that a target calculation model is updated, and the calculation model update instruction comprises a label of the target calculation model; the updating queue module is used for receiving a calculation model updating instruction and sending the calculation model updating instruction to the calculation module; and the calculation module is used for deleting the original calculation script of the target calculation model based on the label of the target calculation model when receiving the calculation model updating instruction, receiving the updating script of the target calculation model, and calculating the vehicle end data based on the updating script of the target calculation model. According to the big data analysis system, the data query management module is connected with the update queue module to notify the calculation model in real time to update the script of the calculation model, so that the timeliness of updating the calculation model is ensured.

Description

Technical Field

[0001] This application relates to the technical field of vehicles, and in particular, to a big data analysis system and a vehicle. Background Art

[0002] In a real-time big data analysis system, complex state management is a common challenge. Especially in the field of intelligent driving, the system needs to process a large amount of real-time vehicle data. The real-time big data analysis system often does not support hot updates during execution, and updating the calculation method usually requires restarting the system.

[0003] In related technologies, the real-time big data analysis system usually needs to maintain the stability of the real-time calculation module, regularly save the state of the real-time calculation module according to the new changes of the calculation model, and update the calculation model by restarting the real-time system, but it cannot guarantee the timeliness of the calculation model update. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems in the related technologies to some extent. For this purpose, the first object of this application is to propose a big data analysis system, which includes: a data query management module, an update queue module, and a calculation module. Among them, the data query management module is used to output a calculation model update instruction when the target calculation model is updated, and the calculation model update instruction includes the label of the target calculation model; the update queue module is used to receive the calculation model update instruction and send the calculation model update instruction to the calculation module; the calculation module is used to, when receiving the calculation model update instruction, delete the original calculation script of the target calculation model based on the label of the target calculation model, receive the update script of the target calculation model, and calculate the vehicle data based on the update script of the target calculation model. The big data analysis system of this application updates the script of the calculation model by the way that the data query management module connects to the update queue module to notify the calculation model in real time, so as to realize the update of the business logic with low latency and ensure the timeliness of the calculation model update to a certain extent.

[0005] The second object of this application is to propose a vehicle.

[0006] To achieve the above object, an embodiment of the first aspect of the present application provides a big data analysis system, which includes: a data query management module, an update queue module, and a calculation module. Among them, the data query management module is used to output a calculation model update instruction when the target calculation model is updated. The calculation model update instruction includes the label of the target calculation model. The update queue module is used to receive the calculation model update instruction and send the calculation model update instruction to the calculation module. The calculation module is used to, when receiving the calculation model update instruction, delete the original calculation script of the target calculation model based on the label of the target calculation model, receive the update script of the target calculation model, and calculate the vehicle-end data based on the update script of the target calculation model.

[0007] According to an embodiment of the present application, the calculation module is used to send a call instruction to the data query management module. The data query management module is used to, when receiving the call instruction, send the update script of the target calculation model to the calculation module through the update queue module.

[0008] According to an embodiment of the present application, the system further includes: a data acquisition module, which is used to acquire vehicle-end data and transmit the vehicle-end data to the calculation module; a calculation result output module, which is used to receive the calculation result and transmit the calculation result to the database for storage.

[0009] According to an embodiment of the present application, the database includes one or more of Redis database and InfluxDb database.

[0010] According to an embodiment of the present application, the data acquisition module includes a Kafka component.

[0011] According to an embodiment of the present application, the calculation module includes a Flink component.

[0012] According to an embodiment of the present application, the update queue module includes a RabbitMq queue component.

[0013] According to an embodiment of the present application, the data query management module is developed based on netty.

[0014] According to an embodiment of the present application, the label of the target calculation model is a json string.

[0015] To achieve the above object, an embodiment of the second aspect of the present application provides a vehicle, which includes the aforementioned big data analysis system.

[0016] A big data analysis system and a vehicle according to an embodiment of the present application. The system includes: a data query management module, an update queue module, and a calculation module. Among them, the data query management module is used to output a calculation model update instruction when the target calculation model is updated, and the calculation model update instruction includes the label of the target calculation model; the update queue module is used to receive the calculation model update instruction and send the calculation model update instruction to the calculation module; the calculation module is used to, when receiving the calculation model update instruction, delete the original calculation script of the target calculation model based on the label of the target calculation model, receive the update script of the target calculation model, and calculate the vehicle-end data based on the update script of the target calculation model. The big data analysis system of the present application updates the script of the calculation model by the way that the data query management module connects to the update queue module to notify the calculation model in real time, so as to realize the update of the service logic with low latency and ensure the timeliness of the calculation model update to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 FIG. is a schematic structural diagram of a big data analysis system according to some embodiments of the present application;

[0018] Figure 2 FIG. is a schematic structural diagram of a big data analysis system according to some other embodiments of the present application;

[0019] Figure 3 FIG. is a schematic structural diagram of a big data analysis system according to some other embodiments of the present application;

[0020] Figure 4 FIG. is a flowchart of hot update according to some embodiments of the present application;

[0021] Figure 5 FIG. is a block diagram of a vehicle according to some embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application and should not be construed as limiting the present application.

[0023] The big data analysis system and the vehicle according to the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0024] Refer to Figure 1, the big data analysis system 1 includes: a data query management module 11, an update queue module 12, and a calculation module 13. Among them, the data query management module 11 is used to output a calculation model update instruction when the target calculation model is updated, and the calculation model update instruction includes the label of the target calculation model; the update queue module 12 is used to receive the calculation model update instruction and send the calculation model update instruction to the calculation module 13; the calculation module 13 is used to, when receiving the calculation model update instruction, delete the original calculation script of the target calculation model based on the label of the target calculation model, receive the update script of the target calculation model, and calculate the vehicle-end data based on the update script of the target calculation model.

[0025] Specifically, the data query management module 11 can provide a human-computer interaction interface, and the user can create or update the target calculation model on the human-computer interaction interface. When the target calculation model is updated, the data query management module 11 can output a calculation model update instruction, where the calculation model update instruction includes the label of the target calculation model, and send the calculation model update instruction to the update queue module 12. After receiving the calculation model update instruction, the update queue module 12 sends the calculation model update instruction to the calculation module 13 through the message queue, so that the calculation module 13 can be notified in real time when the calculation model is updated. After receiving the calculation model update instruction, the calculation module 13 parses the label of the target calculation model in the calculation model update instruction, deletes the original calculation script of the target calculation model corresponding to the label of the target calculation model, and receives the update script of the target calculation model. Among them, the update script of the target calculation model can be directly sent to the calculation module 13 by the data query management module 11, or can be sent to the calculation module 13 through the update queue module 12. After the calculation module 13 receives the vehicle-end data, it will calculate the vehicle-end data based on the update script of the target calculation model. In this way, the hot update of the intelligent driving real-time calculation module is realized.

[0026] In the big data analysis system of this application, when the calculation model is updated, the script of the calculation model is updated by the method of connecting the update queue module through the data query management module to notify the calculation model in real time, and there is no need to perform additional publishing, updating, restarting, etc. operations on the entire system, so as to achieve the effect of real-time hot update response; it can also realize low-latency update business logic, to a certain extent, ensuring the timeliness of the calculation model update.

[0027] In some embodiments, with reference to Figure 2 , the calculation module 13 is used to send a call instruction to the data query management module 11; the data query management module 11 is used to, when receiving the call instruction, send the update script of the target calculation model to the calculation module 13 through the update queue module 12.

[0028] Specifically, after receiving the calculation model update instruction, the calculation module 13 deletes the original calculation script of the target calculation model corresponding to the label of the target calculation model. When the next vehicle-end data arrives at the calculation module 13 and it is found that the calculation script of the target calculation model corresponding to the label of the target calculation model is missing, at this time, the calculation module 13 will send a call instruction to the data query and management module 11. When the data query and management module 11 receives the call instruction, it parses the label of the target calculation model in the call instruction, and sends the update script (such as a groovy script) of the target calculation model corresponding to the label of the target calculation model to the calculation module 13 through the update queue module 12. In this way, without restarting the real-time system, the update script of the calculation model is notified to the real-time calculation module 13.

[0029] In some embodiments, referring to Figure 3 , the system 1 further includes: a data acquisition module 14 for acquiring vehicle-end data and transmitting the vehicle-end data to the calculation module 13; a calculation result output module 15 for receiving the calculation result and transmitting the calculation result to the database 3 for storage.

[0030] Specifically, the data acquisition module 14 is used to acquire vehicle-end data from the vehicle end 2, such as acquiring CAN (Controller Area Network) data, perception fusion data, log data, etc. generated by the vehicle, and transmitting the vehicle-end data to the calculation module 13. The calculation result output module 15 is used to receive the calculation result output by the calculation module 13 and transmit the calculation result to the database 3 for storage. Moreover, the calculation result can also be obtained from the database 3 through the data query and management module 11 and presented to the user. The presentation method can be various forms of reports, charts, dashboards, etc., depending on the business requirements.

[0031] In some embodiments, the database 3 includes one or more of a Redis database and an InfluxDb database.

[0032] Specifically, the Redis database is mainly used for fast data query and update to meet the real-time requirements of the system; while the InfluxDb database is mainly used for storing and analyzing time series data to provide support for intelligent driving data analysis.

[0033] In some embodiments, the data acquisition module 14 includes a Kafka component.

[0034] Specifically, the Kafka component can efficiently process massive data, ensuring the real-time, reliable, and sequential nature of the data, and at the same time providing great convenience for the expansion and maintenance of the system.

[0035] In some embodiments, the calculation module 13 includes a Flink component.

[0036] Specifically, the Flink component can process streaming data with extremely low latency, making it very suitable for data that needs to be analyzed in real time in intelligent driving scenarios, such as vehicle sensor data, road condition information, etc.; the Flink component can process large-scale data streams, support the processing capacity of hundreds of thousands or even millions of messages per second, and can easily handle the real-time computing requirements of massive data in the intelligent driving system; the Flink component also supports dynamic update of the computing logic without restarting the entire computing task.

[0037] In some embodiments, the update queue module 12 includes a RabbitMq queue component.

[0038] Specifically, the RabbitMq queue component supports message persistence to ensure that messages are not lost after the server restarts or fails, and will send an acknowledgment signal after successfully processing the message to ensure that messages are not processed or lost repeatedly. It can efficiently and reliably transmit update instructions, support multiple message modes and flexible routing mechanisms. Its high availability, fault tolerance and easy extensibility enable it to adapt to complex business requirements and ensure the stable operation of the system in a dynamic environment.

[0039] In some embodiments, the data query management module 11 is developed based on netty.

[0040] Specifically, the data query management module 11 developed based on netty can efficiently process high-concurrency query requests, support multiple protocols and custom data formats, and provide high reliability and fault tolerance. The high performance, lightweight and flexible extension mechanism of netty can significantly improve the real-time performance and maintainability of the system.

[0041] In some embodiments, the label of the target calculation model is a json string.

[0042] Specifically, representing the label of the target calculation model as a json (JavaScript Object Notation) string is an efficient, flexible and easy-to-extend design method. It can make full use of the structured characteristics of json, facilitate data parsing, generation and transmission, and at the same time support seamless integration with modern big data and real-time computing frameworks. In the system, json-format labels can significantly improve the development efficiency and maintainability of the system.

[0043] As a specific example, refer to Figure 4, first, on the business side, users create or update a target calculation model through the human-machine interaction interface provided by the data query management module. In the case of updating the target calculation model, the data query management module will publish the API (Application Programming Interface) of the target calculation model and output a calculation model update instruction to the update queue module. After receiving the calculation model update instruction, the update queue module sends the calculation model update instruction to the calculation module, so that the calculation module can be notified in real time when the calculation model is updated. After receiving the calculation model update instruction, the calculation module parses the label of the target calculation model in the calculation model update instruction, deletes the original calculation script of the target calculation model corresponding to the label of the target calculation model from the cache. When the next vehicle-end data arrives at the calculation module and finds that the calculation script of the target calculation model corresponding to the label of the target calculation model is missing, at this time, the calculation module will send a call instruction to the data query management module. When receiving the call instruction, the data query management module parses the label of the target calculation model in the call instruction and publishes the update script of the target calculation model corresponding to the label of the target calculation model through the API. The calculation module calls the update script of the target calculation model to calculate the vehicle-end data and outputs the calculation result.

[0044] In summary, the big data update system of the present application can solve the problem that the calculation module needs to be restarted when creating or changing the calculation model in big data statistics, significantly improves the efficiency of creating and publishing by intelligent driving big data business personnel, and reduces the calculation delay caused by frequent restart of the real-time calculation module. And by connecting the update queue module through the data query management module to notify the calculation model in real time to update the script of the calculation model, so as to realize the update business logic with low latency, ensure the timeliness of the calculation model update to a certain extent, and improve the scalability and maintainability of the system.

[0045] Corresponding to the above embodiments, the present application also proposes a vehicle.

[0046] Referring to Figure 5 , the vehicle 10 includes the aforementioned big data analysis system 1.

[0047] It should be noted that the above explanations of the embodiments and beneficial effects of the big data analysis system are also applicable to the vehicle of the embodiments of the present application. To avoid redundancy, they will not be elaborated in detail here.

[0048] Note that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then storing it in a computer memory.

[0049] It should be understood that the various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0050] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0051] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0052] In this application, unless otherwise clearly specified and defined, terms such as "installed", "connected", "linked", "fixed", etc. shall be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal communication of two components or the interaction relationship between two components, unless otherwise clearly defined. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0053] Although the embodiments of this application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting this application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A big data analysis system, characterized in that, The system includes: a data query management module, an update queue module, and a calculation module, where the data query management module is configured to output a calculation model update instruction in the case of an update of the target calculation model, and the calculation model update instruction includes a label of the target calculation model; the update queue module is configured to receive the calculation model update instruction and send the calculation model update instruction to the calculation module; the calculation module is configured to, when receiving the calculation model update instruction, delete the original calculation script of the target calculation model based on the label of the target calculation model, receive the update script of the target calculation model, and calculate the vehicle-end data based on the update script of the target calculation model.

2. The big data analysis system according to claim 1, wherein the calculation module is configured to send a call instruction to the data query management module; the data query management module is configured to, when receiving the call instruction, send the update script of the target calculation model to the calculation module through the update queue module.

3. The big data analysis system according to claim 1, wherein The system further includes: a data acquisition module configured to acquire the vehicle-end data and transmit the vehicle-end data to the calculation module; a calculation result output module configured to receive the calculation result and transmit the calculation result to a database for storage.

4. The big data analysis system according to claim 3, wherein The database includes one or more of a Redis database and an InfluxDb database.

5. The big data analysis system according to claim 3, wherein The data acquisition module includes a Kafka component.

6. The big data analysis system according to claim 1, wherein The calculation module includes a Flink component.

7. The big data analysis system according to claim 1, wherein, The update queue module includes a RabbitMq queue component.

8. The big data analysis system according to claim 1, characterized in that, The data query management module is developed based on netty.

9. The big data analysis system according to claim 1, characterized in that, The label of the target calculation model is a json string.

10. A vehicle, characterized in that, It includes the big data analysis system according to any one of claims 1-9.

Images