Automatic driving data visualization method and system and vehicle

Abstract

The embodiment of the invention provides an automatic driving data visualization method and system and a vehicle, and the automatic driving data visualization system applied to the vehicle, the automatic driving data visualization system comprises a visualization module, a standardized interface connected with the visualization module, and at least two pieces of middleware, the middleware comprises a standardized interface and a plug-in configured for each middleware, communication protocols of different middleware are different, the middleware is accessed to the standardized interface through the plug-in, and the method comprises the following steps: performing data interaction with the plug-in corresponding to each middleware through the standardized interface to obtain first target data transmitted by the plug-in, the first target data is obtained by performing data format conversion on initial data transmitted by the middleware by the plug-in; and according to the first target data, performing data display on an interface of the visualization module. According to the method, the flexibility and expandability of an automatic driving data visualization system can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, and in particular to an automatic driving data visualization method and system and a vehicle. BACKGROUND

[0002] In the field of vehicle automatic driving technology, the visualization tool in the vehicle usually needs to interact with different types of data sources (such as various sensors on the vehicle) or middleware, and the communication protocols and data protocols of these data sources or middleware are usually different from each other.

[0003] At present, the visualization tool is usually bound to a fixed middleware, and when other middleware or data sources need to access the visualization tool, the communication protocols and data protocols of other middleware need to be bridged to the fixed middleware.

[0004] However, the existing binding of the fixed middleware results in poor expansion capability of the visualization tool. SUMMARY

[0005] The embodiments of the present application provide an automatic driving data visualization method, system and vehicle to improve the flexibility and scalability of the automatic driving data visualization system.

[0006] In a first aspect, the embodiments of the present application provide an automatic driving data visualization method applied to an automatic driving data visualization system in a vehicle, the automatic driving data visualization system including a visualization module, a standardized interface connected to the visualization module, at least two middleware, and a plug-in configured for each middleware, the communication protocols of different middleware being different, the middleware being connected to the standardized interface through the plug-in, and the method including:

[0007] interacting with the plug-in corresponding to each middleware through the standardized interface to obtain first target data transmitted by the plug-in, the first target data being obtained by performing data format conversion on initial data transmitted by the middleware by the plug-in;

[0008] According to the first target data, data is displayed on the interface of the visualization module.

[0009] In a second aspect, the embodiments of the present application provide an automatic driving data visualization system, including a visualization module, a standardized interface, and at least two plug-ins, the communication protocols of different plug-ins being different, and the middleware with the same communication protocol as the plug-in being connected to the standardized interface through the plug-in;

[0010] The plug-in is configured to convert the initial data transmitted by the middleware into target data in a target data format, and the target data format is a data format compatible with the visualization module.

[0011] The standardized interface is configured to obtain target data in the target data format and forward the target data to the data visualization module.

[0012] The data visualization module is configured to display the target data.

[0013] In a third aspect, an embodiment of the present application provides a vehicle comprising the automatic driving data visualization system described above, and the automatic driving data visualization system is configured to execute the method described above.

[0014] The automatic driving data visualization method, system and vehicle provided by the embodiments of the present application can make different middleware easily access the visualization system through the standardized interface, and can further realize automatic conversion of the middleware data format into a data format compatible with the visualization module through the plug-in, thereby decoupling the middleware communication protocol and the data protocol, and improving the expansibility and flexibility of the system. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0016] Figure 1 A flowchart of an automatic driving data visualization method provided by an embodiment of the present application is shown in the following figure.

[0017] Figure 2 A plug-in architecture diagram provided by the present application is shown in the following figure.

[0018] Figure 3 A schema file import flowchart provided by an embodiment of the present application is shown in the following figure.

[0019] Figure 4 A schema document parsing flowchart provided by an embodiment of the present application is shown in the following figure.

[0020] Figure 5 A schema document conversion flowchart provided by an embodiment of the present application is shown in the following figure.

[0021] Figure 6 A flowchart of an automatic driving data conversion provided by an embodiment of the present application is shown in the following figure.

[0022] Figure 7 A vehicle structure diagram provided by the present application is shown in the following figure.

[0023] The above-described figures have shown the specific embodiments of the present application, and the following will have a more detailed description. These figures and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0024] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following description of exemplary embodiments is not representative of all possible embodiments consistent with the present application. Instead, it is merely an example of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] In current automatic driving systems, the data visualization background service mostly adopts a traditional tight coupling architecture, resulting in poor scalability of the system and difficulty in adapting to new business requirements. Taking the Robot Operating System 2 (ROS2) as an example, there is a very important and commonly used data visualization tool RViz. Most of the data visualization tool RViz scenarios are in the ROS2 middleware environment. If other middleware is accessed, the communication protocol and data protocol of other middleware need to be bridged to ROS2. This solution results in a strong binding between the data visualization tool RViz and the ROS middleware, limiting the expansion capability of the data visualization tool RViz. At the same time, the data protocol interface is difficult to connect, and the performance consumption is also large, so that the data visualization tool RViz has general expansion capability, and the cost of connecting other middleware interfaces is high. Similarly, the data visualization tools on other system platforms also have similar solutions, and are strongly bound to the system platform. When accessing other middleware, a bridging node needs to be developed. In combination with the above example, in general, the main problems of current data visualization tools include: (1) strong binding between visualization tool and middleware, high coupling: the visualization tool is tightly bound to specific middleware, data sources, and other components. The high coupling between the visualization service and the middleware in this architecture design makes it difficult to quickly adapt to new requirements when expanding functions, and the expansion requires adjustment of the overall architecture, which has a large invasion on the technical architecture. (2) Strong dependence on data protocol: The data protocol schema document is generally managed by the middleware, and the visualization system relies on the specific schema document to parse data. When expanding new schema, the middleware needs to update it to the system first, and then the visualization software needs to parse the data, resulting in strong dependence of the visualization system development and debugging on the capabilities of the middleware, and insufficient openness and flexibility of the overall business. (3) Single data format: The data format transmitted by the current middleware is fixed and cannot support automatic conversion of data format. When the data format of the visualization software and the middleware is not unified, additional code functions need to be written by one party to complete the data format bridging and mapping, which easily leads to the need for the visualization software to constantly adapt to new data formats, and expansion and compatibility face great difficulties.

[0026] To solve the above problems, the application provides a scheme for realizing automatic driving data visualization in a vehicle automatic driving system, uses a plug-in architecture design, and through a standardized interface, different middleware and data sources can be easily accessed to the system. In addition, the middleware data format is automatically converted to the data format supported by the visualization tool through the plug-in, the middleware communication protocol and the data protocol are decoupled, thereby improving the expansibility and flexibility of the system.

[0027] First, the main application scenarios of the scheme are introduced as follows:

[0028] (1) Cross-domain data fusion: plug-and-play access of domain heterogeneous data.

[0029] Firstly, domain-specific plug-ins need to be developed, that is, independent plug-ins are developed for different domain controllers such as cockpit domain (including Human-Machine Interface (HMI) interaction data), chassis domain (including vehicle control signals), intelligent driving domain (including perception and decision data). Each plug-in connects the domain communication protocol (such as Controller Area Network (CAN) communication protocol, Ethernet communication protocol, Automotive Open System Architecture (AUTOSAR) communication protocol) through a standardized interface, converts the original data into a unified data format, and shields the communication protocol differences.

[0030] Among them, the plug-in is only responsible for data access and format conversion and does not participate in business logic processing. For example, the cockpit domain plug-in directly extracts the driver touch instruction, and the chassis domain plug-in analyzes the brake pedal signal. Both of them are transmitted to the visualization background through the corresponding standardized interface.

[0031] When the automatic driving system detects an abnormal signal in the chassis domain, it can automatically call the real-time perception data (such as camera pictures) accessed by the intelligent driving domain plug-in to synchronously display the vehicle control state and environmental perception results in the interface of the visualization module, thereby assisting in quickly locating the root cause of the fault.

[0032] (2) Multi-modal data access: standardized extension of multi-source sensors.

[0033] Firstly, independent plug-ins need to be developed for lidar, camera, millimeter wave radar and other sensors. The plug-in has a built-in data analysis template, for example:

[0034] Lidar plug-in: analyze point cloud data and output a standardized coordinate array.

[0035] Camera plug-in: extract image frames and convert them to a unified resolution format.

[0036] In the above application scenarios, after the plug-in development is completed, if a sensor needs to be replaced, only the corresponding plug-in needs to be replaced, without modifying the background code of the visualization tool. In the automotive production scenario, when upgrading the laser radar model of the vehicle, a new sensor plug-in can be developed to reuse the original visualization process, avoiding system reconstruction.

[0037] (3) Vehicle-road-cloud integration: lightweight integration of multi-end data sources.

[0038] The multi-end data source involves multi-end plug-ins: ① Vehicle-end plug-in: access to the vehicle bus (such as ROS / ROS2 topic), real-time collection of vehicle status and perception data. ② Roadside plug-in: interface with roadside units, receive traffic light status, pedestrian warning messages, and convert them into visual event labels. ③ Cloud plug-in: connect to the cloud platform Application Programming Interface (API) to obtain real-time traffic data and overlay it to the interface of the visualization module.

[0039] Among them, the plug-in can mark the data priority (such as vehicle-end data = high priority), and when the network bandwidth is insufficient, automatically degrade the display of low-priority data (such as cloud-end traffic) to ensure real-time visualization of critical information.

[0040] Through the cloud plug-in to access multi-vehicle location data, combined with the road event information of the roadside plug-in, the global traffic distribution and risk area can be displayed in real time on the visualization large screen.

[0041] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in detail in some examples. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0042] First, the automatic driving data visualization method of the present application is introduced in combination with the above application scenarios.

[0043] The vehicle can be equipped with an automatic driving system, and in order to improve the safety of the automatic driving system, an automatic driving data visualization system can be configured for the vehicle. The visualization system includes a visualization module, a standardized interface connected to the visualization module, at least two middleware, and a plug-in configured for each middleware. The communication protocols of different middleware are not the same, and the middleware is connected to the standardized interface through the plug-in (the automatic driving data visualization system will be described in detail below, and will not be described in detail here). In some embodiments, Figure 1 The automatic driving data visualization method flowchart provided by the embodiments of the present application is shown as follows: Figure 1 The method comprises the following steps:

[0044] S110, data interaction is performed with the plug-in corresponding to each middleware through a standardized interface, and first target data transmitted by the plug-in is acquired.

[0045] The first target data is obtained by performing data format conversion on initial data transmitted by the plug-in to the middleware.

[0046] S120, data display is performed on an interface of the visualization module according to the first target data.

[0047] In the embodiments of the present application, the standardized interface is configured, so that new functions, new middleware and new data sources (for example, a camera and a radar sensor on a vehicle) can be conveniently connected to the system, and the system can quickly expand new functions without affecting existing functions. Meanwhile, by configuring independent plug-ins for each middleware, the dependency between each module can be reduced, and each plug-in can complete data format conversion of its own middleware, which can solve the problem that the visualization tool needs to constantly adapt to new data formats, and expansion and compatibility face great difficulties, and improve the expansibility and compatibility of the entire system.

[0048] For the above step S110, the middleware can be ROS, Cyber rt, ICVOS, etc., and each middleware can be configured with a corresponding bridge plug-in, for example, ROS Bridge, Cyber rt Bridge and ICVOS Bridge.

[0049] Each plug-in corresponding to each middleware can be configured with a corresponding plug-in directory, and the plug-in directory can store related documents for data format conversion. When the data format of the initial data transmitted by the middleware changes, the plug-in can search for the corresponding data protocol from the plug-in directory to complete the data format conversion.

[0050] In the embodiments, the initial data can be various data collected by the vehicle in an automatic driving scene, for example, traffic signal state, driving parameters of surrounding vehicles, position distance of pedestrians, driving state of the vehicle, etc.

[0051] The initial data transmitted by different middleware is different from each other, and the data formats are also different. For example, some middleware transmits pictures, and some middleware transmits electrical signals. In some embodiments, the middleware can also include at least one of a cabin domain, a chassis domain and an intelligent driving domain of the vehicle.

[0052] The initial data of the cabin domain includes a touch instruction of a driver of the vehicle, the initial data of the chassis domain includes a brake pedal signal of the vehicle, and the initial data of the intelligent driving domain includes real-time perception data of the vehicle on the surrounding environment.

[0053] In the embodiments of the present application, the cabin domain, the chassis domain and the intelligent driving domain of the vehicle are taken as middleware, and the plug-ins configured for the middleware are accessed to the standardized interface, so that the data compatibility of different middleware is ensured. In addition, the independent plug-ins are separately configured for the cabin domain, the chassis domain and the intelligent driving domain, so that the independent deployment of the plug-ins and the background services is realized, the complexity of system upgrading and maintenance is reduced, and the quick response to the changes of business requirements is supported. Through the standardized interface, the unified access of the independent plug-ins is realized, the independent development of each part of the system is realized, the dependence between the modules is reduced, and the system stability is improved.

[0054] In addition, in the process of displaying the data transmitted by the cabin domain, the chassis domain or the intelligent driving domain of the vehicle on the interface of the visualization module, there may be a data abnormality scenario. For example, in the case of displaying the initial data transmitted by the chassis domain on the interface, a data abnormality signal may be received, and the data abnormality signal is used to represent the abnormality of the initial data transmitted by the chassis domain. At this time, the visualization module can obtain second target data from the plug-in corresponding to the intelligent driving domain in response to the received data abnormality signal, wherein the second target data is obtained by performing data format conversion on the real-time perception data by the plug-in corresponding to the intelligent driving domain, and then the real-time perception data is displayed on the interface.

[0055] In the embodiments of the present application, through the response of the visualization module to the data abnormality signal, the vehicle control state and the environmental perception result are synchronously displayed in the visualization interface, so that fault diagnosis can be realized, the driver can be quickly positioned to the root cause of the fault, and the driving safety can be improved.

[0056] In addition, in addition to the above-mentioned middleware such as the cabin domain, the chassis domain and the intelligent driving domain, there are also some data sources on the vehicle, for example, radar sensors and cameras, and the data collected by these data sources can also be accessed to the system to realize data visualization. Specifically, each data source can be configured with a corresponding data source plug-in, and the data source plug-in is used to realize the access of the data source to the standardized interface to realize data interaction.

[0057] The third target data is obtained by performing data analysis on the initial data transmitted by the data source, and then performing format conversion on the analyzed data. Then, the autonomous driving data visualization system can display the data on the interface according to the third target data.

[0058] In the embodiments, the data sources not only include various sensors (such as laser radars, cameras, etc.), but also include package recording plug-ins and simulation plug-ins, etc. These plug-ins perform data interaction with other parts of the system through the standardized interface, so as to ensure the data compatibility between different data sources.

[0059] In order to facilitate system debugging and testing, the packet recording plug-in and the simulation plug-in can be verified for functions and debugged for systems without relying on actual middleware plug-ins. Specifically, the packet recording plug-in can be used to record data streams in actual running and play back, simulating real data interaction environment. This helps developers to debug and verify without real data sources. The simulation plug-in can provide virtual middleware or services, simulate data sources or services in a real environment, and support developers to perform unit testing, function verification and system debugging without relying on a complete middleware environment.

[0060] In the embodiments of the present application, each plug-in is managed as an independent module to ensure that the system does not affect existing functions when it is expanded. When a data source fails or needs to be replaced, only the corresponding plug-in needs to be replaced, without the need to modify the code of the entire visualization system, without affecting the background service, and ensuring the high availability of the system. In addition, the conversion of data formats is realized independently by the data source plug-in, which can also improve the scalability and maintainability of the system.

[0061] Further, on the basis of the above embodiments, in some embodiments, the data source plug-in can be specifically divided into a radar plug-in and a camera plug-in. The radar plug-in is adapted to a radar sensor and is used to parse point cloud data collected by the radar sensor, convert the standardized coordinate array into a data format compatible with the visualization module, and thus realize the visualization display of the point cloud data. The camera plug-in is adapted to a camera and is used to parse image frames collected by the camera to a uniform resolution, improving the visual effect in subsequent visualization.

[0062] In the embodiments of the present application, the corresponding plug-in is configured for each data source transmitting different data formats to realize the conversion of data formats, solve the problem that when the data formats of the visualization module and the data source are not unified, one party needs to write additional code functions to complete the data format bridging mapping, affecting the compatibility of the system as a whole, and since each data source plug-in is independent, it is also convenient to maintain each data source.

[0063] In other embodiments, a vehicle-side plug-in, a roadside plug-in and a cloud-side plug-in can also be configured to access standardized interfaces. The vehicle-side plug-in is used to collect vehicle status and real-time perception data, the roadside plug-in is used to obtain first information between the vehicle and other road users and / or second information between the vehicle and traffic infrastructure, and the cloud-side plug-in is used to obtain real-time traffic data.

[0064] In the embodiment, the system can interact with the vehicle-side plug-in, the roadside plug-in and the cloud plug-in through the standardized interface, obtain the vehicle state and real-time perception data, the first information between the vehicle and other traffic participants and / or the second information between the vehicle and the traffic infrastructure, and real-time road condition data, and display the data on the interface of the visualization module, so that the vehicle-cloud-end integration is realized, and after the multi-end data integration, the safety during the vehicle driving process can be improved.

[0065] The vehicle-side plug-in can access the vehicle bus (for example, ROS / ROS2 topic), so as to collect the vehicle state and perception data in real time. The roadside plug-in can receive the traffic light state, pedestrian warning and other vehicle-to-everything (V2X) messages by communicating with the roadside unit (RSU), and convert the messages into visual event labels. The cloud plug-in can connect the cloud platform API to obtain real-time road condition data and superimpose the data on the interface of the visualization module.

[0066] In the embodiment, for some vehicle fleet composed of multiple vehicles, the vehicle fleet monitoring can be realized through the cloud plug-in and the roadside plug-in. Specifically, the multi-vehicle position data can be accessed through the cloud plug-in, and the road event information of the roadside plug-in can be combined to display the global traffic distribution and risk area in real time on the visualization large screen.

[0067] In the embodiment, by configuring the vehicle-side plug-in, the roadside plug-in and the cloud plug-in to access the standardized interface respectively, the multi-end data can be displayed on the interface, so that the driver can view more abundant information, and the safety during the vehicle driving process can be further improved.

[0068] Further, on the basis of the above embodiment, the priorities of the vehicle-side plug-in, the roadside plug-in and the cloud plug-in can be pre-configured, and in some scenarios, the data transmitted by the target plug-in with the highest priority can be preferentially displayed on the interface of the visualization module.

[0069] For example, the vehicle-side plug-in, the roadside plug-in and the cloud plug-in can mark the data priority (for example, the vehicle-side plug-in=high priority), when the network bandwidth is insufficient, the low-priority data (for example, the real-time road condition data of the cloud plug-in) is automatically degraded, only the vehicle state and real-time perception data of the vehicle-side plug-in are displayed, and the real-time visualization of the key information is ensured.

[0070] The automatic driving data visualization system of the present application is described below through some embodiments.

[0071] In some embodiments, the automatic driving data visualization system comprises a visualization module, a standardized interface, and at least two plugins. Different plugins are configured with different communication protocols, and each plugin can be connected to a middleware that is the same as the communication protocol of the plugin, and the middleware accesses the standardized interface through the plugin.

[0072] The plugin is configured to convert the initial data transmitted by the middleware into target data in a target data format that is compatible with the visualization module. The standardized interface is configured to obtain the target data in the target data format and forward it to the data visualization module. The data visualization module is configured to display the target data.

[0073] In this embodiment, the plugin and the background service (e.g., the data visualization service) can be independently deployed, allowing each plugin and background service module to be independently upgraded, expanded, or maintained. The update and replacement of the plugin will not affect the operation of the background service, ensuring the smooth operation of the system. In addition, new plugins can be added or existing plugins can be replaced according to actual needs, supporting flexible expansion of the system. This independent deployment of plugins and background services has higher maintainability, reduces the coupling between systems, and facilitates modular maintenance and troubleshooting.

[0074] Exemplary, Figure 2 The plugin architecture provided in this application is shown in the following figure: Figure 2 As shown, it includes a network communication layer, a business service layer, an interface abstraction layer, a plugin implementation layer, a middleware, and a hardware platform.

[0075] The network communication layer includes a network server, a connection management, a protocol analysis, and a data processing module, responsible for processing and transmitting network data. The business service layer includes a plugin management, a channel management, a service management, and a data management module, responsible for dynamic loading and subscription of plugins, publishing, and data protocol conversion, etc. to ensure stable execution and efficient operation of the system business process. The interface abstraction layer provides a standard and stable plugin interface, including event notification, topic subscription, topic publishing, and service calling, etc. The plugin implementation layer specifically implements the bridge plugin of different middleware (such as the ROS bridge plugin), ensuring the compatibility between each middleware.

[0076] In this embodiment, the plugin architecture allows different middleware to access the automatic driving data visualization system through the standardized interface, avoiding tight coupling between modules in the system. In addition, each plugin is independently developed, deployed, and interacts with the core business layer. The access of new middleware does not require modification of the entire system, but only needs to implement the plugin interface.

[0077] In some embodiments, the middleware can be a domain controller such as the cockpit domain, chassis domain, and intelligent driving domain mentioned above. Since the communication protocols of different middleware are not the same, a corresponding plug-in needs to be developed for each middleware, and the communication protocol configured for the plug-in is the same as the communication protocol of the middleware, so that the middleware can access the standardized interface through the plug-in to realize data interaction.

[0078] In addition, various data sources such as cameras and radar sensors can also be included on the vehicle. The radar sensor collects point cloud data, and the camera collects image data, and the data formats of the two are not the same. Therefore, a corresponding data source plug-in can also be configured, through which the initial data collected by the data source can be parsed, and then the parsed data is converted into a target data format and transmitted to the visualization module through the standardized interface. The target data format refers to a data format that the visualization module can be compatible with, so as to be read and recognized by the visualization module and displayed.

[0079] For example, the data source plug-in can refer to the laser radar plug-in or camera plug-in mentioned above. In addition, the data source not only includes various sensors directly connected (such as laser radar, camera, etc.), but also includes support for recording package files and simulation plug-ins. These plug-ins interact with other parts of the system through the standardized interface to ensure data compatibility between different middleware. In order to facilitate the debugging and testing of the system, the system is designed with a recording package plug-in and a simulation plug-in to perform function verification and system debugging without relying on the plug-in corresponding to the actual middleware.

[0080] The recording package plug-in is used to record the data stream in the actual running process and play it back, simulating the real data interaction environment. This helps developers to debug and verify without real data sources. The simulation plug-in is used to provide virtual middleware or services, simulating data sources or services in a real environment, supporting developers to perform unit testing, function verification, and system debugging without relying on a complete middleware environment.

[0081] In this embodiment, each plug-in is managed as an independent module, ensuring that the system does not affect existing functions when it is expanded.

[0082] In the embodiments of the present application, the plug-in architecture allows different data sources to access the automatic driving data visualization system through the standardized interface, avoiding tight coupling between modules in the system. In addition, each plug-in is developed and deployed independently and interacts with the core business layer. New data sources do not need to modify the entire system, but only need to implement the plug-in interface, which has stronger scalability.

[0083] Continuing to refer to the above Figure 2In the interface abstraction layer, at least one of the following operations can be performed: (1) processing of event asynchronous notification of the middleware, wherein the event asynchronous notification includes a topic list, a service list, etc., to provide real-time event updates for the system; (2) middleware topic subscription and destruction, and data return from the middleware to the subscriber to ensure that each subscriber receives data updates in a timely manner; (3) middleware topic publishing and destruction to ensure stable data publishing; and (4) middleware service creation and destruction to ensure accurate transmission of requests and responses.

[0084] Further, in some embodiments, each middleware (e.g., ROS middleware) has a corresponding bridge plug-in (e.g., ROS Bridge), and the middleware and the bridge plug-in with the same communication protocol are connected to each other, so that the middleware accesses the standardized interface through the bridge plug-in. In this embodiment, the bridge plug-in can be used to convert the initial data transmitted by the middleware into a data format compatible with the data receiver and transmit the data to the data receiver.

[0085] The data receiver includes at least one of the standardized interface accessed by the middleware, a data visualization module, and other middleware.

[0086] In this embodiment, the core function of the bridge plug-in is to establish an efficient communication bridge between different middleware or between the middleware and the visualization module, to solve the data compatibility problem, for example, the data format used by a certain type of middleware and the visualization module is different, and the bridge plug-in can be used to convert the data format, so as to realize data intercommunication and function cooperation.

[0087] In the embodiments of the present application, the system integration is simplified by the plug-in method, and the complex configuration and debugging work is reduced. Each middleware is configured with an independent bridge plug-in as an independent module for management, which can be updated and replaced flexibly without affecting the operation of other system parts.

[0088] Further, the automatic driving data visualization system can provide a schema document (including a data protocol) dynamic import function for users through a command line interface. Specifically, the plug-in configured for each middleware can have a plug-in directory, and the plug-in directory can be used to store the schema document imported from the external interface. Each plug-in can have an independent schema document. In some embodiments, the bridge plug-in can obtain the target data protocol from the plug-in directory, and convert the initial data transmitted by the middleware accessed to the standardized interface into the target data according to the target data protocol.

[0089] Each plug-in directory can include a plurality of different schema documents, i.e., a plurality of different data protocols, and the bridge plug-in selects the target data protocol based on actual needs to perform data conversion, thereby improving the adaptability and expandability of different middleware.

[0090] The schema document can be in a compressed package format when imported into the plug-in directory, and contains a directory, and all schema files should be stored under the directory without limitation on the number. After import, all schema files are first analyzed and checked to confirm whether the schema definition conforms to the data specification; if the check is passed, the schema is immediately persisted in the plug-in storage scope, and the maintenance information is updated, for example, Figure 3 The schema file import process provided by the embodiment of the application is shown in a schematic diagram as shown in Figure 3 The schema file import process provided by the embodiment of the application is shown in a schematic diagram as shown in

[0091] In some embodiments, the schema document dynamic import function can be performed by the protocol management module, which can be divided into three layers as follows,

[0092] The schema core layer is mainly responsible for dynamic analysis of the schema document, can automatically analyze the nested structure and cross-file reference relationship (for example, there is a complex introduction relationship of cross-file inclusion) in the schema document, and generate a data structure set to provide support for subsequent acquisition of the schema document corresponding to the data structure. Each data structure in the data structure set corresponds to at least one schema document;

[0093] The schema management layer is mainly responsible for schema document import and persistent storage, and completely encapsulates the capabilities required by the interface layer.

[0094] The schema interface layer mainly reads the corresponding schema document (including the target data protocol) by receiving the data structure name sent by the plug-in, and provides the ability of dynamic schema document acquisition for the plug-in.

[0095] In the above embodiment, the schema document is imported into the plug-in through an external interface, the data protocol is imported, the schema expansion process is more flexible, and the middleware is no longer strongly dependent on the data protocol update, so that the automatic driving data visualization system can flexibly adapt to the needs of multiple data protocols, and the adaptability is improved.

[0096] In addition, in some embodiments, the automatic driving data visualization system can also design a dynamic schema converter to improve the degree of freedom of different schema conversion. Specifically, the automatic driving data visualization system includes a file converter, and a first file conversion module and a second file conversion module registered to the file converter.

[0097] The first file conversion module is configured to parse a field name and a type defined in the first data protocol and a nested relationship, and to gather the intermediate description information; and the second file conversion module is configured to obtain a data specification of the second data protocol, and to convert the intermediate description information into the second data protocol according to the data specification.

[0098] In the embodiment, the first file conversion module and the second file conversion module are different schema conversion modules, and the different schema conversion modules need to implement functions in the specification interface, that is, the schema conversion modules can be loaded into the file converter (i.e., the schema converter) through a registration mechanism; the schema conversion module should have a schema parsing and schema conversion function.

[0099] Further, in some embodiments, the first file conversion module is taken as one of the schema conversion modules, and a schema document parsing process is taken as an example, which includes the following steps: (a) recursively processing the nested relationship until a basic field is parsed; (b) extracting at least one of a field name, a field type, a default value, an enumeration, and an annotation from the basic field; and (c) using a preset descriptor structure to describe at least one of the field name, the field type, the default value, the enumeration, and the annotation to obtain the intermediate description information.

[0100] In the embodiment, Figure 4 A schema document parsing process schematic diagram provided by the embodiment of the application is shown in FIG. 1. Figure 4 As shown in FIG. 1, the schema document parsing process is to completely analyze the original schema document definition content, explicitly determine the nested level relationship of the complex type, recursively process the nested relationship, and explicitly obtain all field names and types, and gather the information into the schema description information.

[0101] The schema description information specifically represents an intermediate state expression process of each field of the schema document, and the intermediate state expression method can be generated by other converters to carry the schema definition format.

[0102] In the embodiment, the preset descriptor structure can be a schema descriptor structure, and the schema descriptor is a key data structure, which contains a composite structure in the outermost layer of the schema document; and a descriptor iterator is provided to traverse all standard descriptors. The standard descriptor is divided into a basic member descriptor and a structure descriptor, which both inherit from a standard descriptor base class; the base class contains core attributes, including a name, a classification (a basic type / complex type), and a type.

[0103] In some embodiments, the schema document conversion process is performed by the second file conversion module as another schema conversion module, and the process includes the following steps: (a) traversing the fields in the preset descriptor structure, mapping the field attributes to the syntax of the target format according to the data specification; (2) dynamically creating context information based on the syntax of the target format; (c) retaining the file introduction relationship of the first data protocol in the context information, and obtaining the second data protocol.

[0104] In the embodiment, Figure 5 The schema document conversion process provided by the embodiment of the application is shown in FIG. 2. Figure 5 As shown in FIG. 2, the conversion process is to arrange the data structure nesting relationship, the contained fields and the type identifier corresponding to the schema descriptor, according to the schema specification of other registered modules, to arrange the corresponding schema document definition context, while retaining the multi-file introduction relationship defined in the original schema document. Thus, the dynamic conversion of the schema document is realized.

[0105] For the complete process of schema parsing and conversion, a unified schema parsing interface and a schema conversion interface need to be designed, to extend different schema conversion modules and support schema documents of different specifications.

[0106] In the schema parsing interface, the schema document is analyzed from top to bottom, and the nesting relationship is recursively processed until all the basic fields are parsed; all the field names, field types, default values, enumerations, annotations and the like are extracted and represented by using the schema descriptor structure; in the schema conversion interface, the schema descriptor structure is traversed, the context information of the schema document is dynamically created according to the field attributes, and all the composite structures and basic fields are saved into the schema document according to the structure.

[0107] In the above embodiment, by parsing and converting the schema document, the original schema document can be converted into a new schema document, the dynamic conversion between data protocols is realized, and the expandability and maintainability of the system are improved.

[0108] In addition, in some embodiments, the automatic driving data visualization system can also extend the dynamic conversion between different data by designing a dynamic data converter. Figure 6 The data conversion process provided by the embodiment of the application is shown in FIG. 3. Figure 6 As shown in FIG. 3, different data conversion modules can be loaded into the data converter by a registration mechanism.

[0109] The data analysis process is to traverse the schema description information to obtain each data structure, member, type and other attributes uniquely determining the field, and according to the attribute, the index positioning of the field can be completed through the dynamic analysis interface of the data object, so as to analyze the specific value; and the schema descriptor is bound to the value.

[0110] The data conversion process is to traverse the associated content of the previous process to obtain the attributes and values of the complete data member, and then through the dynamic structure creation interface of the data object, the addition of all data structures, fields and specific values is sequentially completed, and the support of composite structures such as data structure nesting, array and dictionary is provided, so as to realize the dynamic conversion of the data format. The data conversion process is strongly dependent on the content of the schema description information, and it is naturally compatible with the schema description information, and does not need to care about the original schema definition content to dynamically analyze the data.

[0111] In the embodiment, a unified data analysis and conversion interface can be designed to support different data conversion modules and different schema protocol data. The data analysis interface first deserializes the data to obtain the actual data object, traverses the attributes of all composite structures and basic members of the schema descriptor, reflects the field value in the data object according to the field name and type, and extracts the specific value to bind it to the pair with the standard descriptor. The data conversion interface traverses the schema descriptor pair structure, and according to the field attribute and value, the new data object reflection attribute is appended, and after all the values are appended, serialization is performed, so as to complete the dynamic conversion of the data type.

[0112] Figure 7 A vehicle structure diagram is provided for the present application, and the vehicle includes the automatic driving data visualization system described above. As shown in the figure, the vehicle 70 provided in the embodiment includes at least one processor 701 and a memory 702. Optionally, the vehicle 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected through a bus. Figure 7

[0113] In the specific implementation process, the at least one processor 701 executes the computer execution instructions stored in the memory 702, so that the at least one processor 701 executes the method described above.

[0114] The specific implementation process of the processor 701 can refer to the method embodiments described above, which has similar implementation principles and technical effects, and will not be described here in detail.

[0115] ​In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0116] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0117] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0118] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above method.

[0119] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the above method is implemented.

[0120] The above readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0121] An example readable storage medium is coupled to the processor such that the processor can read information from the readable storage medium and can write information to the readable storage medium. Of course, the readable storage medium can also be a part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0122] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0123] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0124] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0125] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0126] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various media capable of storing program codes, such as ROM, RAM, magnetic disk, or optical disk.

[0127] Finally, it should be noted that other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the present application disclosed herein. The present application is intended to include all such variations and modifications as fall within the scope of the present application, which is defined by the following claims, as well as the full scope of equivalents to which such claims are entitled. It is intended, therefore, that the present application be considered as including all possibilities falling within the scope of the application and their equivalents.

Claims

1. A method for visualizing autonomous driving data, characterized in that: An autonomous driving data visualization system for a vehicle includes a visualization module, a standardized interface connected to the visualization module, at least two middlewares, and a plug-in configured for each middleware, wherein different middlewares have different communication protocols, and the middlewares are connected to the standardized interface via the plug-ins. The method includes: Performing data interaction with the plug-in corresponding to each middleware through the standardized interface to obtain first target data transmitted by the plug-in, where the first target data is obtained by the plug-in through data format conversion of initial data transmitted by the middleware; Data is displayed on the interface of the visualization module according to the first target data.

2. The method according to claim 1, characterized in that The middleware includes at least one of the vehicle's cockpit domain, chassis domain and intelligent driving domain. The initial data of the cockpit domain includes vehicle driver touch commands, the initial data of the chassis domain includes vehicle brake pedal signals, and the initial data of the intelligent driving domain includes the vehicle's real-time perception data of the surrounding environment.

3. The method according to claim 2, characterized in that Displaying data on the interface of the visualization module according to the first target data includes: When the interface displays the initial data transmitted by the chassis domain, in response to a received data anomaly signal, second target data is obtained from the plug-in corresponding to the intelligent driving domain, wherein the data anomaly signal is used to indicate that the initial data transmitted by the chassis domain is abnormal, and the second target data is obtained by the plug-in corresponding to the intelligent driving domain performing data format conversion on the real-time perception data; The real-time perception data is displayed on the interface.

4. The method according to claim 1, wherein The standardized interface is connected to a data source plug-in corresponding to a data source, the data source including at least one sensor on a vehicle, and the method further includes: Performing data interaction with the data source plug-in corresponding to each data source through the standardized interface to obtain third target data transmitted by the data source plug-in, wherein the third target data is obtained by the data source plug-in parsing the initial data transmitted by the data source and further performing format conversion on the parsed data; Data is displayed on the interface according to the third target data.

5. The method according to claim 4, characterized in that The data source plug-in includes at least one of a radar plug-in and a camera plug-in. The radar plug-in is used to parse the point cloud data collected by the radar sensor to obtain a parsed standardized coordinate array. The camera plug-in is used to parse the image frames collected by the camera sensor to a uniform resolution to obtain parsed data.

6. The method according to claim 1, characterized in that The standardized interface is connected to a vehicle-side plug-in, a roadside plug-in, and a cloud plug-in. The vehicle-side plug-in is used to collect vehicle status and real-time perception data. The roadside plug-in is used to obtain first information between the vehicle and other traffic participants and / or second information between the vehicle and traffic infrastructure. The cloud plug-in is used to obtain real-time road condition data. The method further includes: Interacting with the vehicle-side plug-in, the roadside plug-in, and the cloud-side plug-in through the standardized interface to obtain data transmitted by the vehicle-side plug-in, the roadside plug-in, and the cloud-side plug-in; At least one of the data transmitted by the vehicle-side plug-in, the roadside plug-in and the cloud-side plug-in is displayed on the interface.

7. The method according to claim 6, characterized in that The displaying, on the interface, at least one of the data transmitted by the vehicle-side plug-in, the roadside plug-in, and the cloud plug-in, includes: Obtaining the preset priorities of the vehicle-side plug-in, roadside plug-in, and cloud-side plug-in; Determine a target plug-in from the vehicle-side plug-in, the roadside plug-in, and the cloud plug-in, wherein the target plug-in has the highest priority; The data transmitted by the target plug-in is displayed on the interface.

8. An autonomous driving data visualization system, characterized in that: include: A visualization module, a standardized interface, and at least two plug-ins, wherein different plug-ins are configured with different communication protocols, and middleware with the same communication protocol as the plug-ins is connected to the standardized interface through the plug-ins; The plug-in is used to convert the initial data transmitted by the middleware into target data in a target data format, wherein the target data format is a data format compatible with the visualization module; The standardized interface is used to obtain target data in the target data format and forward it to the data visualization module; The data visualization module is used to display the target data.

9. The system according to claim 8, characterized in that The plug-in includes a bridge plug-in, and the bridge plug-in is connected to the middleware with the same communication protocol; The bridge plug-in is used to convert the initial data transmitted by the middleware into a data format compatible with a data receiver, and transmit the data to the data receiver, wherein the data receiver includes at least one of the standardized interface, the data visualization module and other middleware; The bridge plug-in is configured with a corresponding plug-in directory and a protocol management module, and the plug-in directory stores data protocols imported from an external interface; The bridge plug-in is specifically configured to obtain a target data protocol from the plug-in directory and convert the initial data transmitted by the middleware connected to the standardized interface into the target data according to the target data protocol; The protocol management module is used to perform the following operations: Automatically parsing nested structures and cross-file reference relationships in the data protocol imported from the external interface to generate a data structure set, each data structure in the data structure set corresponding to at least one data protocol; In response to receiving the data structure name sent by the bridge plug-in, querying the target data protocol corresponding to the data structure name and returning it to the bridge plug-in.

10. A vehicle, characterized in that: The autonomous driving data visualization system comprises the above-mentioned claim 8 or 9, and the above-mentioned autonomous driving data visualization system is used to execute the method described in any one of the above-mentioned claims 1-7.

Images