Heterogeneous data fusion method and device based on cab integrated vehicle and vehicle

Abstract

The invention discloses a heterogeneous data fusion method and device based on a cab integrated vehicle, and the vehicle. The method comprises the steps of collecting a multi-source heterogeneous data set inside and outside the vehicle in real time through a plurality of cameras, laser radars and millimeter wave radars which are uniformly wired in the vehicle; and inputting the multi-source heterogeneous data set into a multi-modal fusion network to perform fusion processing of the multi-source heterogeneous data, obtaining vehicle environment sensing data for describing the real-time environment inside and outside the vehicle, and performing unified decision control on an automatic driving system and a cabin system. According to the technical scheme of the embodiment of the invention, on the premise that the internal wiring structure of the vehicle is simplified and the hardware cost is reduced, the multi-source heterogeneous data can be quickly fused to obtain the vehicle environment sensing data in various complex scenes through an efficient multi-source heterogeneous fusion algorithm; unified decision control is carried out on the automatic driving system and the cabin system accurately and efficiently, and the collaboration between the automatic driving system and the cabin system is improved.

Description

Technical Field

[0001] The present invention relates to the field of driverless technology, and in particular, to a heterogeneous data fusion method, device, and vehicle for a cockpit-integrated vehicle. Background Art

[0002] A cockpit-driving integrated vehicle refers to an intelligent vehicle in which the cockpit system and the driving assistance or autonomous driving system of the vehicle are deeply integrated. The cockpit system and the driving system of traditional vehicles usually have their own independent hardware. The vehicle will use a set of cameras, lidars, and millimeter-wave radars to provide environmental perception data for autonomous driving, and use another set of cameras, lidars, and millimeter-wave radars to serve functions such as intelligent display in the cockpit.

[0003] The inventors found in the process of implementing the present invention that the existing implementation architecture of cockpit-driving integrated vehicles not only increases the hardware cost, but also increases the complexity of the vehicle's internal wiring. In addition, the software such as the in-cockpit information entertainment system and intelligent interaction system lacks coordination with the autonomous driving software. For example, when the sensors of the autonomous driving system detect that the road conditions ahead are complex and manual takeover is required, relevant information cannot be displayed on the cockpit display in a timely manner and a reminder cannot be issued, and at the same time, the cockpit environment such as the seat and lights cannot be adjusted to prepare for the driver's takeover. Summary of the Invention

[0004] Embodiments of the present invention provide a heterogeneous data fusion method, device, and vehicle for a cockpit-integrated vehicle, which realize unified decision control of the autonomous driving system and the cockpit system by performing unified wiring of multi-source heterogeneous sensors on the cockpit-integrated autonomous driving vehicle.

[0005] According to one aspect of the embodiments of the present invention, a heterogeneous data fusion method for a cockpit-integrated vehicle is provided, which is executed by a cockpit-integrated autonomous driving vehicle that has completed unified wiring of multi-source heterogeneous sensors. The method includes:

[0006] Real-time collection of a multi-source heterogeneous data set inside and outside the vehicle through multiple cameras, multiple lidars, and multiple millimeter-wave radars that are uniformly wired in the vehicle; wherein, the multi-source heterogeneous data set includes multiple groups of image acquisition data, multiple groups of lidar data, and multiple groups of millimeter-wave data;

[0007] Inputting the multi-source heterogeneous data set into a pre-trained multi-modal fusion network for fusion processing of the multi-source heterogeneous data to obtain vehicle environment perception data for describing the real-time environment inside and outside the vehicle;

[0008] wherein, the multi-modal fusion network is pre-trained based on a knowledge graph constructed from multiple historical acquisition data obtained by cameras, lidars, and millimeter-wave radars on the vehicle;

[0009] Based on the vehicle environment perception data, unified decision-making control is performed on the autonomous driving system and the cockpit system.

[0010] According to another aspect of the embodiments of the present invention, there is also provided a heterogeneous data fusion device for a cockpit integrated vehicle, which is configured in a cockpit integrated autonomous driving vehicle that has completed unified wiring of multi-source heterogeneous sensors. The device includes:

[0011] A multi-source heterogeneous data set acquisition module, which is used to collect multi-source heterogeneous data sets inside and outside the vehicle in real time through multiple cameras, multiple lidars, and multiple millimeter wave radars that are uniformly wired in the vehicle; among them, the multi-source heterogeneous data set includes multiple groups of image acquisition data, multiple groups of lidar data, and multiple groups of millimeter wave data;

[0012] A vehicle environment perception data acquisition module, which is used to input the multi-source heterogeneous data set into a pre-trained multi-modal fusion network for fusion processing of the multi-source heterogeneous data, and obtain vehicle environment perception data for describing the real-time environment inside and outside the vehicle;

[0013] Among them, the multi-modal fusion network is pre-trained based on a knowledge graph constructed from multiple historical acquisition data obtained by cameras, lidars, and millimeter wave radars on the vehicle;

[0014] A unified decision-making module, which is used to perform unified decision-making control on the autonomous driving system and the cockpit system according to the vehicle environment perception data.

[0015] According to another aspect of the embodiments of the present invention, there is also provided a cockpit integrated autonomous driving vehicle, and the vehicle includes:

[0016] Multiple cameras, multiple lidars, and multiple millimeter wave radars that have completed unified wiring, which are used to collect multi-source heterogeneous data sets inside and outside the vehicle in real time;

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; among them,

[0019] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor can execute the heterogeneous data fusion method for a cockpit integrated vehicle according to any embodiment of the present invention.

[0020] According to another aspect of the embodiments of the present invention, there is provided a computer-readable storage medium, and the computer-readable storage medium stores computer instructions, and when the computer instructions are used by a processor to execute, the heterogeneous data fusion method for a cockpit integrated vehicle according to any embodiment of the present invention is realized.

[0021] According to another aspect of the embodiments of the present invention, there is also provided a computer program product, including a computer program, which when executed by a processor, implements the steps of the heterogeneous data fusion method for a cockpit-integrated vehicle as described in any embodiment of the present invention.

[0022] The technical solution of the embodiments of the present invention is as follows: First, in a cockpit-integrated autonomous vehicle, unified wiring of multi-source heterogeneous sensors is completed. Then, a multi-source heterogeneous data set inside and outside the vehicle is collected in real time through multiple cameras, multiple lidars, and multiple millimeter-wave radars with unified wiring in the vehicle. The multi-source heterogeneous data set is input into a multi-modal fusion network pre-trained via a knowledge graph for fusion processing of multi-source heterogeneous data, and vehicle environment perception data for describing the real-time environment inside and outside the vehicle is obtained. According to the vehicle environment perception data, a technical means of unified decision control for the autonomous driving system and the cockpit system is used to realize that the autonomous driving system and the cockpit system share some hardware resources. On the premise of simplifying the internal wiring structure of the vehicle and reducing the hardware cost, through an efficient multi-source heterogeneous fusion algorithm, in various complex scenarios, multi-source heterogeneous data can be quickly fused to obtain vehicle environment perception data, so as to accurately and efficiently perform unified decision control on the autonomous driving system and the cockpit system, effectively improving the coordination between the autonomous driving system and the cockpit system, and further effectively ensuring the safety of the people in the vehicle and the user experience.

[0023] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0025] Figure 1 is a flowchart of a heterogeneous data fusion method for a cockpit-integrated vehicle provided according to an embodiment of the present invention;

[0026] Figure 2 is a flowchart of a method for training a multi-modal fusion network applicable to an embodiment of the present invention;

[0027] Figure 3 is a flowchart of another heterogeneous data fusion method for a cockpit-integrated vehicle provided by an embodiment of the present invention;

[0028] Figure 4 It is a schematic structural diagram of a heterogeneous data fusion device for a cockpit-integrated vehicle provided by an embodiment of the present invention;

[0029] Figure 5 It is a schematic structural diagram of a cockpit-integrated autonomous vehicle for implementing the heterogeneous data fusion method for a cockpit-integrated vehicle according to an embodiment of the present invention. Detailed implementation manners

[0030] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0032] In various embodiments of the present invention, in order to achieve unified decision-making control over the autonomous driving system and the cockpit system, the inventor first realized unified wiring of multi-source heterogeneous sensors (that is, cameras, lidars, and millimeter-wave radars) on the cockpit-integrated autonomous vehicle in the hardware architecture to simplify the internal wiring structure of the vehicle and save some hardware sensor resources.

[0033] In an optional implementation manner of this embodiment, a unified wiring method that has been experimentally confirmed to be feasible is specifically given.

[0034] Specifically, when carrying out the integrated wiring of multi-source heterogeneous sensors, the sensor data collected by cameras, lidars, and millimeter-wave radars can be aggregated to the central computing platform of the autonomous vehicle through an Ethernet switch. A centralized topology networking method can be adopted to reduce the harness length. In a specific example, this centralized topology can support data transmission greater than 10 Gbps. Further, a regionalized architecture can also be adopted. Independent ECUs (Electronic Control Unit) can be configured in the front, left, right, and rear regions of the autonomous vehicle, and the data collected by each multi-source sensor set nearby can be preprocessed through the above four ECUs. In addition, the door camera can use an image sensor that supports short-range wireless communication. Furthermore, wireless transmission can be used to replace physical harnesses, so as to reduce interference in signal transmission, improve system integration, simplify wiring, and enhance reliability. In addition, regarding wiring design, the data collected by lidars, millimeter-wave radars, and cameras are all transmitted using optical fibers to achieve high-speed transmission of the collected signals.

[0035] Further, the main autonomous driving camera can adopt dual-power supply to achieve power redundancy and reduce the harness length through a domain centralized architecture (such as Zone ECU) for integration. Further, the front-view camera can be routed along the inner side of the vehicle's A-pillar and connected to the intelligent driving domain controller. The side-view camera can enter the vehicle body through the silicone sheath at the door hinge; the harnesses of the front and rear surround cameras can be hidden and routed along the front longitudinal beam and the rear longitudinal beam respectively to access the surround ECU. An occlusion switch can be set on the in-vehicle DMS (Driver Monitoring System) camera, and the wire can be routed through the dashboard to the cockpit domain controller; the roof camera can be routed along the edge of the roof and fixed to the roof crossbeam, thus simplifying the wiring structure. Further, the wire harness interface of the external camera can adopt an IP6K9K-class waterproof and dustproof design. Each camera covers a 360° field of view, and an automatic water spraying and air sweeping device self-cleaning system is used to reduce pollution and reduce the deviation of the radial distortion coefficients k1 (±0.05) and k2 (±0.005) of the camera.

[0036] In addition, a small solid-state lidar can be integrated into the lamp group or body gap. The horizontal field of view (abbreviated as HFOV) is preferably set to 120°, the ROI (Region of Interest) of the horizontal field of view is preferably set to 30°, and the vertical field of view (VFOV) is controlled between 25° and 40°. Optionally, a multi-lidar redundancy setting can be adopted on the autonomous vehicle to achieve omnidirectional coverage. Further, on the lidar sensor, by arranging the power cord of the cooling fan, harness thermal management can be achieved to support high-power DC power supply. Using AOC (Active Optical Cable), the power supply and data are integrated into a single optical fiber to achieve the transmission of millions of point clouds per second. The signals are converged to the domain controller of the central computing platform, and a redundant design of the dual Ethernet ring network architecture ensures that a single point of failure does not affect the overall system.

[0037] Meanwhile, the 4D imaging millimeter-wave radar can use TI's DCA1000 data capture adapter to obtain forward radar data, and the front and rear corner radars are powered by wireless charging technology. The beam center plane of the forward millimeter-wave radar is parallel to the road surface, and the angle deviation is less than 5°; the front-side millimeter-wave radar forms a 45° angle with the vehicle driving direction; the rear-side millimeter-wave radar forms a 30° angle with the vehicle driving direction, and the angle deviation does not exceed 5°. The field of view angle of the front camera is 120°, and the field of view angle of the side camera is 90°. Such settings lay a foundation for accurately fusing radar data and reducing deviations.

[0038] Figure 1 The figure is a flowchart of a heterogeneous data fusion method for a cockpit-integrated vehicle provided by an embodiment of the present invention. This embodiment is applicable to the situation where a cockpit-integrated autonomous vehicle performs unified decision control on the autonomous driving system and the cockpit system according to a multi-source heterogeneous data set collected in real time by multi-source heterogeneous sensors with unified wiring in the vehicle. This method can be executed by a heterogeneous data fusion device based on a cockpit-integrated vehicle. The device can be implemented in the form of hardware and / or software and is generally configured in a cockpit-integrated autonomous vehicle. Typically, it is configured in the central computing platform of the vehicle.

[0039] Correspondingly, as Figure 1 shown, the method includes:

[0040] S110. Real-time collect a multi-source heterogeneous data set inside and outside the vehicle through multiple cameras, multiple lidars, and multiple millimeter-wave radars with unified wiring in the vehicle.

[0041] Among them, the multi-source heterogeneous data set includes multiple groups of image acquisition data, multiple groups of lidar data, and multiple groups of millimeter-wave data.

[0042] As shown above, for the integrated cockpit autonomous vehicle according to the embodiments of the present invention, unified wiring of various heterogeneous sensors is pre-completed. Multiple cameras, multiple lidars, and multiple millimeter-wave radars configured at different positions inside and outside the vehicle will transmit and converge in real time multiple sets of image acquisition data, multiple sets of lidar data, and multiple sets of millimeter-wave data collected to the central computing platform of the vehicle at high speed.

[0043] S120. Input the multi-source heterogeneous data set into a pre-trained multi-modal fusion network for fusion processing of multi-source heterogeneous data, and obtain vehicle environment perception data for describing the real-time environment inside and outside the vehicle.

[0044] Among them, the multi-modal fusion network is pre-trained in advance based on a knowledge graph constructed from a number of historical acquisition data obtained by at least one camera, lidar, and millimeter-wave radar on the vehicle.

[0045] Specifically, every preset time period, such as 1 s, 2 s, or 5 s, etc., the data collected by the multiple cameras, multiple lidars, and multiple millimeter-wave radars on the vehicle within the preset time period can be summarized to form the multi-source heterogeneous data set.

[0046] Generally speaking, the image acquisition data collected by cameras are mostly unstructured data, the lidar data collected by lidars are mostly semi-structured data, and the millimeter-wave data collected by millimeter-wave radars are mostly structured or unstructured data. To facilitate subsequent fusion processing of multi-source heterogeneous data, the lidar data and millimeter-wave data can first be converted into structured data according to preset structured data processing rules. Further, to ensure the comparability between various heterogeneous data, a preset spatio-temporal alignment algorithm can first be used to align the heterogeneous data in the multi-source heterogeneous data set in terms of time and space. Further, unified data noise filtering processing can also be performed on the above multi-source heterogeneous data set, such as removing outliers or abnormal points, etc. This embodiment does not limit this.

[0047] Finally, the multi-source heterogeneous data set after completing the above preprocessing operations can be input into a pre-trained multi-modal fusion network for fusion processing of multi-source heterogeneous data, and the vehicle environment perception data output by the multi-modal fusion network can be correspondingly obtained.

[0048] Among them, the vehicle environment perception data refers to structured knowledge data for describing the real-time environment inside and outside the vehicle. For example: Driver inside the vehicle: Adult female, sitting height XX; Passenger in the co-pilot seat inside the vehicle: Child, sitting height YY, watching video; Brightness inside the vehicle: ZZ; Temperature inside the vehicle: aa; Pedestrian 1 outside the vehicle: bb meters in front of the right side of the vehicle, passing through the crosswalk; Vehicle 2 outside the vehicle: cc meters on the left side of the vehicle, accelerating to overtake; Traffic light in front of the vehicle: Will turn red in 5 seconds, etc.

[0049] In this embodiment, the multi-modal fusion network is pre-trained based on a knowledge graph, and the knowledge graph is constructed based on a plurality of historical acquisition data obtained by cameras, lidars, and millimeter-wave radars on at least one vehicle. Among them, a set of historical acquisition data that is spatio-temporally aligned at the same time contains data collected by multiple cameras, multiple lidars, and multiple millimeter-wave radars on the same vehicle in the same time interval. Furthermore, for each set of historical acquisition data, a knowledge graph including entities, entity relationships, and attributes can be constructed respectively.

[0050] Among them, an entity refers to an entity object existing in the actual driving environment of the vehicle. For example, a driver, a passenger, the vehicle itself, other road vehicles, pedestrians, roads, and obstacles, etc. Entity relationships can include spatial relationships, such as the vehicle itself is in the fast lane, the pedestrian is in the crosswalk, the passenger is in the co-pilot position, etc., temporal relationships, such as the vehicle itself is approaching a pedestrian, the traffic light in front of the vehicle is about to turn red, the driver is adjusting the seat backrest, etc., and behavioral relationships, such as the vehicle on the left is overtaking, the pedestrian is crossing the road, etc. Attributes can include static attributes and dynamic attributes. Static attributes can include: the color of the vehicle, the height of the pedestrian, the type of the obstacle, the gender and age of the driver or passenger, etc. Dynamic attributes can include: the speed of the vehicle, the moving direction of the pedestrian, or the current state of the traffic light, etc.

[0051] In this embodiment, a source identifier of the graph content can be further added to each constructed knowledge graph. For example, the "proximity relationship" between the "vehicle itself" and "pedestrian 1" is determined by the data collected by the lidar. Furthermore, in this piece of knowledge content, the source identifier "lidar" is added. Correspondingly, after constructing multiple knowledge graphs from a plurality of historical acquisition data, based on the multiple knowledge graphs with the source identifier added in advance, a fusion model based on deep learning, such as a multi-sensor feature-level fusion model or a cross-modal Transformer model, etc., can be trained to obtain the multi-modal fusion network.

[0052] In this embodiment, the multi-modal fusion network can first infer a target knowledge graph adapted to the input multi-source heterogeneous data set for the input multi-source heterogeneous data set, and then knowledge extraction can be performed on the target knowledge graph to obtain the vehicle environment perception data.

[0053] S130. According to the vehicle environment perception data, unified decision-making control is performed on the autonomous driving system and the cockpit system.

[0054] In this embodiment, since the vehicle environment perception data that can describe the real-time environment inside and outside the vehicle simultaneously has been obtained at one time, and then, based on the actual situations inside and outside the vehicle, unified decision-making control is performed on the autonomous driving system and the cockpit system, that is, the interlocking control of the autonomous driving system and the cockpit system can be achieved.

[0055] In a specific example, when it is determined according to the vehicle environment perception data that the integrated cockpit autonomous driving vehicle is in a very complex road condition environment in the autonomous driving mode, at this time, it is necessary to switch the autonomous driving mode to the manual driving mode. When switching to the manual driving mode, the seat setting of the driver can be synchronously adjusted to the setting item used by the driver for manual driving, and after turning off the entertainment content on the driver operation screen, the prompt information about the complex road condition environment is displayed in real time to assist the driver in making driving decisions.

[0056] Correspondingly, in an optional implementation manner of this embodiment, the unified decision-making control of the autonomous driving system and the cockpit system according to the vehicle environment perception data may include:

[0057] According to the vehicle environment perception data, the driving decision-making control of the autonomous driving system, the adjustment decision-making control of at least one adjustable facility in the cockpit, and the display decision-making control of the real-time display content on the cockpit display screen are uniformly executed.

[0058] Among them, the driving decision-making control of the autonomous driving system may include: switching of the autonomous driving trajectory, for example, switching from the current lane to an adjacent lane to overtake, switching of the current driving mode, for example, switching from the autonomous driving mode to the assisted driving mode, and adjustment of driving parameters. For example, decelerating or accelerating the vehicle, etc.

[0059] Furthermore, the adjustment decision-making control of at least one adjustable facility in the cockpit may include: setting the seat posture of the driver's seat, setting the seat postures of the co-driver's seat or the rear seats, setting the brightness of at least one ambient light or at least one reading light in the vehicle, and setting the temperature of at least one air conditioner outlet in the vehicle, etc.

[0060] In addition, the display decision-making control of the real-time display content on the cockpit display screen may include: switching the current display content on the driver's display screen or the co-driver's display screen, and displaying the abnormal information that appears in real time in the driving environment outside the vehicle (for example, a vehicle that suddenly overtakes from behind, a vehicle that suddenly decelerates in front) on the driver's display screen in real time, etc.

[0061] Based on the above embodiments, when making a decision on the autonomous driving trajectory of the autonomous driving system according to the vehicle environment perception data, there may be a situation where multiple target trajectories are decided. At this time, the above multiple target trajectories can be fused through the Kalman filtering algorithm or the Bayesian probability network model. Further, the D-S evidence theory can also be applied to process the uncertainty and conflict information in the fusion process, and finally decide on an autonomous driving trajectory with the highest reliability.

[0062] Further, in another optional implementation manner of this embodiment, after obtaining the vehicle environment perception data for describing the real-time environment inside and outside the vehicle, it may further include:

[0063] Intelligently answer the real-time questions of the people in the cockpit according to the vehicle environment perception data.

[0064] As described above, after the multi-modal fusion network performs fusion processing on each multi-source heterogeneous data in the input multi-source heterogeneous data set, it can first obtain a target knowledge graph corresponding to the multi-source heterogeneous data set. While outputting the vehicle environment perception data, the multi-modal fusion network can also synchronously output the target knowledge graph. Based on this target knowledge graph, the cockpit integrated autonomous driving vehicle can handle various complex problems of the people in the cockpit to provide more targeted or more accurate answers to the people in the cockpit.

[0065] In a specific example, if the driver asks the cockpit system: "Can I accelerate through the current section?", the cockpit system can combine the target knowledge graph determined in the most recent preset time period to obtain various environmental information inside and outside the vehicle. And based on the above various environmental information, conduct an intelligent Q&A with the driver. For example, the cockpit system can provide an intelligent answer such as: "There is a vehicle accelerating on your left, a pedestrian is about to enter the crosswalk in front, and at the same time, the user in the back seat is sleeping. Therefore, it is not recommended to accelerate now." to maximize the riding experience and safety of the people in the vehicle.

[0066] The technical solution of the embodiment of the present invention first completes the unified wiring of multi-source heterogeneous sensors in the integrated cockpit autonomous vehicle, and then collects the multi-source heterogeneous data sets inside and outside the vehicle in real time through multiple cameras, multiple lidars, and multiple millimeter-wave radars that are uniformly wired in the vehicle; inputs the multi-source heterogeneous data sets into the multi-modal fusion network pre-trained via a knowledge graph for fusion processing of multi-source heterogeneous data, obtains vehicle environment perception data for describing the real-time environment inside and outside the vehicle, and based on the vehicle environment perception data, uses a technical means of unified decision-making and control for the autonomous driving system and the cockpit system, realizing that the autonomous driving system and the cockpit system share some hardware resources. On the premise of simplifying the internal wiring structure of the vehicle and reducing the hardware cost, through an efficient multi-source heterogeneous fusion algorithm, multi-source heterogeneous data can be quickly fused into vehicle environment perception data in various complex scenarios, so as to accurately and efficiently perform unified decision-making and control on the autonomous driving system and the cockpit system, effectively improving the coordination between the autonomous driving system and the cockpit system, and further effectively ensuring the safety of the people in the vehicle and the user experience.

[0067] Further, in Figure 2 shows a flowchart of a method for training a multi-modal fusion network applicable to the embodiment. Correspondingly, as Figure 2 shown, the method for training the multi-modal fusion network may specifically include:

[0068] S210. Collect a number of historical acquisition data obtained by cameras, lidars, and millimeter-wave radars on the vehicle, and perform structured processing on the number of historical acquisition data to obtain a number of structured acquisition data.

[0069] As mentioned above, the image data collected by the camera is generally unstructured data, and the image data can be converted into structured data by means of object detection or object segmentation of the image data through a set deep learning model, through 3D assisted reconstruction, or through metadata embedding.

[0070] In addition, the lidar point cloud data collected by the lidar is generally semi-structured data, and the lidar point cloud data can be converted into structured data by means of point cloud semantic segmentation, voxelization processing, or object-level abstraction, while the millimeter-wave data collected by the millimeter-wave radar is generally structured data and does not need to be converted as above.

[0071] S220. Perform named entity recognition on each structured acquisition data to obtain each key entity and the attribute information of each key entity included in the vehicle driving scenario, where the attribute information includes static attribute information and dynamic attribute information.

[0072] S230. Input each key entity and the attribute information of each key entity into a pre-trained entity relationship extraction model to obtain the temporal relationship, spatial relationship, and behavioral relationship between each key entity.

[0073] In this embodiment, the entity relationship extraction model can be constructed by fine-tuning a pre-trained language model (e.g., a classic BETR model or a generative model) or through dynamic prompt learning.

[0074] As shown above, a set of time- and space-aligned structured acquisition data can be used to construct a set of key entities and the corresponding attribute information for each key entity. Each time a set of the above information is input, a set of temporal relationships, spatial relationships, and behavioral relationships between the key entities can be obtained.

[0075] S240. Construct a vehicle driving scenario knowledge base based on the attribute information of each key entity, as well as the temporal relationship, spatial relationship, and behavioral relationship between each key entity.

[0076] Based on the above embodiments, constructing a vehicle driving scenario knowledge base according to the attribute information of each key entity, as well as the temporal relationship, spatial relationship, and behavioral relationship between each key entity, may further include:

[0077] Initialize and construct an original knowledge base according to the attribute information of each key entity, as well as the temporal relationship, spatial relationship, and behavioral relationship between each key entity;

[0078] Perform entity disambiguation processing on the original knowledge base according to the local features and / or global features of each key entity in the original knowledge base to obtain a disambiguated knowledge base;

[0079] Perform knowledge base conflict resolution processing according to the attribute information of each key entity in the disambiguated knowledge base to obtain a vehicle driving scenario knowledge base.

[0080] S250. Establish multiple RDF (Resource Description Framework) triples adapted to the vehicle driving scenario in the vehicle driving scenario knowledge base.

[0081] Based on the vehicle driving scenario knowledge base constructed for each group of spatio-temporal aligned structured acquisition data, multiple RDF triples corresponding to each group of structured acquisition data can be summarized. Among them, the RDF triple specifically includes <subject, predicate, object>. In the vehicle driving scenario described in the embodiments of the present invention, the RDF triple can be further extended to further include a timestamp and spatial coordinates to improve the training accuracy of the subsequent multi-modal fusion network.

[0082] In a specific example, the extended RDF triple can be <vehicle sensor 123, observes, obstacle 456> {timestamp: 1111; spatial coordinates: 2222}.

[0083] S260. Train the set neural network model according to each RDF triple to obtain the multi-modal fusion network.

[0084] Using multiple RDF triples respectively corresponding to each group of structured acquisition data aligned in space and time to train the set neural network model respectively, the multi-modal fusion network can be obtained.

[0085] The above implementation method constructs a knowledge graph (i.e., vehicle driving scenario knowledge base) by using historical acquisition data measured by actual vehicle sensors, and trains a multi-modal fusion network for multi-source heterogeneous sensor data fusion based on this knowledge graph, which can enhance semantic-level environmental understanding. For example, the "moving object" detected by lidar and the "white fuzzy image" recognized by the camera can be associated as the same entity through the knowledge graph, which can greatly eliminate the description ambiguity of sensor devices. In addition, it can also realize the complementary mining of features collected by different sensors, and based on some correct rule logics in the knowledge graph (such as <construction section, associated with, temporary speed limit 30km / h>), the original data judgment of the sensor can be corrected in real time. At the same time, based on this knowledge graph, the recognition accuracy of rare objects can be effectively improved, and the vehicle's recognition and processing capabilities for various complex scenarios can be improved.

[0086] Figure 3 The figure is a flowchart of another heterogeneous data fusion method for a cockpit-integrated vehicle provided by the present invention. This embodiment is refined based on the above embodiments. In this embodiment, the operations of "real-time collecting a multi-source heterogeneous data set inside and outside the vehicle through multiple cameras, multiple lidars and multiple millimeter-wave radars uniformly wired in the vehicle" and "inputting the multi-source heterogeneous data set into a pre-trained multi-modal fusion network for multi-source heterogeneous data fusion processing to obtain vehicle environment perception data for describing the real-time environment inside and outside the vehicle" are specifically implemented.

[0087] Correspondingly, as Figure 3 shown, the method may specifically include:

[0088] S310. Real-time collect a multi-source heterogeneous data set inside and outside the vehicle through multiple cameras, multiple lidars and multiple millimeter-wave radars uniformly wired in the vehicle, and add PTP (Precision Time Protocol) timestamps to each item of multi-source heterogeneous data.

[0089] Among them, the multi-source heterogeneous dataset includes multiple groups of image acquisition data, multiple groups of lidar data, and multiple groups of millimeter wave data.

[0090] S320: Use a preset data alignment algorithm, with the PTP timestamps in each multi-source heterogeneous data as the index, to achieve time alignment between each multi-source heterogeneous data.

[0091] S330: According to the first calibration result between the millimeter wave radar and the camera established in advance, and the second calibration result between the lidar and the camera, map each multi-source heterogeneous data to a unified coordinate system respectively to achieve spatial alignment between each multi-source heterogeneous data.

[0092] In this embodiment, in the data preprocessing and alignment link, hardware triggering (PTP protocol) or software interpolation method is used to achieve time synchronization. Through a checkerboard calibration board, jointly optimize the external parameters, and use the point cloud and image corner point matching to achieve lidar-camera spatial calibration; use the geometric constraint between the target reflection point and the image projection to achieve millimeter wave radar-camera calibration.

[0093] Based on the above settings, the above three types of data with time synchronization and spatial alignment can be converted to a unified coordinate system. On the basis of solving the spatio-temporal alignment, complete the feature complementarity mining, redundancy and conflict resolution problems, and complete the data preprocessing and alignment.

[0094] S340: In the multi-source heterogeneous dataset that has completed time alignment and spatial alignment, extract lidar point cloud data features, millimeter wave data features, and image data features respectively, and splice the lidar point cloud data features, millimeter wave data features, and image features into a multi-channel input.

[0095] The above steps achieve dataset fusion. The so-called data-level fusion refers to projecting the lidar point cloud into the image coordinate system to generate RGB-D data. Through the sensor calibration parameters, splice the radar point cloud, millimeter wave target, and image pixels into a multi-channel input. Align and fuse multi-source data at the raw data level to achieve data-level fusion.

[0096] S350: Input the spliced multi-channel input into the camera branch, lidar branch, and millimeter wave radar branch in the multi-modal fusion network respectively.

[0097] S360: Extract image semantic features from the image data features in the multi-channel input through the camera branch.

[0098] S370: Extract geometric features from the lidar point cloud data features in the multi-channel input through the lidar branch using a point cloud processing network.

[0099] S380. Extract the velocity sequence features from the millimeter-wave data features in the multi-channel input using a timing network through the millimeter-wave radar branch.

[0100] S390. Based on a preset self-attention mechanism algorithm, perform feature fusion on the extracted image semantic features, geometric features, and velocity sequence features to obtain multi-modal fusion features.

[0101] The above operations extract image semantic features through the camera branch, extract geometric features through the point cloud processing network of the lidar branch, process the velocity sequence using the timing network of the millimeter-wave radar branch, and perform feature stitching on this basis to complete the deep learning multi-branch network. By allocating weights through self-attention, dynamically fusing key features, and extracting the intermediate features of each modality, feature-level fusion is achieved.

[0102] S3100. According to the multi-modal fusion features, obtain vehicle environment perception data for describing the real-time environment inside and outside the vehicle.

[0103] S3110. According to the vehicle environment perception data, perform unified decision control on the autonomous driving system and the cockpit system.

[0104] The technical solution of the embodiment of the present invention improves the environmental adaptability, installation stability, and reliability while reducing the cost by comprehensively arranging the installation positions and angles of sensors, sharing some hardware resources, and simplifying the internal wiring structure of the vehicle. By optimizing the algorithm, multi-source data can be quickly and accurately fused in complex scenarios, reducing the perception deviation of the system from the environment and enhancing the user experience.

[0105] Figure 4 It is a schematic structural diagram of a heterogeneous data fusion device for a cockpit-integrated vehicle provided by an embodiment of the present invention. The device is configured in a cockpit-integrated autonomous driving vehicle that has completed unified wiring of multi-source heterogeneous sensors. As Figure 4 shown, the device includes: a multi-source heterogeneous data set acquisition module 410, a vehicle environment perception data acquisition module 420, and a unified decision-making module 430, where:

[0106] The multi-source heterogeneous data set acquisition module 410 is used to collect the multi-source heterogeneous data set inside and outside the vehicle in real time through multiple cameras, multiple lidars, and multiple millimeter-wave radars that are uniformly wired in the vehicle; among them, the multi-source heterogeneous data set includes multiple groups of image acquisition data, multiple groups of lidar data, and multiple groups of millimeter-wave data.

[0107] The vehicle environment perception data acquisition module 420 is used to input the multi-source heterogeneous data set into a pre-trained multi-modal fusion network for fusion processing of the multi-source heterogeneous data, and obtain vehicle environment perception data for describing the real-time environment inside and outside the vehicle.

[0108] Among them, the multi-modal fusion network is pre-trained based on a knowledge graph constructed from multiple historical acquisition data obtained by cameras, lidars, and millimeter-wave radars on at least one vehicle;

[0109] The unified decision-making module 430 is used to perform unified decision-making control on the autonomous driving system and the cockpit system according to the vehicle environment perception data.

[0110] The technical solution of the embodiment of the present invention first completes the unified wiring of multi-source heterogeneous sensors in the integrated cockpit autonomous driving vehicle, and then real-time collects a multi-source heterogeneous data set inside and outside the vehicle through multiple cameras, multiple lidars, and multiple millimeter-wave radars that are uniformly wired in the vehicle; the multi-source heterogeneous data set is input into a multi-modal fusion network pre-trained via a knowledge graph for fusion processing of multi-source heterogeneous data, and vehicle environment perception data for describing the real-time environment inside and outside the vehicle is obtained. According to the vehicle environment perception data, the technical means of performing unified decision-making control on the autonomous driving system and the cockpit system realizes the sharing of some hardware resources by the autonomous driving system and the cockpit system. On the premise of simplifying the internal wiring structure of the vehicle and reducing the hardware cost, through an efficient multi-source heterogeneous fusion algorithm, multi-source heterogeneous data can be quickly fused to obtain vehicle environment perception data in various complex scenarios, so as to accurately and efficiently perform unified decision-making control on the autonomous driving system and the cockpit system, effectively improving the coordination between the autonomous driving system and the cockpit system, and further effectively ensuring the safety of the people in the vehicle and the user experience.

[0111] Based on the above embodiments, it may further include a multi-modal fusion network training module for:

[0112] Before inputting the multi-source heterogeneous signal set into the pre-trained multi-modal fusion network for fusion processing, collect multiple historical acquisition data obtained by cameras, lidars, and millimeter-wave radars on the vehicle, and perform structured processing on the multiple historical acquisition data to obtain multiple structured acquisition data;

[0113] Perform named entity recognition on each structured acquisition data to obtain each key entity and the attribute information of each key entity included in the vehicle driving scenario, where the attribute information includes static attribute information and dynamic attribute information;

[0114] Input each key entity and the attribute information of each key entity into a pre-trained entity relationship extraction model to obtain the temporal relationship, spatial relationship, and behavioral relationship between each key entity;

[0115] According to the attribute information of each key entity, as well as the temporal relationship, spatial relationship, and behavioral relationship between each key entity, construct a vehicle driving scenario knowledge base;

[0116] In the vehicle driving scenario knowledge base, establish multiple RDF triples adapted to the vehicle driving scenario;

[0117] Train a set neural network model according to each RDF triple to obtain the multi-modal fusion network.

[0118] Based on the above embodiments, the multi-modal fusion network training module can further be used for:

[0119] Initialize and construct an original knowledge base according to the attribute information of each key entity, as well as the time relationship, spatial relationship, and behavior relationship between each key entity;

[0120] Perform entity disambiguation processing on the original knowledge base according to the local features and / or global features of each key entity in the original knowledge base to obtain a disambiguated knowledge base;

[0121] Perform knowledge base conflict resolution processing according to the attribute information of each key entity in the disambiguated knowledge base to obtain a vehicle driving scenario knowledge base.

[0122] Based on the above embodiments, the multi-source heterogeneous dataset acquisition module 410 can specifically be used for:

[0123] Real-time collect a multi-source heterogeneous dataset inside and outside the vehicle through multiple cameras, multiple lidars, and multiple millimeter-wave radars with unified wiring in the vehicle, and add PTP timestamps to each item of multi-source heterogeneous data;

[0124] Correspondingly, it can further include a spatio-temporal alignment module for:

[0125] Before inputting the multi-source heterogeneous dataset into a pre-trained multi-modal fusion network for multi-source heterogeneous data fusion processing, use a preset data alignment algorithm and use the PTP timestamps in each item of multi-source heterogeneous data as indexes to achieve time alignment between each item of multi-source heterogeneous data;

[0126] According to the first calibration result between the millimeter-wave radar and the camera established in advance, and the second calibration result between the lidar and the camera, map each item of multi-source heterogeneous data into a unified coordinate system to achieve spatial alignment between each item of multi-source heterogeneous data.

[0127] Based on the above embodiments, the vehicle environment perception data acquisition module 420 can specifically be used for:

[0128] In the multi-source heterogeneous dataset that has completed time alignment and spatial alignment, extract lidar point cloud data features, millimeter-wave data features, and image data features respectively, and splice the lidar point cloud data features, millimeter-wave data features, and image features into a multi-channel input;

[0129] Input the spliced multi-channel inputs into the camera branch, lidar branch, and millimeter-wave radar branch in the multi-modal fusion network respectively;

[0130] Extract image semantic features from the image data features in the multi-channel input through the camera branch;

[0131] Extract geometric features from the lidar point cloud data features in the multi-channel input through the lidar branch using a point cloud processing network;

[0132] Extract velocity sequence features from the millimeter-wave data features in the multi-channel input through the millimeter-wave radar branch using a time series network;

[0133] Based on a preset self-attention mechanism algorithm, perform feature fusion on the extracted image semantic features, geometric features, and velocity sequence features to obtain multi-modal fusion features;

[0134] According to the multi-modal fusion features, obtain vehicle environment perception data for describing the real-time environment inside and outside the vehicle.

[0135] Based on the above embodiments, the unified decision-making module 430 can be specifically used for:

[0136] According to the vehicle environment perception data, uniformly execute driving decision control for the autonomous driving system, adjustment decision control for at least one adjustable facility in the cockpit, and display decision control for the real-time display content on the cockpit display screen.

[0137] Based on the above embodiments, it may further include an intelligent response module for:

[0138] After obtaining the vehicle environment perception data for describing the real-time environment inside and outside the vehicle, perform intelligent responses to the real-time questions of the personnel in the cockpit according to the vehicle environment perception data.

[0139] The heterogeneous data fusion device for a cockpit-integrated vehicle provided by the embodiments of the present invention can execute the heterogeneous data fusion method for a cockpit-integrated vehicle provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0140] In the technical solution of the present disclosure, the processing of collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved all complies with the provisions of relevant laws and regulations and does not violate public order and good customs.

[0141] Figure 5The structural schematic diagram of the cockpit integrated autonomous driving vehicle 10 that can be used to implement the embodiments of the present invention is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0142] As Figure 5 shown, the cockpit integrated autonomous driving vehicle 10 includes a multi-source heterogeneous sensor 101, at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the cockpit integrated autonomous driving vehicle 10 can also be stored. The multi-source heterogeneous sensor 101, the processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0143] Among them, the above-mentioned multi-source heterogeneous sensor 101 may specifically include: a plurality of cameras (not shown in the figure) that complete unified wiring, a plurality of lidars (not shown in the figure), and a plurality of millimeter-wave radars (not shown in the figure), which are used to collect multi-source heterogeneous data sets inside and outside the vehicle in real time.

[0144] A plurality of components in the cockpit integrated autonomous driving vehicle 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the cockpit integrated autonomous driving vehicle 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0145] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as executing the heterogeneous data fusion method based on the cockpit integrated vehicle as described in the embodiments of the present invention. That is:

[0146] Collect a multi-source heterogeneous data set inside and outside the vehicle in real time through multiple cameras, multiple lidars, and multiple millimeter-wave radars with unified wiring in the vehicle; among them, the multi-source heterogeneous data set includes multiple groups of image acquisition data, multiple groups of lidar data, and multiple groups of millimeter-wave data;

[0147] Input the multi-source heterogeneous data set into a pre-trained multi-modal fusion network for fusion processing of multi-source heterogeneous data, and obtain vehicle environment perception data for describing the real-time environment inside and outside the vehicle;

[0148] Among them, the multi-modal fusion network is pre-trained based on a knowledge graph constructed from a number of historical acquisition data obtained by cameras, lidars, and millimeter-wave radars on at least one vehicle;

[0149] Perform unified decision control on the autonomous driving system and the cockpit system according to the vehicle environment perception data.

[0150] In some embodiments, the heterogeneous data fusion method based on the integrated cockpit vehicle as described in the embodiments of the present invention can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the integrated cockpit autonomous vehicle 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by the processor 11, one or more steps of the heterogeneous data fusion method based on the integrated cockpit vehicle as described in the embodiments of the present invention above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the heterogeneous data fusion method based on the integrated cockpit vehicle as described in the embodiments of the present invention by any other suitable means (for example, by means of firmware).

[0151] The various embodiments of the systems and technologies described above in this article can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs, the one or more computer programs can be executed and / or interpreted on a programmable system including at least one programmable processor, the programmable processor can be a dedicated or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] A computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0155] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected with each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0156] A computing system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0157] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0158] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A heterogeneous data fusion method for a cockpit-integrated vehicle, which is executed by a cockpit-integrated autonomous vehicle that completes the unified wiring of multi-source heterogeneous sensors, and is characterized in that, The method includes: Real - time collecting a multi - source heterogeneous data set inside and outside the vehicle through multiple cameras, multiple lidars, and multiple millimeter - wave radars with unified wiring in the vehicle; wherein, the multi - source heterogeneous data set includes multiple groups of image acquisition data, multiple groups of lidar data, and multiple groups of millimeter - wave data; Inputting the multi - source heterogeneous data set into a pre - trained multi - modal fusion network for fusion processing of the multi - source heterogeneous data to obtain vehicle environment perception data for describing the real - time environment inside and outside the vehicle; Among them, the multi - modal fusion network is pre - trained based on a knowledge graph constructed from multiple historical acquisition data obtained by at least one camera, lidar, and millimeter - wave radar on the vehicle; According to the vehicle environment perception data, unified decision - making control is performed on the autonomous driving system and the cockpit system.

2. The method according to claim 1, characterized in that Before inputting the multi - source heterogeneous signal set into the pre - trained multi - modal fusion network for fusion processing, it further includes: Collecting multiple historical acquisition data obtained by the cameras, lidars, and millimeter - wave radars on the vehicle, and performing structured processing on the multiple historical acquisition data to obtain multiple structured acquisition data; Performing named - entity recognition on each structured acquisition data to obtain each key entity and the attribute information of each key entity included in the vehicle driving scenario, wherein the attribute information includes static attribute information and dynamic attribute information; Inputting each key entity and the attribute information of each key entity into a pre - trained entity - relationship extraction model to obtain the temporal relationship, spatial relationship, and behavioral relationship between each key entity; According to the attribute information of each key entity, as well as the temporal relationship, spatial relationship, and behavioral relationship between each key entity, constructing a vehicle driving scenario knowledge base; In the vehicle driving scenario knowledge base, establishing multiple resource description framework RDF triples adapted to the vehicle driving scenario; Training a set neural network model according to each RDF triple to obtain the multi - modal fusion network.

3. The method according to claim 2, wherein Constructing a vehicle driving scenario knowledge base according to the attribute information of each key entity, as well as the temporal relationship, spatial relationship, and behavioral relationship between each key entity, including: Initializing and constructing an original knowledge base according to the attribute information of each key entity, as well as the temporal relationship, spatial relationship, and behavioral relationship between each key entity; Performing entity disambiguation processing on the original knowledge base according to the local features and / or global features of each key entity in the original knowledge base to obtain a disambiguated knowledge base; Performing knowledge - base conflict resolution processing according to the attribute information of each key entity in the disambiguated knowledge base to obtain a vehicle driving scenario knowledge base.

4. The method according to any one of claims 1 to 3, characterized in that, Real - time collecting a multi - source heterogeneous data set inside and outside the vehicle through multiple cameras, multiple lidars, and multiple millimeter - wave radars with unified wiring in the vehicle, specifically including: Real - time collecting a multi - source heterogeneous data set inside and outside the vehicle through multiple cameras, multiple lidars, and multiple millimeter - wave radars with unified wiring in the vehicle, and respectively adding a Precision Time Protocol PTP timestamp to each item of the multi - source heterogeneous data; Before inputting the multi - source heterogeneous data set into the pre - trained multi - modal fusion network for fusion processing, it further includes: Adopt a preset data alignment algorithm, and use the PTP timestamps in each item of multi-source heterogeneous data as indexes to achieve time alignment between each item of multi-source heterogeneous data; According to the first calibration result between the millimeter-wave radar and the camera, and the second calibration result between the lidar and the camera established in advance, map each item of multi-source heterogeneous data to a unified coordinate system respectively to achieve spatial alignment between each item of multi-source heterogeneous data.

5. The method according to claim 4, characterized in that, Input the multi-source heterogeneous data set into a pre-trained multi-modal fusion network for fusion processing of multi-source heterogeneous data, and obtain vehicle environment perception data for describing the real-time environment inside and outside the vehicle, including: In the multi-source heterogeneous data set with time alignment and spatial alignment completed, extract lidar point cloud data features, millimeter-wave data features, and image data features respectively, and splice the lidar point cloud data features, millimeter-wave data features, and image features into a multi-channel input; Input the spliced multi-channel input into the camera branch, lidar branch, and millimeter-wave radar branch in the multi-modal fusion network respectively; Extract image semantic features from the image data features in the multi-channel input through the camera branch; Extract geometric features from the lidar point cloud data features in the multi-channel input through the lidar branch using a point cloud processing network; Extract speed sequence features from the millimeter-wave data features in the multi-channel input through the millimeter-wave radar branch using a timing network; Based on a preset self-attention mechanism algorithm, perform feature fusion on the extracted image semantic features, geometric features, and speed sequence features to obtain multi-modal fusion features; According to the multi-modal fusion features, obtain vehicle environment perception data for describing the real-time environment inside and outside the vehicle.

6. The method according to claim 1, wherein According to the vehicle environment perception data, perform unified decision control on the autonomous driving system and the cockpit system, including: According to the vehicle environment perception data, uniformly execute driving decision control for the autonomous driving system, adjustment decision control for at least one adjustable facility in the cockpit, and display decision control for the real-time display content of the cockpit display screen.

7. The method according to claim 1, characterized in that After obtaining the vehicle environment perception data for describing the real-time environment inside and outside the vehicle, it further includes: According to the vehicle environment perception data, perform intelligent response to the real-time questions of the people in the cockpit.

8. A heterogeneous data fusion device for a cockpit-integrated vehicle, configured in a cockpit-integrated autonomous vehicle that has completed unified wiring of multi-source heterogeneous sensors, characterized in that, The device includes: A multi-source heterogeneous data set acquisition module for real-time acquisition of a multi-source heterogeneous data set inside and outside the vehicle through multiple cameras, multiple lidars, and multiple millimeter-wave radars with unified wiring in the vehicle; among them, the multi-source heterogeneous data set includes multiple groups of image acquisition data, multiple groups of lidar data, and multiple groups of millimeter-wave data; A vehicle environment perception data acquisition module for inputting the multi-source heterogeneous data set into a pre-trained multi-modal fusion network for fusion processing of multi-source heterogeneous data, and obtaining vehicle environment perception data for describing the real-time environment inside and outside the vehicle; Among them, the multi-modal fusion network is pre-trained based on a knowledge graph constructed from multiple historical acquisition data obtained by at least one camera, lidar, and millimeter-wave radar on the vehicle; A unified decision-making module for making unified decision control over the autonomous driving system and the cockpit system according to vehicle environment perception data.

9. An integrated cockpit autonomous vehicle, characterized in that, The vehicle includes: Multiple cameras, multiple lidars, and multiple millimeter-wave radars that have completed unified wiring, for real-time collection of multi-source heterogeneous data sets inside and outside the vehicle; At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the heterogeneous data fusion method for a cockpit-integrated vehicle according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for implementing the heterogeneous data fusion method for a cockpit-integrated vehicle according to any one of claims 1-7 when executed by a processor.

Images