Working condition prediction and energy management method, device and equipment based on multi-modal information fusion and storage medium

Abstract

The invention discloses a working condition prediction and energy management method and device based on multi-modal information fusion, equipment and a storage medium, and relates to the technical field of working condition prediction. The multi-modal information fusion-based working condition prediction and energy management method comprises the steps of receiving three-dimensional point cloud data acquired by a laser radar in real time and two-dimensional image data acquired by a camera in real time; extracting and fusing features in the three-dimensional point cloud data and the two-dimensional image data to generate a fused feature vector; off-line map data are obtained, the off-line map data are updated in real time according to the fusion feature vector, and real-time map data are obtained; and obtaining historical speed data of the vehicle, obtaining a predicted vehicle speed according to the fusion feature vector and the historical speed data of the vehicle, and performing energy management according to the real-time map data and the predicted vehicle speed. The accuracy of working condition prediction can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of operating condition prediction, and in particular to an operating condition prediction and energy management method, apparatus, device and storage medium based on multimodal information fusion. Background Art

[0002] With the continuous development of autonomous driving technology and intelligent transportation systems, lidar (LiDAR) plays a vital role in environmental perception and operating condition prediction. However, traditional lidar systems can exhibit unstable performance in complex traffic scenarios and harsh environmental conditions. Therefore, more intelligent and reliable operating condition prediction methods are needed to improve system performance and safety. Currently, most research on operating condition predictions has low accuracy, so improving the accuracy of operating condition prediction remains an unresolved issue.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device, equipment and storage medium for operating condition prediction and energy management based on multimodal information fusion, aiming to solve the technical problem of how to improve the accuracy of operating condition prediction.

[0005] To achieve the above objectives, this application proposes a method for operating condition prediction and energy management based on multimodal information fusion, the method comprising:

[0006] Receive 3D point cloud data collected by the lidar in real time and 2D image data collected by the camera in real time;

[0007] Extracting and fusing features from the three-dimensional point cloud data and the two-dimensional image data to generate a fused feature vector;

[0008] Obtaining offline map data, and updating the offline map data in real time according to the fused feature vector to obtain real-time map data;

[0009] Acquire historical vehicle speed data, obtain a predicted vehicle speed based on the fused feature vector and the historical vehicle speed data, and perform energy management based on the real-time map data and the predicted vehicle speed.

[0010] In one embodiment, the step of extracting and fusing features from the three-dimensional point cloud data and the two-dimensional image data to generate a fused feature vector includes:

[0011] Processing the two-dimensional image data through a convolutional neural network to generate an image feature vector;

[0012] Processing the three-dimensional point cloud data through a point cloud processing network to generate a point cloud feature vector;

[0013] The image feature vector and the point cloud feature vector are weightedly spliced ​​and fused to generate the fused feature vector.

[0014] In one embodiment, the step of updating offline map data in real time according to the fused feature vector to obtain real-time map data includes:

[0015] Obtaining environmental perception information according to the fused feature vector;

[0016] The road information in the offline map data is updated according to the environmental perception information to obtain real-time map data.

[0017] In one embodiment, the step of obtaining the predicted vehicle speed based on the fused feature vector and the vehicle historical speed data includes:

[0018] Processing the fused feature vector and the historical vehicle speed data through a gated recurrent unit neural network to obtain an initial predicted vehicle speed;

[0019] Generate a state transition probability matrix based on historical vehicle speed data;

[0020] The initial predicted vehicle speed and the state transition probability matrix are processed by a Markov chain model to obtain a predicted vehicle speed.

[0021] In one embodiment, the step of processing the initial predicted vehicle speed and the state transition probability matrix through a Markov chain model to obtain the predicted vehicle speed includes:

[0022] determining a current vehicle speed state according to the initial predicted vehicle speed;

[0023] Obtaining the transition probability from the current vehicle speed state to the next state according to the state transition probability matrix;

[0024] The predicted vehicle speed is obtained according to the transition probability.

[0025] In one embodiment, the step of performing energy management based on the real-time map data and the predicted vehicle speed includes:

[0026] Identifying the current operating condition type of the vehicle based on the real-time map data;

[0027] Building a dynamic energy consumption optimization model according to the operating condition type and the predicted vehicle speed;

[0028] Obtain the power battery state of charge constraints and maximum output power constraints;

[0029] Obtaining motor power distribution parameters and braking control parameters according to the dynamic energy consumption optimization model, the power battery state of charge constraint, and the maximum output power constraint;

[0030] The vehicle power is adjusted according to the motor power distribution parameters and the brake control parameters.

[0031] In one embodiment, the step of constructing a dynamic energy consumption optimization model according to the operating condition type and the predicted vehicle speed includes:

[0032] When the operating condition type is a normal operating condition, a dynamic energy consumption optimization model is constructed based on the predicted vehicle speed;

[0033] When the operating condition type is an emergency operating condition, an emergency operating condition penalty factor is obtained, and a dynamic energy consumption optimization model is constructed according to the predicted vehicle speed and the emergency operating condition penalty factor.

[0034] In addition, to achieve the above objectives, the present application also proposes a working condition prediction and energy management device based on multimodal information fusion, the working condition prediction and energy management device based on multimodal information fusion comprising:

[0035] A receiving module is used to receive the three-dimensional point cloud data collected by the lidar in real time and the two-dimensional image data collected by the camera in real time;

[0036] A fusion module, configured to extract and fuse features from the three-dimensional point cloud data and the two-dimensional image data to generate a fused feature vector;

[0037] An updating module, configured to obtain offline map data and update the offline map data in real time according to the fused feature vector to obtain real-time map data;

[0038] The prediction module is used to obtain vehicle historical speed data, obtain a predicted vehicle speed based on the fused feature vector and the vehicle historical speed data, and perform energy management based on the real-time map data and the predicted vehicle speed.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a working condition prediction and energy management device based on multimodal information fusion, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the working condition prediction and energy management method based on multimodal information fusion as described above.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the operating condition prediction and energy management method based on multimodal information fusion as described above are implemented.

[0041] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the operating condition prediction and energy management method based on multimodal information fusion as described above.

[0042] The present application provides a working condition prediction and energy management method based on multimodal information fusion. The present application receives three-dimensional point cloud data collected in real time by a lidar and two-dimensional image data collected in real time by a camera; extracts and fuses features in the three-dimensional point cloud data and two-dimensional image data to generate a fusion feature vector; obtains offline map data, updates the offline map data in real time according to the fusion feature vector to obtain real-time map data; obtains vehicle historical speed data, obtains a predicted vehicle speed according to the fusion feature vector and the vehicle historical speed data, and performs energy management according to the real-time map data and the predicted vehicle speed.

[0043] In summary, this application improves the accuracy of working condition prediction by integrating multiple data to predict vehicle speed and combining it with real-time map data to manage energy. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 A flowchart of the first embodiment of the method for operating condition prediction and energy management based on multimodal information fusion provided in this application;

[0047] Figure 2 A flowchart of special operating condition prediction provided in Example 1 of the operating condition prediction and energy management method based on multimodal information fusion of this application;

[0048] Figure 3 A flowchart of the second embodiment of the method for operating condition prediction and energy management based on multimodal information fusion provided in this application;

[0049] Figure 4 A structural flow chart of a GRU neural network provided in Example 2 of the working condition prediction and energy management method based on multimodal information fusion of this application;

[0050] Figure 5 A schematic diagram of a simplified process of the method for operating condition prediction and energy management based on multimodal information fusion provided in Example 1 of the present application;

[0051] Figure 6 This is a schematic diagram of the module structure of the operating condition prediction and energy management device based on multimodal information fusion according to an embodiment of the present application;

[0052] Figure 7 Schematic diagram of the equipment structure of the hardware operating environment involved in the working condition prediction and energy management method based on multimodal information fusion in the embodiment of the present application.

[0053] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0055] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0056] The main solution of this application is to receive three-dimensional point cloud data collected by the lidar in real time and two-dimensional image data collected by the camera in real time; extract and fuse the features in the three-dimensional point cloud data and two-dimensional image data to generate a fused feature vector; obtain offline map data, update the offline map data in real time according to the fused feature vector to obtain real-time map data; obtain vehicle historical speed data, obtain predicted vehicle speed according to the fused feature vector and the vehicle historical speed data, and perform energy management based on the real-time map data and the predicted vehicle speed.

[0057] At present, the accuracy of working condition prediction in most studies is not high. Therefore, how to improve the accuracy of working condition prediction is a problem that still needs to be solved.

[0058] In summary, this application improves the accuracy of working condition prediction by integrating multiple data to predict vehicle speed and combining it with real-time map data to manage energy.

[0059] Based on this, the embodiment of the present application provides a working condition prediction and energy management method based on multimodal information fusion, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the operating condition prediction and energy management method based on multimodal information fusion of this application.

[0060] In this embodiment, the operating condition prediction and energy management method based on multimodal information fusion includes steps S10 to S40:

[0061] Step S10: receiving the three-dimensional point cloud data collected in real time by the laser radar and the two-dimensional image data collected in real time by the camera;

[0062] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication, and program execution capabilities, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of implementing the aforementioned functions, or an operating condition prediction and energy management device based on multimodal information fusion. This embodiment and the following embodiments will be described below using an operating condition prediction and energy management device based on multimodal information fusion as an example.

[0063] It should be noted that LiDAR captures 3D point cloud data of the surrounding environment by emitting laser pulses and receiving reflected signals. The core output of LiDAR is point cloud, which is usually represented as discrete points (x i ,y i ,z i ), these points represent the surface of objects in the surrounding environment. Assume that the point cloud data captured by the lidar at time t is:

[0064] P t ={(x i ,y i ,z i )|i=1,2,…,N}

[0065] Where N is the number of points captured by the lidar, (x i ,y i ,z i ) are the coordinates of each point in three-dimensional space.

[0066] Step S20: extracting and fusing features from the three-dimensional point cloud data and the two-dimensional image data to generate a fused feature vector;

[0067] It is understandable that through multimodal neural networks, data from different sensors (such as cameras and lidars) are input into different branch networks, the camera obtains two-dimensional images, the lidar obtains three-dimensional point clouds, and feature fusion is performed in high-dimensional space to improve the perception of the environment.

[0068] In a feasible embodiment, the steps of extracting and fusing features from the three-dimensional point cloud data and the two-dimensional image data to generate a fused feature vector include: processing the two-dimensional image data through a convolutional neural network to generate an image feature vector; processing the three-dimensional point cloud data through a point cloud processing network to generate a point cloud feature vector; and performing weighted splicing and fusing the image feature vector and the point cloud feature vector to generate the fused feature vector.

[0069] It should be noted that image feature extraction is to extract the two-dimensional feature vector F from the camera image. img , processed by a convolutional neural network (CNN):

[0070] F img =CNN(I t )

[0071] Among them, I t is the image captured by the camera at time t.

[0072] Point cloud feature extraction, extracting 3D geometric features from LiDAR point clouds lidar , which can be achieved through voxelization or direct point cloud processing networks:

[0073] F lidar =PointNet(P t )

[0074] Feature fusion, the image feature F img and point cloud features F lidar Fusion can be achieved through weighted summation, concatenation or attention mechanism:

[0075] F fusion =Fusion(F img ,F lidar )

[0076] Among them, F fusion It is the fused feature vector, which contains the environmental perception information from different modalities.

[0077] Step S30: acquiring offline map data, and updating the offline map data in real time according to the fused feature vector to obtain real-time map data;

[0078] It is understandable that the real-time map data can be used to obtain the road sections that the vehicle will travel in the future, so as to determine whether it will encounter sudden nodes, such as intersections.

[0079] In a feasible manner, the step of updating offline map data in real time according to the fused feature vector to obtain real-time map data includes: obtaining environmental perception information according to the fused feature vector; and updating road information in the offline map data according to the environmental perception information to obtain real-time map data.

[0080] It is understandable that the fused data P t It can be updated in real time with the road information in the offline map. The mathematical description of real-time map update is:

[0081] M t =M t-1 ∪f(Pt )

[0082] Among them, M t represents the map state at time t, P t is the fusion data captured at time t, function f(P t ) is responsible for integrating this data into the map, which can enhance the perception of roads and emergencies.

[0083] Step S40: Acquire historical vehicle speed data, obtain a predicted vehicle speed based on the fused feature vector and the historical vehicle speed data, and perform energy management based on the real-time map data and the predicted vehicle speed.

[0084] It should be noted that multimodal data fusion combines data from different sensors into a joint feature vector, which represents information in multiple dimensions. The GRU neural network captures time series information and uses it to predict the vehicle's speed at the next moment. The GRU output (based on historical data and multimodal fusion features) can then be combined with a Markov chain. The GRU neural network uses the vehicle's historical speed and multimodal fusion environmental perception features to generate a prediction for the next moment's speed. The GRU-predicted speed is then combined with the Markov chain's state transition matrix to calculate the final predicted speed at the next moment. When a vehicle enters a road section with sudden nodes (such as an intersection), the speed prediction model requires real-time adjustments to the vehicle's power and braking systems. The environmental model captured by the lidar not only provides real-time road information but also identifies vehicles, pedestrians, and obstacles on the road. Under sudden driving conditions, the speed prediction model can be integrated with the vehicle's energy management system (EMS). The EMS needs to reassess power distribution, for example, adjusting motor power and engine load based on the current speed and road conditions.

[0085] In one feasible embodiment, the step of performing energy management based on the real-time map data and the predicted vehicle speed includes: identifying the vehicle's current operating condition based on the real-time map data; constructing a dynamic energy consumption optimization model based on the operating condition and the predicted vehicle speed; obtaining power battery state of charge constraints and maximum output power constraints; obtaining motor power distribution parameters and brake control parameters based on the dynamic energy consumption optimization model, the power battery state of charge constraints, and the maximum output power constraints; and adjusting vehicle power based on the motor power distribution parameters and brake control parameters. The step of constructing a dynamic energy consumption optimization model based on the operating condition and the predicted vehicle speed includes: constructing a dynamic energy consumption optimization model based on the predicted vehicle speed when the operating condition is a normal operating condition; and obtaining an emergency operating condition penalty factor when the operating condition is an emergency operating condition, and constructing a dynamic energy consumption optimization model based on the predicted vehicle speed and the emergency operating condition penalty factor.

[0086] It should be noted that this embodiment defines energy consumption models according to the type of vehicle working condition, general or special, and can refer to Figure 2 , Figure 2 This is a flow chart for predicting special working conditions. Under normal working conditions, based on the predicted speed An energy consumption optimization function is established to achieve dynamic motor power scheduling and energy minimization. To ensure that the physical constraints during the optimization problem are met, the system must comply with two types of constraints: global and local. Global constraints primarily control the range of the power battery's SOC. To prevent overcharging or overdischarging and ensure its service life, the minimum battery SOC is set to 0.3 to prevent low battery levels from affecting vehicle performance. Local constraints, on the other hand, are more concerned with the actual physical conditions of the power system, the maximum output power and torque range of the power battery, and ensure that each actuator operates within its permitted range. By introducing these constraints, the EMS optimization problem becomes an optimal control problem with constraints.

[0087]

[0088] to represents the start time, and tf represents the end time. int Represents the initial charge, SOC min Represents the cut-off power. battmax Represents the maximum discharge power of the power battery, which changes with SOC. P battmin Represents the maximum charging power of the power battery, which also changes with SOC. If P batt (t) greater than 0 indicates that the power battery is discharging, otherwise it indicates that the power battery is charging. fuelmax Represents the maximum output power.

[0089] When the vehicle is moving, the motor's required power P mot , satisfying the following formula:

[0090]

[0091] P Batt is the battery charging and discharging power; P Fuel is the fuel output power; η Mot is the motor system efficiency.

[0092] Its objective function is:

[0093]

[0094] is the vehicle speed predicted by the model, is the predicted acceleration, f0, f1, f2 are the rolling resistance and wind resistance coefficient, η(t) is the power system efficiency,

[0095] In emergency scenarios, such as intersections, sudden stops of vehicles ahead, and obstacles, special penalty items and safety factors are introduced to adjust the energy consumption strategy as follows:

[0096]

[0097] The penalty term ψ(t) represents the emergency control cost caused by sudden changes in operating conditions such as sudden braking or acceleration, realizing a dynamic energy optimization strategy based on predicted operating conditions.

[0098] This embodiment provides a working condition prediction and energy management method based on multimodal information fusion. The application receives three-dimensional point cloud data collected in real time by a lidar and two-dimensional image data collected in real time by a camera; extracts and fuses features in the three-dimensional point cloud data and two-dimensional image data to generate a fusion feature vector; obtains offline map data, updates the offline map data in real time according to the fusion feature vector to obtain real-time map data; obtains vehicle historical speed data, obtains a predicted vehicle speed according to the fusion feature vector and the vehicle historical speed data, and performs energy management according to the real-time map data and the predicted vehicle speed.

[0099] In summary, this embodiment improves the accuracy of operating condition prediction by fusing multiple data to predict vehicle speed and combining it with real-time map data to manage energy.

[0100] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 3 , step S40 further includes steps S401 to S403:

[0101] Step S401: Processing the fused feature vector and the vehicle historical speed data through a gated recurrent unit neural network to obtain an initial predicted vehicle speed;

[0102] It should be noted that the multimodal feature F fusion and the vehicle's historical speed data v t It can be used as the input of GRU. GRU neural network can predict the future speed of the vehicle through historical data. The network structure of GRU can be referred to Figure 4 , Figure 4 This is the structural flow chart of the GRU neural network. The calculation process of GRU is as follows:

[0103] Reset Gate r t :

[0104] r t =σ(W r ·[h t-1 ,vt ,F fusion ]+b r )

[0105] Update gate z t :

[0106] z t =σ(W z ·[h t-1 ,v t ,F fusion ]+b z )

[0107] Candidate hidden states

[0108]

[0109] Current hidden state h t :

[0110]

[0111] The hidden state h of the final output of GRU t Can be used to predict the vehicle speed at the next moment

[0112]

[0113] Step S402: generating a state transition probability matrix based on historical vehicle speed data;

[0114] It should be noted that the matrix elements of the state transition probability matrix represent the conditional probability of the vehicle transitioning from the current speed state to the state at the next moment.

[0115] Step S403: Processing the initial predicted vehicle speed and the state transition probability matrix through a Markov chain model to obtain a predicted vehicle speed.

[0116] In a feasible embodiment, the step of processing the initial predicted vehicle speed and the state transition probability matrix through a Markov chain model to obtain the predicted vehicle speed includes: determining the current vehicle speed state based on the initial predicted vehicle speed; obtaining the transition probability from the current vehicle speed state to the next state based on the state transition probability matrix; and obtaining the predicted vehicle speed based on the transition probability.

[0117] In order to improve the prediction effect of the Markov chain, the output of GRU (based on historical data and multimodal fusion features) can be combined with the Markov chain. Through the GRU neural network, the vehicle's historical speed and multimodal fusion environmental perception features are used to generate a prediction of the speed at the next moment. Then the speed predicted by GRU Combined with the Markov chain state transfer matrix, the final predicted speed at the next moment is calculated:

[0118]

[0119] predict.

[0120] When generating predicted speed, the optimal goal of the model is to predict the future vehicle speed based on historical data and current perception information. Combining the output of GRU and the transition probability of the Markov chain, a prediction loss function can be defined to optimize the model parameters. The objective function for predicting vehicle speed can be defined as minimizing the actual speed v t+1 and prediction speed The error between , the commonly used loss function is the mean square error (MSE):

[0121]

[0122] By minimizing this loss function, the model parameters can be optimized, and accurate vehicle speed predictions can be generated in real time under complex working conditions.

[0123] This embodiment processes the fused feature vector and the historical vehicle speed data using a gated recurrent unit neural network to obtain an initial predicted vehicle speed; generates a state transition probability matrix based on the historical vehicle speed data; and processes the initial predicted vehicle speed and the state transition probability matrix using a Markov chain model to obtain a predicted vehicle speed. This embodiment combines the output of the GRU with the Markov chain to obtain the final predicted vehicle speed, thereby improving the accuracy of the operating condition prediction.

[0124] For example, in order to help understand the implementation process of the working condition prediction and energy management method based on multimodal information fusion obtained by combining this embodiment with the above embodiment 1, please refer to Figure 5 , Figure 5 This paper provides a simplified flowchart of a method for operating condition prediction and energy management based on multimodal information fusion. Specifically, it acquires real-time environmental data from lidar and cameras, as well as offline map data, to perform real-time environmental perception map information and path planning. It then uses a multimodal data fusion model and a GRU neural network mathematical model to predict vehicle speed. Furthermore, it uses an improved Markov chain to predict operating conditions, segmenting roads according to road type to generate a prediction of the next-moment speed. The GRU-predicted speed is combined with the Markov chain state transition matrix to generate sub-segment speeds. Finally, the final predicted speed is obtained based on general and special demand conditions. General demand conditions require acceleration and deceleration to optimize energy distribution using EMS. Special demand conditions require identifying sudden nodes, performing acceleration and deceleration, and emergency braking to optimize energy distribution using EMS.

[0125] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the working condition prediction and energy management method based on multimodal information fusion of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0126] This application also provides a working condition prediction and energy management device based on multimodal information fusion, please refer to Figure 6 , the operating condition prediction and energy management device based on multimodal information fusion includes:

[0127] The receiving module 10 is used to receive the three-dimensional point cloud data collected by the laser radar in real time and the two-dimensional image data collected by the camera in real time;

[0128] A fusion module 20 is configured to extract and fuse features from the three-dimensional point cloud data and the two-dimensional image data to generate a fused feature vector;

[0129] An updating module 30 is used to obtain offline map data and update the offline map data in real time according to the fused feature vector to obtain real-time map data;

[0130] The prediction module 40 is used to obtain historical vehicle speed data, obtain a predicted vehicle speed based on the fused feature vector and the historical vehicle speed data, and perform energy management based on the real-time map data and the predicted vehicle speed.

[0131] This embodiment provides a working condition prediction and energy management method based on multimodal information fusion. The application receives three-dimensional point cloud data collected in real time by a lidar and two-dimensional image data collected in real time by a camera; extracts and fuses features in the three-dimensional point cloud data and two-dimensional image data to generate a fusion feature vector; obtains offline map data, updates the offline map data in real time according to the fusion feature vector to obtain real-time map data; obtains vehicle historical speed data, obtains a predicted vehicle speed according to the fusion feature vector and the vehicle historical speed data, and performs energy management according to the real-time map data and the predicted vehicle speed.

[0132] In summary, this embodiment improves the accuracy of operating condition prediction by fusing multiple data to predict vehicle speed and combining it with real-time map data to manage energy.

[0133] In one embodiment, the fusion module 20 is further used to process the two-dimensional image data through a convolutional neural network to generate an image feature vector; process the three-dimensional point cloud data through a point cloud processing network to generate a point cloud feature vector; and perform weighted splicing and fusion of the image feature vector and the point cloud feature vector to generate the fused feature vector.

[0134] In one embodiment, the updating module 30 is further configured to obtain environmental perception information according to the fused feature vector; and update the road information in the offline map data according to the environmental perception information to obtain real-time map data.

[0135] In one embodiment, the prediction module 40 is further used to process the fused feature vector and the vehicle historical speed data through a gated recurrent unit neural network to obtain an initial predicted vehicle speed; generate a state transition probability matrix based on the historical vehicle speed data; and process the initial predicted vehicle speed and the state transition probability matrix through a Markov chain model to obtain a predicted vehicle speed.

[0136] In one embodiment, the prediction module 40 is further used to determine the current vehicle speed state based on the initial predicted vehicle speed; obtain the transition probability from the current vehicle speed state to the next state based on the state transition probability matrix; and obtain the predicted vehicle speed based on the transition probability.

[0137] In one embodiment, the prediction module 40 is further used to identify the current operating condition type of the vehicle based on the real-time map data; construct a dynamic energy consumption optimization model according to the operating condition type and the predicted vehicle speed; obtain the power battery state of charge constraint and the maximum output power constraint; obtain motor power distribution parameters and braking control parameters according to the dynamic energy consumption optimization model, the power battery state of charge constraint and the maximum output power constraint; and adjust the vehicle power according to the motor power distribution parameters and braking control parameters.

[0138] In one embodiment, the prediction module 40 is also used to construct a dynamic energy consumption optimization model based on the predicted vehicle speed when the operating condition type is a normal operating condition; when the operating condition type is an emergency operating condition, obtain an emergency operating condition penalty factor, and construct a dynamic energy consumption optimization model based on the predicted vehicle speed and the emergency operating condition penalty factor.

[0139] The operating condition prediction and energy management device based on multimodal information fusion provided in this application adopts the operating condition prediction and energy management method based on multimodal information fusion in the above-mentioned embodiment, which can solve the technical problem of how to improve the accuracy of operating condition prediction. Compared with the prior art, the beneficial effects of the operating condition prediction and energy management device based on multimodal information fusion provided in this application are the same as the beneficial effects of the operating condition prediction and energy management method based on multimodal information fusion provided in the above-mentioned embodiment, and the other technical features of the operating condition prediction and energy management device based on multimodal information fusion are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0140] The present application provides an operating condition prediction and energy management device based on multimodal information fusion. The operating condition prediction and energy management device based on multimodal information fusion includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the operating condition prediction and energy management method based on multimodal information fusion in the above-mentioned embodiment one.

[0141] Reference below Figure 7 , which shows a schematic structural diagram of an operating condition prediction and energy management device based on multimodal information fusion suitable for implementing an embodiment of the present application. The operating condition prediction and energy management device based on multimodal information fusion in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The operating condition prediction and energy management device based on multimodal information fusion shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0142] like Figure 7As shown, the operating condition prediction and energy management device based on multimodal information fusion may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in ROM (Read Only Memory) 1002 or the program loaded from the storage device 1003 to RAM (Random Access Memory) 1004. Various programs and data required for the operation of the operating condition prediction and energy management device based on multimodal information fusion are also stored in RAM 1004. The processing device 1001, ROM 1002 and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the operating condition prediction and energy management device based on multimodal information fusion to communicate wirelessly or wired with other devices to exchange data. Although the figure shows an operating condition prediction and energy management device based on multimodal information fusion with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0143] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0144] The operating condition prediction and energy management device based on multimodal information fusion provided by this application adopts the operating condition prediction and energy management method based on multimodal information fusion in the above-mentioned embodiment, which can solve the technical problem of how to improve the accuracy of operating condition prediction. Compared with the prior art, the beneficial effects of the operating condition prediction and energy management device based on multimodal information fusion provided by this application are the same as the beneficial effects of the operating condition prediction and energy management method based on multimodal information fusion provided by the above-mentioned embodiment, and the other technical features of the operating condition prediction and energy management device based on multimodal information fusion are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0145] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0146] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0147] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer program) stored thereon, and the computer-readable program instructions are used to execute the operating condition prediction and energy management method based on multimodal information fusion in the above-mentioned embodiment.

[0148] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0149] The above-mentioned computer-readable storage medium may be included in the operating condition prediction and energy management device based on multimodal information fusion; or it may exist independently without being assembled into the operating condition prediction and energy management device based on multimodal information fusion.

[0150] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the operating condition prediction and energy management device based on multimodal information fusion, the operating condition prediction and energy management device based on multimodal information fusion: receives three-dimensional point cloud data collected in real time by the lidar and two-dimensional image data collected in real time by the camera; extracts and fuses features in the three-dimensional point cloud data and two-dimensional image data to generate a fusion feature vector; obtains offline map data, updates the offline map data in real time according to the fusion feature vector to obtain real-time map data; obtains vehicle historical speed data, obtains a predicted vehicle speed according to the fusion feature vector and the vehicle historical speed data, and performs energy management according to the real-time map data and the predicted vehicle speed.

[0151] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0152] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0153] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0154] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned method for operating condition prediction and energy management based on multimodal information fusion, thereby solving the technical problem of improving the accuracy of operating condition prediction. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the method for operating condition prediction and energy management based on multimodal information fusion provided in the aforementioned embodiments, and are not further elaborated here.

[0155] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned operating condition prediction and energy management method based on multimodal information fusion.

[0156] The computer program product provided in this application can solve the technical problem of improving the accuracy of operating condition prediction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the operating condition prediction and energy management method based on multimodal information fusion provided in the above-mentioned embodiment, and will not be elaborated here.

[0157] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for operating condition prediction and energy management based on multimodal information fusion, characterized in that: The method includes: Receive 3D point cloud data collected by the lidar in real time and 2D image data collected by the camera in real time; Extracting and fusing features from the three-dimensional point cloud data and the two-dimensional image data to generate a fused feature vector; Obtaining offline map data, and updating the offline map data in real time according to the fused feature vector to obtain real-time map data; Acquire historical vehicle speed data, obtain a predicted vehicle speed based on the fused feature vector and the historical vehicle speed data, and perform energy management based on the real-time map data and the predicted vehicle speed.

2. The method according to claim 1, wherein The step of extracting and fusing features from the three-dimensional point cloud data and the two-dimensional image data to generate a fused feature vector comprises: Processing the two-dimensional image data through a convolutional neural network to generate an image feature vector; Processing the three-dimensional point cloud data through a point cloud processing network to generate a point cloud feature vector; The image feature vector and the point cloud feature vector are weightedly spliced ​​and fused to generate the fused feature vector.

3. The method according to claim 1, wherein The step of updating the offline map data in real time according to the fused feature vector to obtain the real-time map data comprises: Obtaining environmental perception information according to the fused feature vector; The road information in the offline map data is updated according to the environmental perception information to obtain real-time map data.

4. The method according to claim 1, wherein The step of obtaining the predicted vehicle speed based on the fused feature vector and the vehicle historical speed data comprises: Processing the fused feature vector and the historical vehicle speed data through a gated recurrent unit neural network to obtain an initial predicted vehicle speed; Generate a state transition probability matrix based on historical vehicle speed data; The initial predicted vehicle speed and the state transition probability matrix are processed by a Markov chain model to obtain a predicted vehicle speed.

5. The method according to claim 4, wherein The step of processing the initial predicted vehicle speed and the state transition probability matrix through a Markov chain model to obtain the predicted vehicle speed includes: determining a current vehicle speed state according to the initial predicted vehicle speed; Obtaining the transition probability from the current vehicle speed state to the next state according to the state transition probability matrix; The predicted vehicle speed is obtained according to the transition probability.

6. The method according to claim 1, wherein The step of performing energy management according to the real-time map data and the predicted vehicle speed includes: Identifying the current operating condition type of the vehicle based on the real-time map data; Building a dynamic energy consumption optimization model according to the operating condition type and the predicted vehicle speed; Obtain the power battery state of charge constraints and maximum output power constraints; Obtaining motor power distribution parameters and braking control parameters according to the dynamic energy consumption optimization model, the power battery state of charge constraint, and the maximum output power constraint; The vehicle power is adjusted according to the motor power distribution parameters and the brake control parameters.

7. The method according to claim 6, wherein The step of constructing a dynamic energy consumption optimization model according to the operating condition type and the predicted vehicle speed includes: When the operating condition type is a normal operating condition, a dynamic energy consumption optimization model is constructed based on the predicted vehicle speed; When the operating condition type is an emergency operating condition, an emergency operating condition penalty factor is obtained, and a dynamic energy consumption optimization model is constructed according to the predicted vehicle speed and the emergency operating condition penalty factor.

8. A working condition prediction and energy management device based on multimodal information fusion, characterized in that: The device comprises: A receiving module is used to receive the three-dimensional point cloud data collected by the lidar in real time and the two-dimensional image data collected by the camera in real time; A fusion module, configured to extract and fuse features from the three-dimensional point cloud data and the two-dimensional image data to generate a fused feature vector; An updating module, configured to obtain offline map data and update the offline map data in real time according to the fused feature vector to obtain real-time map data; The prediction module is used to obtain vehicle historical speed data, obtain a predicted vehicle speed based on the fused feature vector and the vehicle historical speed data, and perform energy management based on the real-time map data and the predicted vehicle speed.

9. A working condition prediction and energy management device based on multimodal information fusion, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the operating condition prediction and energy management method based on multimodal information fusion as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the operating condition prediction and energy management method based on multimodal information fusion according to any one of claims 1 to 7 is implemented.

Images