Internet-of-Vehicles vehicle road multi-device joint modeling system

Abstract

The invention discloses an Internet of Vehicles vehicle road multi-device joint modeling system, and belongs to the technical field of Internet of Vehicles, the system comprises a fixed camera assembly, a data processing unit and a vehicle control unit, the data processing unit processes vehicle image data collected by the fixed camera assembly; the vehicle control unit calculates the optimal driving strategy according to the processing result of the data processing unit, all the road surfaces are necessarily monitored through the fixed cameras in the scene, obstacles in original observation dead corners can be effectively observed in combination with judgment of the camera of the vehicle, the detection precision and reliability of automatic driving are improved, and the driving safety is improved. The problem of observation dead angles of a traditional single sensor is effectively solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of Internet of Vehicles, and particularly relates to a system for vehicle-road multi-device joint modeling. BACKGROUND

[0002] With the rapid development of automatic driving technology, automatic driving systems have been widely applied to various functional robots, industrial transport vehicles and intercity traffic scenarios. At present, automatic driving technology mainly relies on cameras or radars on vehicles to detect the external environment, and judges whether to turn, slow down or stop based on the distance between the front road conditions and the front or side obstacles and the vehicle itself. However, there are many problems in the prior art: 1. Due to the observation dead angle of the vehicle's own camera or radar, and the influence of light and shadow effect, the automatic driving technology often produces misjudgment, which further causes safety hazards. Especially in low-speed scenarios and closed scenarios, the existing automatic driving technology is more limited in application, and it is difficult to effectively cope with complex multiple obstacle environments.

[0003] 2. The existing automatic driving system has an unsatisfactory ranging effect when facing low-pass obstacles, and cannot accurately identify the distance of the obstacles, thereby affecting the driving safety of the vehicle.

[0004] 3. The traditional model-based control strategy and learning-based intelligent decision method has problems such as decision lag, insufficient environmental adaptability and high consumption of computing resources in complex multiple obstacle environments.

[0005] 4. The existing automatic driving technology mainly relies on cameras or radars on vehicles to detect the external environment, lacks comprehensive perception ability of the surrounding environment, and cannot fully support intelligent decision-making.

[0006] Due to the observation dead angle of the vehicle's own camera or radar, and the influence of light and shadow effect, the automatic driving technology often produces misjudgment, which further causes safety hazards.

[0007] In low-speed scenarios and closed scenarios, the existing automatic driving technology is more limited in application. Especially in complex multiple obstacle environments, the traditional model-based control strategy and learning-based intelligent decision method is difficult to effectively cope with, showing problems such as decision lag, insufficient environmental adaptability and high consumption of computing resources. In addition, the existing automatic driving system has an unsatisfactory ranging effect when facing low-pass obstacles, and cannot accurately identify the distance of the obstacles, thereby affecting the driving safety of the vehicle.

[0008] In order to solve these problems, a system is needed to supplement the existing automatic driving technology. SUMMARY

[0009] In order to achieve the above object, the technical scheme of the present application is as follows: The present application aims to overcome the shortcomings in the prior art and provides a system for vehicle-to-everything (V2X) vehicle-road multi-device joint modeling, which comprises a vehicle, a fixed camera assembly, a data processing unit and a vehicle control unit.

[0010] Preferably, the vehicle is provided with a vehicle body front windshield, a vehicle body side mirror and a vehicle body rearview mirror, wherein a front-view camera is arranged in the vehicle body front windshield, a side-view camera is arranged in the vehicle body side mirror, and a rear-view camera is arranged in the vehicle body rearview mirror.

[0011] Preferably, the fixed camera assembly comprises a plurality of fixed cameras arranged along the road surface around the vehicle for all-around monitoring of the environment around the vehicle.

[0012] Preferably, the data processing unit comprises an image processing module, a radar data processing module, an ultrasonic data processing module and a fusion processing module.

[0013] Preferably, the vehicle control unit comprises a decision module and an execution module.

[0014] Preferably, the vehicle is also provided with a global positioning system (GPS), an inertial measurement unit (IMU) and a vehicle-mounted communication module for providing vehicle position information, speed information and environmental information.

[0015] Preferably, the workflow of the system is as follows: during vehicle driving, the front-view camera, side-view camera and rear-view camera collect environmental images in real time; the fixed camera assembly collects monitoring images of the environment around the vehicle in real time; the laser radar, millimeter wave radar, angle radar and ultrasonic radar scan the environment around the vehicle in real time; the GPS, IMU and vehicle-mounted communication module acquire vehicle state information in real time; the data processing unit processes the collected data; the fusion processing module fuses various types of data; the decision module calculates the optimal driving strategy of the vehicle according to the fused data combined with the vehicle's own state information; and the execution module controls the vehicle to perform corresponding actions according to the decision result.

[0016] Preferably, the fixed camera assembly further comprises a pan-tilt adjusting mechanism for adjusting the pitch angle and azimuth angle of the fixed camera, so as to ensure that the fixed camera can cover the maximum range around the vehicle. The fixed camera adopts an infrared light supplement device for providing light supplement at night or in poor light conditions.

[0017] Preferably, the vehicle control unit further comprises a safety evaluation module which calculates the safety of vehicle driving in real time according to the current vehicle speed, acceleration, obstacle distance and other factors. When the safety evaluation module detects potential danger, it timely triggers an alarm prompt and automatically adjusts the vehicle driving strategy.

[0018] Preferably, the system further comprises a learning optimization module which learns and optimizes the decision algorithm by itself according to the environmental changes and driving strategy execution in the vehicle driving process, so as to improve the intelligent level and the ability to adapt to complex environments of the system.

[0019] Compared with the prior art, the present application has the following beneficial effects: 1. By utilizing the fixed camera in the scene to monitor all road surfaces as necessary, combined with the judgment of the vehicle's own camera, obstacles in the original observation dead angle can be effectively observed, the detection accuracy and reliability of automatic driving are improved, and the problem of observation dead angle of traditional single sensor is effectively solved; 2. Based on the auxiliary observation function of the fixed camera, it can be accurately judged whether there is a real dangerous obstacle in the light and shadow change, effectively avoiding the misjudgment caused by light and shadow effect, and improving the decision accuracy of the automatic driving system under complex lighting conditions; 3. Through the synchronous identification mechanism, the monitoring result of the fixed camera is combined with the judgment of the vehicle's own camera, realizing multi-angle and all-around environmental perception, and improving the adaptability of the automatic driving system to complex multiple obstacle environments; 4. The time verification mechanism is adopted to ensure that the data synchronization efficiency is within 200ms, which can guarantee the real-time performance and response speed of the automatic driving system, effectively solving the problem of decision lag in the prior art; 5. Through the cooperative work of the dual cameras, the distance of the obstacle can be accurately identified, the ranging accuracy of the automatic driving system for low-pass obstacles is improved, and thus the driving safety of the vehicle is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A workflow diagram of the multi-device joint modeling system of the present application. DETAILED DESCRIPTION

[0021] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application. EMBODIMENT

[0022] The present embodiment provides a vehicle-to-everything vehicle-road multi-device joint modeling system, which comprises a vehicle, a fixed camera assembly, a data processing unit and a vehicle control unit.

[0023] Further, the vehicle is provided with a vehicle body front windshield, a vehicle body side mirror and a vehicle body rearview mirror, wherein the vehicle body front windshield is provided with a front-view camera, the vehicle body side mirror is provided with a side-view camera, and the vehicle body rearview mirror is provided with a rear-view camera.

[0024] Further, the fixed camera assembly comprises a plurality of fixed cameras arranged along the road surface around the vehicle for omnidirectional monitoring of the environment around the vehicle. Among them, the fixed camera adopts a long-focus lens design, and two fixed cameras are respectively located on the road edges or fixed gantries on both sides of the vehicle body, for reducing the monitoring blind area and ensuring that the overall road view can be detected.

[0025] Further, the data processing unit comprises an image processing module, a radar data processing module, an ultrasonic data processing module and a fusion processing module. Among them, the image processing module processes the image data collected by the front-view camera, the side-view camera, the rear-view camera and the fixed camera on the roadside; the radar data processing module processes the data collected by the laser radar, the millimeter wave radar and the angle radar; the ultrasonic data processing module processes the data collected by the ultrasonic radar; and the fusion processing module fuses the data processed by the above modules.

[0026] Further, the vehicle control unit comprises a decision module and an execution module. The decision module calculates the best driving strategy of the vehicle according to the data fusion result and in combination with the vehicle's own state information; and the execution module controls the driving, steering, deceleration and brake execution mechanisms of the vehicle according to the decision result.

[0027] Further, the vehicle is also provided with a global positioning system (GPS), an inertial measurement unit (IMU), and a vehicle communication module, for providing vehicle position information, speed information, and environmental information.

[0028] Further, the workflow of the system is as follows: Step one, during vehicle driving, the front-view camera, side-view camera, and rear-view camera collect environmental images in real time; Step two, the fixed camera assembly collects monitoring images of the environment around the vehicle in real time; Step three, the laser radar, millimeter wave radar, angle radar, and ultrasonic radar scan the environment around the vehicle in real time; Step four, the GPS, IMU, and vehicle communication module acquire vehicle state information in real time; the data processing unit processes the collected data; Step five, the fusion processing module fuses various types of data; Step six, the decision module calculates the best driving strategy of the vehicle based on the fused data and the vehicle's own state information; the execution module controls the vehicle to perform corresponding actions according to the decision result.

[0029] Further, the fixed camera assembly further includes a gimbal adjustment mechanism for adjusting the pitch angle and azimuth angle of the fixed camera, ensuring that the fixed camera can cover the maximum range around the vehicle. The fixed camera uses an infrared light supplement device to provide light supplementation at night or in poor light conditions.

[0030] Further, the vehicle control unit further includes a safety evaluation module, which calculates the safety of vehicle driving in real time based on factors such as current vehicle speed, acceleration, obstacle distance, etc. When the safety evaluation module detects potential danger, it timely triggers an alarm prompt and automatically adjusts the vehicle driving strategy.

[0031] Further, the system further includes a learning optimization module, which learns and optimizes the decision algorithm based on environmental changes and driving strategy execution during vehicle driving, thereby improving the intelligent level and ability to adapt to complex environments of the system. Embodiment

[0032] The embodiment provides an automatic driving auxiliary system based on double cameras, including a vehicle, a fixed camera assembly, a data processing unit, and a vehicle control unit.

[0033] Further, the vehicle is provided with a front windshield, side mirrors, and rearview mirrors, wherein the front windshield is provided with a front-view camera, the side mirrors are provided with side-view cameras, and the rearview mirrors are provided with rear-view cameras.

[0034] Further, the fixed camera assembly comprises a plurality of fixed cameras arranged along the road surface around the vehicle for omnidirectional monitoring of the environment around the vehicle. Among them, the fixed camera adopts a long-focus lens design, and two fixed cameras are respectively located on both sides of the vehicle body to reduce the monitoring blind area and ensure that the rear wheels of the corresponding side vehicle body can be detected.

[0035] Further, the data processing unit comprises an image processing module, a radar data processing module, an ultrasonic data processing module and a fusion processing module. Among them, the image processing module processes the image data collected by the front-view camera, side-view camera and rear-view camera; the radar data processing module processes the data collected by the laser radar, millimeter wave radar and angle radar; the ultrasonic data processing module processes the data collected by the ultrasonic radar; and the fusion processing module fuses the data processed by the above modules.

[0036] Further, the vehicle control unit comprises a decision module and an execution module. The decision module calculates the best driving strategy of the vehicle according to the data fusion result combined with the vehicle's own state information; and the execution module controls the driving, steering, deceleration and brake execution mechanisms of the vehicle according to the decision result.

[0037] Further, the vehicle is also provided with a global positioning system (GPS), an inertial measurement unit (IMU) and a vehicle-mounted communication module for providing vehicle position information, speed information and environmental information.

[0038] Further, the working process of the system is as follows: Step one, during the driving process of the vehicle, the front-view camera, side-view camera and rear-view camera collect environmental images in real time; Step two, the fixed camera assembly collects monitoring images of the environment around the vehicle in real time; Step three, the laser radar, millimeter wave radar, angle radar and ultrasonic radar scan the environment around the vehicle in real time; Step four, the GPS, IMU and vehicle-mounted communication module acquire vehicle state information in real time; and the data processing unit processes the collected data; Step five, the data collected by various sensors of the vehicle are fused in the fusion processing module; Step six, the fixed camera assembly collects monitoring images of the vehicle in real time, focusing on fixed light and shadow effects (such as the dark part before the tunnel, the light belt in the dragon gate in the night, etc.) and 360-degree obstacle recognition around the vehicle. This part of data is synchronized with the vehicle based on the edge network or 5G network, and enters the decision module.

[0039] Step seven, the decision module according to the fused vehicle itself and fixed camera assembly supplementary data, combined with the vehicle state information, delete the obstacles around the vehicle body, while calculating the best driving strategy of the vehicle; the execution module according to the decision result, control the vehicle to execute corresponding action.

[0040] Further, the decision module of the vehicle control unit specifically comprises: The decision module includes a decision core, an environment model construction unit, a prediction unit and an evaluation unit. Among them, the decision core is used to make decision calculation according to the environment model constructed by the environment model construction unit, combined with the prediction result of the prediction unit, using machine learning algorithm, to generate the best driving strategy of the vehicle; the environment model construction unit is used to construct the three-dimensional model of the environment around the vehicle according to the fixed camera, the vehicle body camera and the sensor data; the prediction unit is used to predict the future motion trajectory of the vehicle and the change of the surrounding environment; the evaluation unit is used to evaluate the rationality of the decision according to the prediction result and the actual execution situation.

[0041] Further, the decision module further comprises a diversity strategy generation unit, which is used to generate a plurality of optional decision schemes, and assign a corresponding evaluation score to each decision scheme for comparison and selection by the decision core.

[0042] Further, in the specific implementation process, the decision module can also dynamically adjust the weight coefficient of the decision strategy according to the driving speed of the vehicle, the road type, the weather condition and other factors.

[0043] Further, the fusion processing module of the data processing unit specifically comprises: The fusion processing module adopts a multi-sensor data fusion algorithm based on deep learning to effectively fuse multi-sensor data such as laser radar, millimeter wave radar, ultrasonic radar, front-view camera, side-view camera and rear-view camera. The specific steps include: 1) receiving each sensor data and performing standardization processing; 2) using a deep learning network to extract features from each sensor data, including texture features, shape features, size features, motion features, etc. 3) constructing a correlation model of multi-sensor data to describe the relationship between different sensor data; 4) calculating the confidence of each sensor data using Bayesian inference algorithm; 5) calculating the final fusion result according to the confidence and weight of each sensor data; 6) denoising and smoothing the fusion result to obtain the final environment perception result.

[0044] Further, the deep learning network adopts a ResNet-18 network structure, is trained to convergence through a large amount of training data, and is used to extract features of the multi-sensor data. The correlation model adopts a linear regression model, and a mapping relationship between different sensor data is obtained through learning of training samples. The Bayesian inference algorithm is implemented by using a Monte Carlo method, a plurality of samples are randomly generated, and the likelihood of each sensor data is calculated through the samples. A weight calculation formula is as follows: Weight = (confidence * weight base) / (confidence * weight base + (1-confidence) * (1-weight base), wherein the confidence is a confidence of the sensor data, and the weight base is a preset weight of the sensor data.

[0045] It should be noted that the above content only illustrates the technical idea of the present application, and cannot limit the protection scope of the present application. For ordinary skilled persons in the art, a number of improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements fall within the protection scope of the claims of the present application.

Claims

1. A system for joint modeling of multiple vehicle-road devices in a vehicle-to-everything (V2X) network, characterized in that, The system comprises a fixed camera assembly, a data processing unit and a vehicle control unit, the data processing unit processes vehicle image data collected by the fixed camera assembly, and the vehicle control unit calculates an optimal driving strategy based on the result processed by the data processing unit. 2.The system of claim 1, wherein, The data processing unit comprises an image processing module, a radar data processing module, an ultrasonic data processing module and a fusion processing module, wherein the image processing module processes image data collected by the camera, the radar data processing module processes data collected by laser radar, millimeter wave radar and angle radar, the ultrasonic data processing module processes data collected by ultrasonic radar, and the fusion processing module fuses data processed by the above modules. 3.The system of claim 1, wherein, The vehicle control unit comprises a decision module, which calculates the optimal driving strategy of the vehicle based on the data fusion result and the vehicle state information. 4.The system of claim 2, wherein, The vehicle control unit further comprises an execution module, which controls the driving, steering, deceleration and brake execution mechanism of the vehicle according to the decision result. 5.The system of claim 1, wherein, The system is also provided with a global positioning system, an IMU and a vehicle communication module to provide vehicle position information, speed information and environmental information. 6.The system of claim 1, wherein, The fixed camera assembly comprises a plurality of fixed cameras arranged along the road surface around the vehicle for omnidirectional monitoring of the vehicle environment, wherein the fixed cameras are designed with long focal length lenses, and two fixed cameras are respectively located on the road edges or fixed gantries on both sides of the vehicle body. 7.The system of claim 1, wherein, The vehicle control unit further comprises a safety evaluation module, which calculates the safety of vehicle driving in real time according to the current vehicle speed, acceleration, obstacle distance and other factors, and timely triggers an alarm prompt and automatically adjusts the vehicle driving strategy when potential danger is detected. 8.The system of claim 1, wherein, The system comprises a learning optimization module, which learns and optimizes the decision algorithm based on the environmental changes and driving strategy execution in the vehicle driving process.

Images