Self-adaptive perception fusion driving method and device, storage medium and vehicle

Through the adaptive sensing fusion driving method, the information fusion of on-board sensors and V2X data and the driver feedback adjustment of weight values ​​is solved, and the existing technology cannot effectively detect dangerous situations within the non-sight range range is improved.

CN119058725AActive Publication Date: 2024-12-03BYD CO LTD +1

Patent Information

Application Number
CN202310640620.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03
Estimated Expiration
2043-05-31

AI Technical Summary

Technical Problem

The existing advanced driving assistance system cannot effectively detect and identify potential hazards outside the range of sight, resulting in driving safety hazards. Especially in the context of the development of Internet of Vehicles and intelligent connected vehicles, how vehicles can be used to perceive the environment based on a variety of detection information. Improving driving safety has become a challenge.

Method used

Adaptive sensing fusion driving method is adopted, by obtaining multiple object detection information of on-board sensor data and V2X data, information fusion is carried out according to the current weight value, and weight value is adjusted according to the driver feedback, and finally generating the vehicle pass plan and timely discovering it. Hazardous situations within the range of sight or non-detection.

Benefits of technology

This method can fully extract the effective amount of information from multiple data sources, improve driving safety, reduce the probability of driver being misled, enhance the ability to detect dangerous situations within the non-sight range, and improve driving safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119058725A_ABST
    Figure CN119058725A_ABST
Patent Text Reader

Abstract

The invention provides an adaptive perception fusion driving method and device, a storage medium and a vehicle. The adaptive perception fusion driving method comprises the following steps: for a to-be-detected target, obtaining multiple pieces of target detection information obtained based on vehicle-mounted sensor data and V2X data; performing information fusion on the multiple pieces of target detection information based on the weight obtained by current adaptive adjustment to obtain an information fusion result; based on the information fusion result, determining a target detection result according to a preset judgment rule; and generating a vehicle passing scheme based on the target detection result. According to the method, information fusion is carried out on the multiple pieces of target detection information obtained through the vehicle-mounted sensor data and the V2X data, effective information in multiple kinds of data is fully extracted, dangerous conditions in a non-line-of-sight or non-detection range can be found in time, and the driving safety is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of vehicles, and more specifically to an adaptive perception fusion driving method, device, storage medium and vehicle. Background Art

[0002] The advanced driver assistance system uses various sensors installed on the car to sense the surrounding environment at any time while the car is driving, collect data, identify, detect and track static and dynamic objects, and combine navigation map data to perform systematic calculations and analysis, so as to better help the driver understand the external environment conditions and enable some applications such as assisted parking and vehicle speed control.

[0003] At present, advanced driver assistance systems usually use cameras and radars to detect the surrounding environment, but they only have the ability to detect within the line of sight, which brings certain safety risks to the safe driving of vehicles. With the development of vehicle networking and intelligent connected vehicles, the advantages of vehicle networking (Vehicle to Everything, V2X) technology are becoming increasingly obvious. V2X technology can realize detection and scanning beyond the line of sight and two-way communication between vehicles and road networks, which can provide basic functional support for vehicle driving safety.

[0004] V2X communication technology includes V2I (Vehicle to Infrastructure) and V2V (Vehicle to Vehicle), which can provide a variety of detection information beyond the line of sight for vehicles in motion. In this case, how vehicles can perceive the environment based on a variety of detection information to obtain correct perception information and improve driving safety has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application is proposed in consideration of the above problems. The present application provides an adaptive perception fusion driving method, device, storage medium and vehicle, which can fully extract the effective information from multiple data sources and timely detect dangerous situations in non-line-of-sight or non-detection range, which can greatly improve driving safety.

[0006] According to one aspect of the present application, an adaptive perception fusion driving method is provided, the method comprising:

[0007] For the target to be detected, multiple target detection information is obtained based on the vehicle sensor data and the V2X data respectively;

[0008] Based on the weights obtained by the current adaptive adjustment, the plurality of target detection information are fused to obtain an information fusion result;

[0009] Based on the information fusion result, determining the target detection result according to the preset judgment rules;

[0010] A traffic plan for the vehicle is generated based on the target detection result.

[0011] In one embodiment of the present application, a device that collects the vehicle-mounted sensor data or receives the V2X data is used as an information source, and the method for adaptively adjusting the weight includes:

[0012] For each information source, determining a current weight value of the target detection information obtained based on the vehicle-mounted sensor data or the V2X data, including: if the target detection information indicates that the target is detected, the current weight value is a preset value greater than 0 and less than 1; if the target detection information indicates that the target is not detected, the current weight value is 0;

[0013] Adjusting the current weight value based on feedback information from the driver of the vehicle to obtain an adjusted weight value;

[0014] The adjusted weight value is used as a preset value corresponding to the target detection information obtained through the information source during the next information fusion.

[0015] In one embodiment of the present application, the weight adjustment of the current weight value based on the feedback information of the driver of the vehicle includes:

[0016] For each information source, when the feedback information indicates that the target detection in the target detection information is correct, the current weight value is increased; when the feedback information indicates that the target detection in the target detection information is incorrect, the current weight value is decreased.

[0017] In one embodiment of the present application, the information fusion of the plurality of target detection information to obtain the information fusion result includes:

[0018] Obtaining current weight values ​​corresponding to a plurality of target detection information;

[0019] The multiple target detection information are normalized and weighted averaged according to their corresponding current weight values ​​to obtain an information fusion result.

[0020] In one embodiment of the present application, when the target to be detected is an obstacle avoidance target, the vehicle-mounted sensor data includes one or more of position data, speed data, and video image data, and the V2X data includes V2V data; when the target to be detected is a driving guidance sign, the vehicle-mounted sensor data includes video image data, and the V2X data includes V2V data and V2I data.

[0021] In one embodiment of the present application, the preset value of the current weight value corresponding to the V2V data is smaller than the preset value corresponding to the vehicle-mounted sensor data, and the preset value of the current weight value corresponding to the V2V data is smaller than the preset value corresponding to the V2I data.

[0022] In one embodiment of the present application, determining the target detection result based on the information fusion result according to a preset judgment rule includes:

[0023] When the information fusion result is greater than or equal to a first preset threshold, determining that the target detection result is that the current target is detected;

[0024] When the information fusion result is less than or equal to a second preset threshold, it is determined that the target detection result is that the current target is not detected.

[0025] When the information fusion result is less than the first preset threshold and greater than the second preset threshold, it is determined that the target detection result is that the current target is in an uncertain state.

[0026] In one embodiment of the present application, the method further includes:

[0027] The target detection result and the vehicle's traffic plan are broadcasted in the Internet of Vehicles.

[0028] In one embodiment of the present application, the method further includes:

[0029] Acquiring map information, wherein the map information includes location information of driving guidance signs on the driving road;

[0030] Based on the position information of the driving guidance sign, determining the distance between the vehicle and the next driving guidance sign;

[0031] Determining a target to be detected based on the distance includes:

[0032] When the distance is greater than a preset distance threshold, the obstacle avoidance target is taken as a target to be detected;

[0033] When the distance is less than or equal to a preset distance threshold, the driving guidance sign is taken as a target to be detected.

[0034] In one embodiment of the present application, when the target to be detected is an obstacle avoidance target and the vehicle's traffic plan is emergency avoidance, the method further includes:

[0035] Generate alarm trigger information to trigger the alarm device to send out an alarm signal, wherein the alarm signal includes one or more of a sound signal, a light signal, and a tactile signal.

[0036] According to the second aspect of the present application, an adaptive perception fusion driving device based on V2X and vehicle-mounted sensors is provided, the device comprising a memory and a processor, the memory storing a computer program run by the processor, and when the computer program is run by the processor, the device equipped with the processor executes the above-mentioned adaptive perception fusion driving method.

[0037] According to a third aspect of the present application, a storage medium is provided, on which a computer program is stored. The computer program runs on a computer, and when the computer program is running, the computer executes the above-mentioned adaptive perception fusion driving method.

[0038] According to a fourth aspect of the present application, a vehicle is provided, the vehicle comprising the above-mentioned adaptive perception fusion driving device or the above-mentioned storage medium.

[0039] The present application fuses multiple target detection information obtained based on multiple data, fully extracts the effective information existing in multiple data sources, and can promptly detect dangerous situations in non-line-of-sight or non-detection range, thereby greatly improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0041] Figure 1 is a schematic block diagram of an electronic device for implementing the adaptive perception fusion driving method and device according to an embodiment of the present invention;

[0042] Figure 2 is a schematic flow chart of an adaptive perception fusion driving method according to an embodiment of the present application;

[0043] Figure 3 is a schematic flow chart of a weight adaptive adjustment method in a traffic light monitoring scenario at an intersection according to the first embodiment of the present application;

[0044] Figure 4 is a schematic flow chart of a weight adaptive adjustment method in an obstacle avoidance target monitoring scenario according to the second embodiment of the present application;

[0045] Figure 5 is a schematic block diagram of a method for adjusting the current weight value and fusing information based on the feedback information of the driver of the vehicle in a T scenario according to an embodiment of the present application;

[0046] Figure 6 is a schematic flow chart of a method for adjusting the current weight value and fusing information based on the feedback information of the driver of the vehicle in a P scenario according to an embodiment of the present application;

[0047] Figure 7 It is a schematic flowchart of the adaptive perception fusion driving method according to the third embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0049] When a vehicle is driving on a city road, the driver needs to understand various information to ensure safe driving, such as the situation of obstacle avoidance targets such as pedestrians / bicycles around the vehicle, the situation of driving guidance signs such as traffic lights ahead, etc. Although current cameras, millimeter-wave radars, lidars and other sensors can inform the driver of the detected warning information through sound, images, etc., there are safety hazards in the detection blind spots of the vehicle's sensors. Based on the shared information between vehicles and between vehicles and roads using V2X technology, this application proposes a driving solution for adaptive perception fusion when the vehicle is driving on urban roads. It performs environmental perception based on multiple detection information to obtain correct perception information, which can promptly detect dangerous situations within non-line-of-sight or non-detection ranges and improve driving safety.

[0050] First, refer to Figure 1 An example electronic device 100 for implementing the adaptive perception fusion driving method and apparatus according to an embodiment of the present invention is described.

[0051] like Figure 1 As shown, the electronic device 100 includes one or more processors 101, one or more storage devices 102, an input device 103, and an output device 104, and these components are interconnected through a bus system 105 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The control device may also have other components and structures as required.

[0052] The processor 101 can be a microcontroller unit (MCU), a central processing unit (CPU), a digital signal processor (DSP), a single-chip microcomputer and an embedded device, or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0053] The storage device 102 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 101 may run the program instructions to implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0054] The input device 103 may be a device used by a user to input instructions, and may include a microphone or a touch screen or other input devices.

[0055] The output device 104 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0056] Next, refer to Figure 2 To describe the adaptive perception fusion driving method based on V2X and vehicle-mounted sensors according to an embodiment of the present application.

[0057] like Figure 2 As shown, the adaptive perception fusion driving method provided by the present application includes:

[0058] In step S210, for a target to be detected, multiple target detection information obtained based on the vehicle-mounted sensor data and the V2X data are obtained.

[0059] Here, the target to be detected may include an obstacle avoidance target and / or a driving guidance mark, and the obstacle avoidance target may be a dynamic object or a static object, wherein the dynamic object may be, for example, a pedestrian, a motor vehicle or a non-motor vehicle; and the static object may be, for example, a road edge, a tree, a road sign, a lane or other obstacles. The driving guidance mark may be a traffic light, a traffic sign, a road marking, etc.

[0060] The sensor data in this application can be collected by vehicle-mounted sensors, which may include image sensors, radar sensors, positioning sensors, wind speed sensors, light sensors, infrared sensors, etc. Position data can be obtained by radar sensors and / or positioning sensors, and speed data can also be obtained by radar sensors and / or positioning sensors. Video image data can be obtained by image sensors. The method for obtaining sensor data can be specifically determined according to actual conditions, and the method for obtaining sensor data is not specifically limited in this application.

[0061] Here, image sensors, such as vehicle forward cameras, vehicle side cameras, and vehicle rear cameras, the environmental information collected by the camera is image data within the camera's field of view. Radar sensors, such as ultrasonic radars, millimeter-wave radars, laser radars, etc., can also be other types of radar sensors, which are not specifically limited here. The positioning sensor can be a global positioning system (GPS), a global navigation satellite system (GNSS), or an inertial navigation system (INS). The environmental information collected by the positioning sensor is the vehicle's location information, the environmental information collected by the wind speed sensor is wind speed, wind direction, and other information, and the environmental information collected by the light sensor is light intensity.

[0062] Accordingly, the target detection information is obtained based on the data collected by the sensor according to a preset target detection algorithm. Those skilled in the art can flexibly set the target detection algorithm according to actual needs, and this application does not make any specific limitations.

[0063] The V2X data in this application may include data directly obtained through the vehicle to everything (V2X), and may also include information data of corresponding vehicles or driving guidance signs obtained by the vehicle through the Internet of Vehicles (IoV) or device to device (D2D) communication.

[0064] For example, when V2X data is data obtained through the Internet of Vehicles, the V2X data may include V2V data and V2I data, and the V2V data and V2I data may include target detection information and other information. Specifically, V2X data may be received through a V2X communication module. The V2X communication module includes a V2V module for vehicle-to-vehicle communication with other vehicles and a V2I module for vehicle-to-infrastructure communication with roadside equipment; the V2X communication module sends and receives in the form of messages, and the message format meets the V2X communication standard protocol.

[0065] In step S220, based on the weights obtained by the current adaptive adjustment, the plurality of target detection information are fused to obtain an information fusion result.

[0066] It should be understood that when a plurality of target detection information are fused, the target detection information can be represented by numbers that are easy to normalize.

[0067] For example, for the scenario where the obstacle avoidance target is a vulnerable traffic participant (Pedestrian, P) (hereinafter referred to as the P scenario) and the driving guidance sign is the traffic light (Traffic-light, T) of the current road section (hereinafter referred to as the T scenario), the targets to be detected are vulnerable traffic participants and traffic lights, respectively. In the P scenario, 1 represents the presence of vulnerable traffic participants nearby, and 0 represents the absence of vulnerable traffic participants nearby. The value between 0 and 1 represents the probability of there being vulnerable traffic participants nearby. Similarly, in the T scenario, 1 represents the current road section has a green light, and 0 represents the current road section has no green light. The value between 0 and 1 represents the probability of the current road end having a green light.

[0068] It should be noted that each target detection information has a corresponding weight obtained by current adaptive adjustment. During the adaptive adjustment process, the sum of all current weight values ​​may be equal to 1 or may not be equal to 1.

[0069] In step S230, based on the information fusion result, the target detection result is determined according to a preset judgment rule.

[0070] Specifically, when the target detection information is represented by numbers that are easy to normalize, the information fusion result is a value between 0 and 1.

[0071] In step S240, a traffic plan for the vehicle is generated based on the target detection result.

[0072] In the present application, the specific method for generating the vehicle passage plan based on the target detection result can adopt the existing method, which can be set according to the actual situation, and the present application does not make specific limitations on this.

[0073] In an embodiment of the present application, by fusing multiple target detection information obtained based on multiple data, the effective information existing in multiple data sources is fully extracted, and when certain information is inaccurate or has a large offset, the fusion with other information can effectively offset the impact of the offset; even if there is a problem with a certain type of information, the probability of the driver being misled is greatly reduced, thereby greatly improving driving safety.

[0074] According to one embodiment of the present application, when the obstacle avoidance target is a vulnerable traffic participant, the on-board sensor data includes millimeter-wave radar data and video image data, and the V2X data includes V2V data; when the target to be detected is a driving guidance sign, the on-board sensor data includes video image data, and the V2X data includes V2V data and V2I data.

[0075] Next, refer to Figure 3 and Figure 4 To describe the adaptive adjustment method of weights according to an embodiment of the present application.

[0076] The adaptive adjustment method of weights includes: weight initialization at the first use and adaptive adjustment of weights at other times.

[0077] Figure 3 is a schematic flow chart of a weight adaptive adjustment method in a traffic light monitoring scenario at an intersection according to the first embodiment of the present application, such as Figure 3 As shown, in this embodiment, various vehicle sensor data and V2X data are sent to the program for weight adaptive adjustment in the form of messages, specifically including V2I signal light phase and timing (Signal Phase and Timing message, SPAT message), messages about vulnerable traffic participants obtained based on image processing detection, and V2V messages of traffic lights on the current road section. The weight adaptive adjustment method of this embodiment includes:

[0078] S301: If a V2I SPAT message (hereinafter referred to as V2I message) from a roadside unit that is filtered according to the current road section traffic light ID exists, its initial message weight can be set to 0.4; otherwise, it is fixed to 0 and does not participate in the subsequent adaptive module weight adjustment and normalization calculation. The initial message weight can be customized.

[0079] S302: If the message about vulnerable traffic participants (hereinafter referred to as camera message) obtained by the on-board binocular camera based on image processing detection exists, its initial message weight can be set to 0.4; otherwise, it is fixed to 0 and does not participate in the subsequent adaptive module weight adjustment and normalization calculation. The initial message weight can be customized. If the camera message has been initialized or updated in the P scene before the T scene is initialized, it is initialized to the corresponding latest updated value in the P scene.

[0080] S303: If there is a V2V message about the current road section traffic light from other nearby vehicles (hereinafter referred to as V2V message), its initial message weight can be set to 0.2; otherwise, it is fixed to 0 and does not participate in the subsequent adaptive module weight adjustment and normalization calculation. If the V2V message has been initialized or updated in the P scene before the T scene is initialized, it is initialized to the corresponding latest updated value in the P scene.

[0081] If the camera message weight and V2V message weight have not been initialized or updated in the P scenario, the initial weight ratio of the three messages is as follows: V2I: camera: V2V = 2:2:1; if the camera message weight and V2V message weight have been updated in the P scenario, it is possible that the sum of the three message weights in the T scenario is not 1. In this case, there is no need to perform normalization adjustment in initialization, and the subsequent message weight adaptive adjustment module will perform normalization.

[0082] S304: Generate T scene fusion message m according to the initial value or the weight adjusted after the driver feedback information T The message weight adaptive adjustment module is used to adaptively adjust the credibility / weight of the valid message type based on the driver's feedback information. The specific steps of the module will be described later.

[0083] Since each adjustment is performed after S310, the initial or current weight value is adjusted based on the information fed back by the driver of the vehicle, and the adjusted weight value is used as the preset value corresponding to the target detection information obtained by the vehicle-mounted sensor during the next target detection.

[0084] Here, the current weight value refers to the weight value after adjusting the initial value.

[0085] S305, S306: merge the T scene message m obtained in S304 T A determination is made as to whether it is less than 0.4. If the result is "yes", the process proceeds to S306, and the message fusion decision is "green light ahead, normal driving"; if not, the process proceeds to S307.

[0086] S307, S308: Determine whether the T scene fusion message obtained in S304 is greater than or equal to 0.6. If the result is "yes", enter S308, and the message fusion decision is "slow down / stop for red light ahead"; otherwise, directly enter S309.

[0087] The threshold used for judgment here can be set according to specific circumstances, and this embodiment does not specifically limit the size of the threshold.

[0088] S309: The driving strategy decisions obtained in S306 and S308 need to be provided to the driver so that he can decide whether to accept the decision and take corresponding actions; for the input of S307, the driver decides whether to slow down / stop.

[0089] S310: Feedback the avoidance decision results of all current valid message sources to the driver, and the driver determines whether each message source is correct or not, and feeds back to the message weight adaptive adjustment module.

[0090] It should be noted that the initial weight value can be set according to actual conditions, and this embodiment does not specifically limit the size of the initial weight value.

[0091] Figure 4 is a schematic flow chart of a method for adaptively adjusting weights in a vulnerable traffic participant monitoring scenario according to the second embodiment of the present application, such as Figure 4 As shown, in this embodiment, various vehicle sensor data and V2X data are sent to the program for adaptive weight adjustment in the form of messages, specifically including messages about vulnerable traffic participants obtained by millimeter wave radar processing, messages about vulnerable traffic participants obtained by vehicle-mounted binocular cameras based on image processing detection, and V2V messages about vulnerable traffic participants from other vehicles. The method for adaptive weight adjustment in this embodiment includes:

[0092] S401: If the message about vulnerable traffic participants obtained from millimeter wave radar processing (hereinafter referred to as millimeter wave radar message) exists, its initial message weight is set to 0.4; otherwise, it is fixed to 0 and does not participate in the subsequent adaptive module weight adjustment and normalization calculation. The initial message weight can be customized.

[0093] S402: If the message about vulnerable traffic participants (hereinafter referred to as camera message) obtained by the on-board binocular camera based on image processing detection exists, its initial message weight is set to 0.4; otherwise, it is fixed to 0 and does not participate in the subsequent adaptive module weight adjustment and normalization calculation. The initial message weight can be customized. If the camera message has been initialized or updated in the T scene before the P scene is initialized, it is initialized to the corresponding latest updated value in the T scene.

[0094] S403: If there is a V2V message about a vulnerable traffic participant from other vehicles (hereinafter referred to as V2V message), its initial message weight is set to 0.2; otherwise, it is fixed to 0 and does not participate in the subsequent adaptive module weight adjustment and normalization calculation. If the V2V message has been initialized or updated in the T scene before the P scene is initialized, it is initialized to the latest updated value corresponding to the T scene.

[0095] If the camera message weight and V2V message weight have not been initialized or updated in the T scenario, the initial weight ratio of the three messages is as follows: millimeter wave radar: camera: V2V = 2:2:1; if the camera message weight and V2V message weight have been updated in the T scenario, it may cause the sum of the three message weights in the P scenario to be different from 1. In this case, it is not necessary to make normalization adjustments in initialization, and the subsequent message weight adaptive adjustment module will make normalization processing.

[0096] S404: Generate P scene fusion message m according to the initial value or the weight adjusted after the driver feedback information P The message weight adaptive adjustment module is used to adaptively adjust the credibility / weight of the valid message type based on the driver's feedback information. The specific steps will be described later.

[0097] Since each adjustment is performed after S410, the initial or current weight value is adjusted based on the feedback information from the driver of the vehicle, and the adjusted weight value is used as the preset value corresponding to the target detection information obtained by the vehicle-mounted sensor during the next target detection.

[0098] Here, the current weight value refers to the weight value after adjusting the initial value.

[0099] S405, S406: merging the P scene message m obtained in S204 P A determination is made as to whether it is less than 0.4. If the result is "yes", the process proceeds to S206, and the message fusion decision is "normal driving"; otherwise, the process proceeds to S207.

[0100] S407, S408: Determine whether the P scene fusion message obtained in S204 is greater than or equal to 0.6. If the result is "yes", enter S208, and the message fusion decision is "slow down to avoid"; otherwise, directly enter S209.

[0101] The threshold used for judgment here can be set according to specific circumstances, and this embodiment does not specifically limit the size of the threshold.

[0102] S409: The driving strategy decisions obtained in S406 and S408 need to be provided to the driver so that he can decide whether to accept the decision and take corresponding actions; for the input of S207, the driver decides whether to slow down and avoid.

[0103] S410: Feedback the avoidance decision results of all current valid message sources to the driver, and the driver determines whether each message source is correct or not, and feeds back to the message weight adaptive adjustment module.

[0104] According to an embodiment of the present application, a method for adjusting the current weight value based on feedback information from the driver of the vehicle using a device that collects the vehicle-mounted sensor data or receives the V2X data as an information source specifically includes:

[0105] For each information source, when the feedback information indicates that the target detection in the target detection information is correct, the current weight value is increased; when the feedback information indicates that the target detection in the target detection information is incorrect, the current weight value is decreased.

[0106] According to an embodiment of the present application, a plurality of target detection information are fused to obtain an information fusion result, including:

[0107] Obtaining current weight values ​​corresponding to a plurality of target detection information;

[0108] The multiple target detection information are normalized and weighted averaged according to their corresponding current weight values ​​to obtain an information fusion result.

[0109] Next, refer to Figure 5 and Figure 6 To describe the method for adjusting the current weight value and the method for information fusion based on the feedback message from the driver of the vehicle according to an embodiment of the present application.

[0110] like Figure 5 As shown, according to an embodiment of the present application, the method for weight adjustment and information fusion of the current weight value based on the feedback information of the driver of the vehicle in the T scenario includes:

[0111] S501, S502, S503: In S501, the driver determines whether the V2I message is accurate. If it is accurate, the process proceeds to S502 and the weight of the V2I message is adjusted to w1=w1+0.02. If it is inaccurate, the process proceeds to S503 and the weight of the V2I message is adjusted to w1=w1-0.02.

[0112] It should be noted that the adjustment value can be set according to actual conditions, and this embodiment does not specifically limit the size of the adjustment value.

[0113] S504, S505, S506: In S504, the driver determines whether the camera message is accurate. If accurate, the process proceeds to S505 to adjust the camera message weight to w2=w2+0.02; if inaccurate, the process proceeds to S506 to adjust the camera message weight to w2=w2-0.02.

[0114] S507, S508, S509: In S507, the driver determines whether the V2V message is accurate. If accurate, the process proceeds to S508 to adjust the V2V message weight to w3=w3+0.01; if inaccurate, the process proceeds to S509 to adjust the V2V message weight to w3=w3-0.01.

[0115] S510: Perform normalized weighted averaging on the target detection information obtained from each information source according to its corresponding weight value to obtain an information fusion result.

[0116] like Figure 6 As shown, according to an embodiment of the present application, the method for adjusting the weight and fusing the information of the current weight value based on the feedback information of the driver of the vehicle in the P scenario includes:

[0117] S601, S602, S603: In S601, the driver determines whether the camera message is accurate. If accurate, the process proceeds to S602 to adjust the camera message weight to w2=w2+0.02; if inaccurate, the process proceeds to S603 to adjust the camera message weight to w2=w2-0.02.

[0118] S604, S605, S606: In S604, the driver determines whether the millimeter-wave radar message is accurate. If accurate, the process proceeds to S605 to adjust the weight of the millimeter-wave radar message to w4=w4+0.02; if inaccurate, the process proceeds to S406 to adjust the weight of the millimeter-wave radar message to w4=w4-0.02.

[0119] S607, S608, S609: In S607, the driver determines whether the V2V message is accurate. If accurate, the process proceeds to S608 to adjust the V2V message weight to w3=w3+0.01; if inaccurate, the process proceeds to S609 to adjust the V2V message weight to w3=w3-0.01.

[0120] S610: Perform normalized weighted averaging on the target detection information obtained from each information source according to its corresponding weight value to obtain an information fusion result.

[0121] It should be noted that all adjustment values ​​involved in the above two embodiments can be set according to actual conditions, and this embodiment does not specifically limit the size of the adjustment value.

[0122] The reliability of all corresponding message sources is evaluated in two scenarios through driver feedback, and then their corresponding message weights are adjusted in a step-by-step manner, so that the corresponding weights of the sensors can better adapt to changes in the driving environment.

[0123] According to one embodiment of the present application, the adaptive perception fusion driving method based on V2X and vehicle-mounted sensors also includes: broadcasting the target detection result and the vehicle's traffic plan in the Internet of Vehicles.

[0124] Specifically, the communication between the vehicle and the Internet of Vehicles can include two communication interfaces, one is a short-distance direct communication interface (PC5) between the vehicle, the person, and the road, and the other is a communication interface (Uu) between the terminal and the base station, which can achieve reliable communication over long distances and a larger range. Preferably, the vehicle performs short-distance broadcasting through the PC5 interface to share information with nearby vehicles and provide driving decision references for surrounding vehicles.

[0125] For example, when the vehicle detects a target, such as a vulnerable traffic participant in the surrounding area who needs to be avoided, the vehicle's driving plan will affect the driving plans of the surrounding vehicles. When the vehicle automatically makes an emergency avoidance, the vehicle's operation may cause danger to normal vehicles around it. Therefore, the vehicle status information when the vehicle implements automatic avoidance, the information detected by the vehicle on the vulnerable traffic participants around it, etc. are broadcasted through the PC5 air interface through V2X to complete information sharing with surrounding vehicles, thereby providing driving decision references for surrounding vehicles.

[0126] Next, refer to Figure 7 The following describes the adaptive perception fusion driving method according to the third embodiment of the present application. This embodiment provides a driving method for detecting, warning and avoiding vulnerable traffic participants and traffic lights at intersections by fusion of sensors and V2X for collaborative perception and adaptive adjustment based on driver feedback, the method comprising:

[0127] S701: The vehicle obtains map information through the vehicle navigation while driving on a municipal road, wherein the map information includes the location information of traffic lights at the intersection;

[0128] S702: Determine the distance between the vehicle and the next traffic light intersection using the traffic light position information obtained in step S701. If the distance is greater than 150 meters, execute step S703 to detect vulnerable traffic participants. When the vehicle travels until the distance to the next traffic light intersection is less than 150 meters, execute step S709 to detect and determine the traffic light.

[0129] S703: If the determination in step S702 is no, then this step is executed to detect vulnerable traffic participants such as pedestrians and bicycles;

[0130] S704: The vehicle obtains target detection information shared by the camera, millimeter-wave radar, and V2V of surrounding vehicles. The camera connects the identified target with the surrounding environment, finds the connection between targets and targets, targets and the environment, and obtains information such as target type, status, and distance; the millimeter-wave radar receives the electromagnetic wave energy reflected by the target, processes the information, extracts useful information, and performs object recognition and tracking to obtain information such as the position, speed, size, and type of the target; synchronously obtains warning information such as the position and distance of the target detected by surrounding vehicles to vulnerable traffic participants in the blind spot of the vehicle through the V2X PC5 air interface;

[0131] S705: The above three types of detection results of the target are integrated, and the weights of different information sources are dynamically and adaptively adjusted to increase the reliability of the judgment result. Then, a vehicle passage plan is formulated based on the judgment result, that is, whether the vehicle needs to avoid the target;

[0132] S706: When the threshold value determines that it is not an emergency situation and no avoidance operation is required for the target in step S705, the vehicle continues to drive normally or the driver decides on the driving plan;

[0133] S707: When step S705 determines through the threshold that there are vulnerable traffic participants around the vehicle and the current driving state will cause collision damage to the target and emergency avoidance is required, after a reasonable decision is made, the driver is informed or warned by the alarm device in various ways such as sound, light and touch, and the vehicle is actively intervened in time through the control execution device so that the vehicle can automatically make emergency avoidance;

[0134] S708: Regardless of the vehicle's driving status (avoidance or normal driving), the vehicle BSM message of the vehicle's traffic plan and the detected surrounding target information are broadcast through the PC5 air interface to provide decision-making reference information for other surrounding vehicles.

[0135] S709: If the result of step S702 is yes, then this step is executed to detect the status of the traffic lights at the intersection;

[0136] S710: The vehicle integrates the detection information of the traffic light image by the vehicle camera, the traffic light information of the intersection broadcasted by the PC5 air interface of the surrounding vehicles received through V2V, and the SPAT information filtered by the roadside unit according to the intersection number ID;

[0137] S711: The above three types of traffic light information detection results are integrated, and the weights of different information sources are dynamically and adaptively adjusted to increase the reliability of the judgment results. Then, a vehicle passage plan is formulated based on the judgment results, that is, whether the vehicle needs to wait for the traffic light at the intersection;

[0138] S712: When the threshold value is passed in step S711 to determine that the red light ahead requires deceleration and stopping, after a reasonable decision is made, the driver is informed or warned by the alarm device in various ways such as sound, light and touch, and the vehicle is actively intervened in time by the control execution device to make the vehicle decelerate and stop to wait for the red light;

[0139] S713: When the threshold value in step S711 determines that the road ahead is passable or the driver needs to participate in the determination, the driver decides to continue driving normally or slow down to pass the traffic light intersection;

[0140] S714: The vehicle BSM information of the vehicle's traffic plan (normal traffic or waiting for the red light) and the traffic light detection information are broadcasted through the PC5 air interface to provide decision-making reference information for surrounding vehicles.

[0141] It should be noted that in S705 and S711, the preset value corresponding to the V2V message is smaller than the preset value corresponding to the vehicle sensor message, and the preset value corresponding to the V2V message is smaller than the preset value corresponding to the V2I message.

[0142] Preferably, the preset value of the current weight value corresponding to the V2V message is half of the preset value corresponding to other types of messages. Whether in the P scenario or the T scenario, when initializing or adaptively updating, the credibility weight of the V2V message is reduced by about half compared to other types of messages. This setting is more in line with the driver's usage habits. Because compared with the information provided by other vehicles, the driver naturally trusts the source of the sensor message of the vehicle more; compared with the V2V message, the driver will naturally trust the V2I roadside unit message more.

[0143] The method of this embodiment provides specific information fusion solutions for vulnerable traffic participants and traffic lights on the current road section. Such fusion solutions fully extract the effective information from multiple message sources, namely, millimeter wave radar, camera, V2V message in P scene and V2I, camera, V2V message in T scene. Moreover, when a certain message is inaccurate or has a large offset, the fusion with other messages can offset the impact of the offset, thereby greatly reducing the probability of the driver being misled, and the probability of driving safety risk or violation risk is correspondingly reduced.

[0144] An embodiment of the present application also provides an adaptive perception fusion driving device, which includes a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the device equipped with the processor executes the adaptive perception fusion driving method described in any of the above embodiments.

[0145] An embodiment of the present application also provides a storage medium, characterized in that a computer program is stored on the storage medium, and the computer program runs on a computer. When the computer program is running, the computer executes the adaptive perception fusion driving method described in any of the above embodiments.

[0146] An embodiment of the present application also provides a vehicle, which includes the above-mentioned adaptive perception fusion driving device or the above-mentioned storage medium.

[0147] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0148] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0150] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0151] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0152] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0153] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0154] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in the article analysis device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0155] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0156] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment, and the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An adaptive perception fusion driving method, It is characterized in that The method comprises: For the target to be detected, multiple target detection information is obtained based on the vehicle sensor data and the V2X data respectively; Based on the weights obtained by the current adaptive adjustment, the plurality of target detection information are fused to obtain an information fusion result; Based on the information fusion result, determining the target detection result according to the preset judgment rules; A traffic plan for the vehicle is generated based on the target detection result.

2. The adaptive perception fusion driving method according to claim 1, It is characterized in that Taking a device that collects the vehicle-mounted sensor data or receives the V2X data as an information source, the method for adaptively adjusting the weight includes: For each information source, determining a current weight value of the target detection information obtained based on the vehicle-mounted sensor data or the V2X data, including: if the target detection information indicates that the target is detected, the current weight value is a preset value greater than 0 and less than 1; if the target detection information indicates that the target is not detected, the current weight value is 0; Adjusting the current weight value based on feedback information from the driver of the vehicle to obtain an adjusted weight value; The adjusted weight value is used as a preset value corresponding to the target detection information obtained through the information source during the next information fusion.

3. The adaptive perception fusion driving method according to claim 2, It is characterized in that The weight adjustment of the current weight value based on the feedback information of the driver of the vehicle includes: For each information source, when the feedback information indicates that the target detection in the target detection information is correct, the current weight value is increased; when the feedback information indicates that the target detection in the target detection information is incorrect, the current weight value is decreased.

4. The adaptive perception fusion driving method according to any one of claims 1 to 3, It is characterized in that The step of fusing the plurality of target detection information to obtain an information fusion result includes: Obtaining current weight values ​​corresponding to a plurality of target detection information; The multiple target detection information are normalized and weighted averaged according to their corresponding current weight values ​​to obtain an information fusion result.

5. The adaptive perception fusion driving method according to any one of claims 1 to 3, It is characterized in that When the target to be detected is an obstacle avoidance target, the on-board sensor data includes one or more of position data, speed data, and video image data, and the V2X data includes V2V data; when the target to be detected is a driving guidance sign, the on-board sensor data includes video image data, and the V2X data includes V2V data and V2I data.

6. The adaptive perception fusion driving method according to claim 5, It is characterized in that The preset value of the current weight value corresponding to the V2V data is smaller than the preset value corresponding to the on-board sensor data, and the preset value of the current weight value corresponding to the V2V data is smaller than the preset value corresponding to the V2I data.

7. The adaptive perception fusion driving method according to any one of claims 1 to 6, It is characterized in that The determining of the target detection result based on the information fusion result according to a preset judgment rule includes: When the information fusion result is greater than or equal to a first preset threshold, determining that the target detection result is that the current target is detected; When the information fusion result is less than or equal to a second preset threshold, it is determined that the target detection result is that the current target is not detected. When the information fusion result is less than the first preset threshold and greater than the second preset threshold, it is determined that the target detection result is that the current target is in an uncertain state.

8. The adaptive perception fusion driving method according to any one of claims 1 to 6, It is characterized in that The method further comprises: The target detection result and the vehicle's traffic plan are broadcasted in the Internet of Vehicles.

9. The adaptive perception fusion driving method according to any one of claims 1 to 6, It is characterized in that The method further comprises: Acquiring map information, wherein the map information includes location information of driving guidance signs on the driving road; Based on the position information of the driving guidance sign, determining the distance between the vehicle and the next driving guidance sign; Determining a target to be detected based on the distance includes: When the distance is greater than a preset distance threshold, the obstacle avoidance target is taken as a target to be detected; When the distance is less than or equal to a preset distance threshold, the driving guidance sign is taken as a target to be detected.

10. The adaptive perception fusion driving method according to any one of claims 1 to 6, It is characterized in that When the target to be detected is an obstacle avoidance target and the vehicle's traffic plan is emergency avoidance, the method further includes: Generate alarm trigger information to trigger the alarm device to send out an alarm signal, wherein the alarm signal includes one or more of a sound signal, a light signal, and a tactile signal.

11. An adaptive perception fusion driving device, It is characterized in that The device includes a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the device equipped with the processor executes the adaptive perception fusion driving method as described in any one of claims 1-10.

12. A storage medium, It is characterized in that The storage medium stores a computer program, which runs on a computer. When the computer program is running, it enables the computer to execute the adaptive perception fusion driving method as described in any one of claims 1 to 10.

13. A vehicle, It is characterized in that The vehicle includes the adaptive perception fusion driving device according to claim 11 or the storage medium according to claim 12.

Citation Information

Patent Citations

  • Data fusion method, electronic equipment and storage medium

    CN111428759A

  • Information processing method and device in automatic driving and storage medium

    CN112241167A

  • Vehicle driving scene acquisition method, device, equipment and medium

    CN113611008A

  • Vehicle early warning control method and system and vehicle

    CN115482685A

  • Multi sensor based obstacle detection apparatus and method

    KR1020180007412A

Cited By

  • Vehicle emergency braking method, electronic equipment and storage medium

    CN119796134A

  • Internet-of-Vehicles vehicle road multi-device joint modeling system

    CN121635313A