A method and system for regulating multimodal stimulation during manual takeover of autonomous driving
By constructing a multimodal stimulation database and individual driver database, dynamically adjusting auditory, vibration and visual stimulation, the problem of modal stimulation differences in timing adjustment of autonomous driving takeover requests is solved, the quality and safety of takeover are improved, and high users are achieved Experience.
Patent Information
- Application Number
- CN202210419353.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-20
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-04-20
AI Technical Summary
The prior art fails to effectively consider the differences in different modal stimulation types in the timing adjustment of autonomous driving takeover requests, resulting in limited takeover quality and safety.
By constructing a driver cloud database and an individual driver database, dynamically adjust the takeover time and action coefficient of auditory, vibration and visual stimulation, select a multimodal stimulation method that meets the current takeover needs, and optimize the takeover process in real time.
The quality and user experience of autonomous driving takeover have been improved. By learning the historical data of individual drivers, the stimulation methods are updated, and the habits and attention of different drivers are adapted to the habits and attention of different drivers, and the quality and safety of takeover have been achieved.
Smart Images

Figure CN114771566B_ABST
Abstract
Description
Technical Field
[0001] The present invention provides a method and system for regulating multimodal stimulation of manual takeover of autonomous driving, and relates to the fields of autonomous driving technology and ergonomics. Background Art
[0002] According to the classification of vehicle automation level standards by the Society of Automotive Engineers (SAE), car autonomous driving is divided into six levels, from L0 to L5. Autonomous driving takeover mainly occurs in L2-L3 autonomous vehicles. When the autonomous driving system cannot handle certain situations, the driver must take over vehicle control within a safe time.
[0003] The entire takeover process can be divided into three stages: consciousness awakening, takeover decision and behavioral decision. Different ways of providing takeover information, that is, multimodal stimulation, such as the combination of visual, auditory, steering wheel vibration and other prompts, have a great impact on the driver's performance in the entire takeover process, and even directly determine whether the driver can complete the takeover task well. Therefore, multimodal stimulation plays an important role in the quality of driving takeover and the safety of vehicle driving.
[0004] At present, with the development of autonomous driving technology, the safety of autonomous driving has become a focus of attention. With the continuous maturity of simulated driving simulation systems, using driving simulation experimental platforms and combining different modal stimuli to study the quality of driver takeover from automatic takeover to manual takeover is of great significance to the development and progress of autonomous driving vehicles.
[0005] In patent document CN113460074A, parameters such as takeover quality P0 and action coefficient, lead time of takeover request and takeover readiness have been stored in the big data cloud database in advance. There is no mention of the impact of multimodal stimulation on driver behavior, and the patent goal is to adjust the timing of takeover requests for autonomous driving, and there are no differences in the adjustment targets for stimulation of different modal types. Summary of the invention
[0006] The purpose of the present invention is to provide a method and system for dynamically adjusting three multimodal stimuli, namely sound, vibration and vision, based on the target takeover quality in an environment where manual takeover of autonomous driving is required, and continuously updating and optimizing in real time.
[0007] A method for adjusting multimodal stimulation of manual takeover of automatic driving, wherein the multimodal stimulation includes auditory stimulation, vibration stimulation and visual stimulation; the method has the following specific steps:
[0008] Step 1: construct a driver cloud database through driving simulator experiments and real vehicle experiments based on takeover scenarios; the driver cloud database includes the target takeover quality P0, the takeover time A0 caused by multimodal stimulation and corresponding auditory stimulation, the takeover time B0 caused by vibration stimulation, the takeover time C0 caused by visual stimulation, the effect coefficient a0 of the takeover quality of the driver when the auditory stimulation is used to issue a takeover request, the effect coefficient b0 of the takeover quality of the driver when the vibration stimulation is used to issue a takeover request, and the effect coefficient c0 of the takeover quality of the driver when the visual stimulation is used to issue a takeover request;
[0009] Step 2: construct an individual driver database through the actual vehicle takeover data of the driven vehicle; the individual driver database includes the effect coefficient of the auditory stimulation request of the driven vehicle on the takeover quality of the driver, the effect coefficient of the vibration stimulation request of the driven vehicle on the takeover quality, the effect coefficient of the visual stimulation request of the driven vehicle on the takeover quality of the driver, the multimodal stimulation mode of each takeover event, and the corresponding target takeover quality and actual takeover quality;
[0010] Step 3: Determine whether the current autonomous driving exceeds the performance boundary of the autonomous driving system. If it reaches the performance boundary, this method is not executed; if it is not at the performance boundary, proceed to step 4;
[0011] Step 4: According to the target takeover quality P' of the current takeover event and the effect coefficient a of the auditory stimulation request of the driven vehicle on the takeover quality of the driver, the effect coefficient b of the vibration stimulation request of the driven vehicle on the takeover quality, and the effect coefficient c of the visual stimulation request of the driven vehicle on the takeover quality of the driver, calculate the expected takeover time A caused by the auditory stimulation, the expected takeover time B caused by the vibration stimulation, and the expected takeover time C caused by the visual stimulation, and select a multimodal stimulation method that meets the requirements in the corresponding cloud driver database;
[0012] Step 5: The individual driver completes the takeover, and the actual takeover quality P is calculated based on the takeover operation of the individual driver;
[0013] Step 6: Upload P', P, a, b, c corresponding to the current takeover event to the individual driver database, and update the individual driver database.
[0014] Furthermore, in step 1, P0=a0A0+b0B0+c0C0.
[0015] Furthermore, the initial value of the coefficient of the auditory stimulation request of the driven vehicle on the driver's takeover quality is a0, the initial value of the coefficient of the vibration stimulation request of the driven vehicle on the takeover quality is b0, and the initial value of the coefficient of the visual stimulation request of the driven vehicle on the driver's takeover quality is c0.
[0016] Furthermore, the calculation method of the takeover time in step 4 is as follows:
[0017] X=max(0,P' / x)
[0018] Y=max(0,(P'-xX) / y)
[0019] Z=max(0,(P'-xX-yY) / z)
[0020] Where x=max(a,b,c), z=min(a,b,c), y is the middle value among a, b, c, and X, Y, and Z are the takeover times corresponding to x, y, and z respectively.
[0021] Furthermore, in step 5, the actual takeover quality P=TTBT-TOT, TTBT is the time distance between the driven vehicle and the boundary of the automatic driving system when taking over, and TOT is the driver's reaction time.
[0022] Furthermore, in step 6, a, b, and c are updated by using a multivariate linear regression method and the formula P=aA+bB+cC.
[0023] A multi-modal stimulation adjustment system for manual takeover of an autonomous driving system includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps of the multi-modal stimulation adjustment method for manual takeover of an autonomous driving system are implemented:
[0024] Step 1: construct a driver cloud database through driving simulator experiments and real vehicle experiments based on takeover scenarios; the driver cloud database includes the target takeover quality P0, the takeover time A0 caused by multimodal stimulation and corresponding auditory stimulation, the takeover time B0 caused by vibration stimulation, the takeover time C0 caused by visual stimulation, the effect coefficient a0 of the takeover quality of the driver when the auditory stimulation is used to issue a takeover request, the effect coefficient b0 of the takeover quality of the driver when the vibration stimulation is used to issue a takeover request, and the effect coefficient c0 of the takeover quality of the driver when the visual stimulation is used to issue a takeover request;
[0025] Step 2: construct an individual driver database through the actual vehicle takeover data of the driven vehicle; the individual driver database includes the effect coefficient of the auditory stimulation request of the driven vehicle on the takeover quality of the driver, the effect coefficient of the vibration stimulation request of the driven vehicle on the takeover quality, the effect coefficient of the visual stimulation request of the driven vehicle on the takeover quality of the driver, the multimodal stimulation mode of each takeover event, and the corresponding target takeover quality and actual takeover quality;
[0026] Step 3: Determine whether the current autonomous driving exceeds the performance boundary of the autonomous driving system. If it reaches the performance boundary, this method is not executed; if it is not at the performance boundary, proceed to step 4;
[0027] Step 4: According to the target takeover quality P' of the current takeover event and the effect coefficient a of the auditory stimulation request of the driven vehicle on the takeover quality of the driver, the effect coefficient b of the vibration stimulation request of the driven vehicle on the takeover quality, and the effect coefficient c of the visual stimulation request of the driven vehicle on the takeover quality of the driver, calculate the expected takeover time A caused by the auditory stimulation, the expected takeover time B caused by the vibration stimulation, and the expected takeover time C caused by the visual stimulation, and select a multimodal stimulation method that meets the requirements in the corresponding cloud driver database;
[0028] Step 5: The individual driver completes the takeover, and the actual takeover quality P is calculated based on the takeover operation of the individual driver;
[0029] Step 6: Upload P', P, a, b, c corresponding to the current takeover event to the individual driver database, and update the individual driver database.
[0030] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the multi-modal stimulation adjustment method for manual takeover of autonomous driving are implemented as follows:
[0031] Step 1: construct a driver cloud database through driving simulator experiments and real vehicle experiments based on takeover scenarios; the driver cloud database includes the target takeover quality P0, the takeover time A0 caused by multimodal stimulation and corresponding auditory stimulation, the takeover time B0 caused by vibration stimulation, the takeover time C0 caused by visual stimulation, the effect coefficient a0 of the takeover quality of the driver when the auditory stimulation is used to issue a takeover request, the effect coefficient b0 of the takeover quality of the driver when the vibration stimulation is used to issue a takeover request, and the effect coefficient c0 of the takeover quality of the driver when the visual stimulation is used to issue a takeover request;
[0032] Step 2: construct an individual driver database through the actual vehicle takeover data of the driven vehicle; the individual driver database includes the effect coefficient of the auditory stimulation request of the driven vehicle on the takeover quality of the driver, the effect coefficient of the vibration stimulation request of the driven vehicle on the takeover quality, the effect coefficient of the visual stimulation request of the driven vehicle on the takeover quality of the driver, the multimodal stimulation mode of each takeover event, and the corresponding target takeover quality and actual takeover quality;
[0033] Step 3: Determine whether the current autonomous driving exceeds the performance boundary of the autonomous driving system. If it reaches the performance boundary, this method is not executed; if it is not at the performance boundary, proceed to step 4;
[0034] Step 4: According to the target takeover quality P' of the current takeover event and the effect coefficient a of the auditory stimulation request of the driven vehicle on the takeover quality of the driver, the effect coefficient b of the vibration stimulation request of the driven vehicle on the takeover quality, and the effect coefficient c of the visual stimulation request of the driven vehicle on the takeover quality of the driver, calculate the expected takeover time A caused by the auditory stimulation, the expected takeover time B caused by the vibration stimulation, and the expected takeover time C caused by the visual stimulation, and select a multimodal stimulation method that meets the requirements in the corresponding cloud driver database;
[0035] Step 5: The individual driver completes the takeover, and the actual takeover quality P is calculated based on the takeover operation of the individual driver;
[0036] Step 6: Upload P', P, a, b, c corresponding to the current takeover event to the individual driver database, and update the individual driver database.
[0037] The beneficial effects of the present invention are as follows: according to the multimodal stimulation adjustment method and system for the above-mentioned automatic driving manual takeover request of the present invention, when each takeover event is triggered, the sound action coefficient a, vibration action coefficient b and visual action coefficient c related to the takeover quality P are added when determining the takeover time A of the sound prompt, the takeover time B of the vibration prompt and the takeover time C of the visual prompt for the driver to take over the automatic driving, and each stimulus and its corresponding takeover time are pre-stored in the cloud database, so as to facilitate the determination of the stimulus mode. An individual driver database is constructed to save the historical driving data of the individual driver, that is, the actual takeover quality P, the expected takeover quality P' and the action coefficients a, b and c of the multimodal stimulation received by the driver are updated for each takeover event. Through such steps, the action coefficients a, b and c reflecting the operating habits of the individual driver can be updated in time by learning the historical driving data of the individual driver, so that when different prompting methods are selected in the face of different driving takeover events, the values that meet the current driving habits and attention of each driver can be obtained. Therefore, high user experience and takeover quality can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 This is a flowchart showing a control method of an automatic driving system according to the present invention. DETAILED DESCRIPTION
[0039] The embodiment of the present invention is a multi-modal stimulation adjustment method for an automatic driving takeover request. The method selects a multi-modal stimulation scheme that meets the current takeover quality requirements and the driver's state based on the continuous collection and update of the driver's real-time signal reception coefficient, and realizes the overall process as follows: Figure 1 The method comprises the following steps:
[0040] Step 1: Build a driver cloud database
[0041] A driver big data cloud database is constructed through driving simulator experiments and real vehicle experiments based on takeover scenarios. The parameters stored in this database include P0, A0, B0, C0, a0, b0, c0, and each parameter satisfies the formula: P0=a0A0+b0B0+c0C0.
[0042] Among them, P0 is the target takeover quality based on experimental big data. In order to ensure the takeover quality of the driver's safe takeover, based on experimental data and different takeover scenarios, it can be 6 seconds to 12 seconds; a0 is the effect coefficient of the takeover quality of the driver issued by auditory information obtained through simulated driving experiments, b0 is the effect coefficient of the takeover quality of the driver issued by vibration information obtained through simulated driving experiments, and c0 is the effect coefficient of the takeover quality of the driver issued by visual information obtained through simulated driving experiments. a0, b0 and c0 are all obtained by multivariate linear regression analysis of the data obtained from the driving simulation experiment; A0, B0, C0 are the general takeover times caused by various auditory information, vibration information and takeover information pre-stored in the database, that is, the takeover times corresponding to various types of modal stimulation schemes based on the statistics and analysis of driving simulation experiment data.
[0043] Step 2: Build an individual driver database
[0044] An individual driver database is constructed through the actual vehicle takeover data of the driven vehicle (hereinafter referred to as "this vehicle"). Obviously, the individual driver databases of different vehicles are different. A takeover completed by the driver is recorded as a takeover event. The parameters stored in the database include a, b, c and the data corresponding to each takeover event, including the expected takeover quality P' and the actual takeover quality P.
[0045] a is the effect coefficient of the auditory stimulation request of the vehicle on the driver's takeover quality, and its initial value is a0. b is the effect coefficient of the vibration stimulation request of the vehicle on the takeover quality, and its initial value is b0. c is the effect coefficient of the visual stimulation request of the vehicle on the driver's takeover quality, and its initial value is c0. In the data P' and P corresponding to each takeover event, P' is the target takeover quality, and its initial value is P0 in the driver's big data cloud database. P is the actual takeover quality, and the units of both are seconds.
[0046] Step 3: Determine the performance boundary of the autonomous driving system
[0047] Determine whether the current autonomous driving exceeds the performance boundary of the autonomous driving system (i.e., conditions that cannot be handled by L3 and L4 level autonomous driving systems in autonomous driving mode). If the performance boundary is not reached, proceed to step 4. If the performance boundary is exceeded, exit this method.
[0048] Step 4: Take over the request prompt step
[0049] According to the expected takeover quality P' required by the current driving takeover scenario and the driver's current acceptance coefficients a, b, c for the multimodal stimulus, the corresponding takeover time A, B, C is calculated, and then a multimodal stimulus method that meets the requirements is selected from the resource library. In the takeover request prompt module, the magnitude relationship of the coefficients a, b, c of the hearing, vibration and vision on the driver stored in the storage module is used as the standard, and the calculation priority is sorted in this order, so as to calculate the takeover time caused by the corresponding stimulus in turn.
[0050] In one embodiment, taking the auditory coefficient>vibration coefficient>visual coefficient as an example, at this time a>b>c, the stimulus priority given to the driver in the database is auditory>vibration>vision, and the specific stimulus takeover time is calculated according to the following formula:
[0051] A=max(0,P' / a)
[0052] B=max(0,(P'-aA) / b)
[0053] C = max(0,(P'-aA-bB) / c)
[0054] When the size relationship of the three coefficients in the database changes, the size relationship and calculation priority order of the corresponding a, b, and c will also change accordingly.
[0055] Step 5: Take over the quality assessment step
[0056] The driver completes the current takeover according to the current takeover request. Then, the current actual takeover quality P is calculated according to the current takeover operation data of the driver. There are many indicators for the takeover quality, such as the statistical value of the input data of the driver's steering wheel, the statistical value of the input data of the driver's accelerator pedal, the statistical value of the input data of the driver's decelerator pedal, and the driver's reaction time.
[0057] In one embodiment, the difference between the time distance TTBT between the vehicle and the boundary of the automatic driving system and the driver's reaction time TOT during takeover is used as the takeover quality, that is:
[0058] P = TTBT-TOT.
[0059] In particular, when the takeover fails, that is, a collision occurs, P takes 0.
[0060] Step 6: Record the steps to take over data
[0061] A database updating step, based on the takeover quality P of the driver and the takeover time A, B and C corresponding to the stimulus signal given recorded in the data group of the individual driver database, updates the action coefficients a, b and c in the individual driver database to be used as a calculation basis for the stimulus given in the next takeover event.
[0062] According to the above embodiments, the following beneficial effects can be obtained.
[0063] (1) In the above-mentioned automatic driving manual takeover request timing adjustment method and system, when selecting which multimodal stimulation such as auditory stimulation, vibration stimulation and visual stimulation to use, the auditory action coefficient a, the vibration action coefficient b and the action coefficient c related to the takeover quality P are added, and the individual driver data library is constructed to store the individual driver's historical driving data, that is, the takeover quality P and the action coefficients a, b, c of each takeover event. Through such steps, the action coefficients a, b, c that reflect the multimodal stimulation received by the individual driver can be updated in time by learning the individual driver's historical driving data, so that when calculating which multimodal stimulation to use, a value that meets the driving habits of each driver can be obtained. Therefore, a high user experience and takeover quality can be achieved.
[0064] (2) Since the data recorded in the individual driver database will be uploaded to the driver big data cloud database, and the parameters P0, a0, b0, c0 stored in the driver big data cloud database, which serve as the initial values of the parameters P′, a, b, c in the individual driver database, are regularly updated based on the uploaded data, the initial values of the cloud database can be updated based on more takeover data from more drivers, thereby providing better initial values for later produced vehicles.
[0065] (3) In the step of updating the individual driver database, the multivariate linear regression method is used and the action coefficients a, b, c are updated through the formula, P = aA + bB + cC, which can simply obtain data that can accurately reflect the driving habits and status of individual drivers, thereby calculating the appropriate multimodal stimulation method.
[0066] In one embodiment, a multimodal stimulation adjustment system for manual takeover of autonomous driving is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned multimodal stimulation adjustment method for manual takeover of autonomous driving when executing the computer program.
[0067] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multi-modal stimulation adjustment method for manual takeover of autonomous driving are implemented.
[0068] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0069] The above embodiments are only for illustrating the technical idea of the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.
Claims
1. A multi-modal stimulation adjustment method for manual takeover of autonomous driving, characterized in that: The multimodal stimulation includes auditory stimulation, vibration stimulation and visual stimulation; the specific steps of the method are as follows: Step 1: construct a driver cloud database through driving simulator experiments and real vehicle experiments based on takeover scenarios; the driver cloud database includes the target takeover quality P0, the takeover time A0 caused by multimodal stimulation and corresponding auditory stimulation, the takeover time B0 caused by vibration stimulation, the takeover time C0 caused by visual stimulation, the effect coefficient a0 of the takeover quality of the driver when the auditory stimulation is used to issue a takeover request, the effect coefficient b0 of the takeover quality of the driver when the vibration stimulation is used to issue a takeover request, and the effect coefficient c0 of the takeover quality of the driver when the visual stimulation is used to issue a takeover request; Step 2: construct an individual driver database through the actual vehicle takeover data of the driven vehicle; the individual driver database includes the effect coefficient of the auditory stimulation request of the driven vehicle on the takeover quality of the driver, the effect coefficient of the vibration stimulation request of the driven vehicle on the takeover quality, the effect coefficient of the visual stimulation request of the driven vehicle on the takeover quality of the driver, the multimodal stimulation mode of each takeover event, and the corresponding target takeover quality and actual takeover quality; Step 3: Determine whether the current autonomous driving exceeds the performance boundary of the autonomous driving system. If it reaches the performance boundary, this method is not executed; if it is not at the performance boundary, proceed to step 4; Step 4: According to the target takeover quality P' of the current takeover event and the effect coefficient a of the auditory stimulation request of the driven vehicle on the takeover quality of the driver, the effect coefficient b of the vibration stimulation request of the driven vehicle on the takeover quality, and the effect coefficient c of the visual stimulation request of the driven vehicle on the takeover quality of the driver, calculate the expected takeover time A caused by the auditory stimulation, the expected takeover time B caused by the vibration stimulation, and the expected takeover time C caused by the visual stimulation, and select a multimodal stimulation method that meets the requirements in the corresponding cloud driver database; Step 5: The individual driver completes the takeover, and the actual takeover quality P is calculated based on the takeover operation of the individual driver; Step 6: Upload P', P, a, b, c corresponding to the current takeover event to the individual driver database, and update the individual driver database; The calculation method of the takeover time in step 4 is as follows: X=max(0,P' / x) Y=max(0,(P'-xX) / y) Z=max(0,(P'-xX-yY) / z) Where x = max(a, b, c), z = min(a, b, c), y is the middle value among a, b, c, and X, Y, and Z are the takeover times corresponding to x, y, and z respectively; In step 5, the actual takeover quality P=TTBT-TOT, where TTBT is the time distance between the driven vehicle and the boundary of the automatic driving system when taking over, and TOT is the driver's reaction time.
2. The method for adjusting multi-modal stimulation of automatic driving manual takeover according to claim 1, characterized in that: In step 1, P0=a0A0+b0B0+c0C0.
3. The method for adjusting multi-modal stimulation of automatic driving manual takeover according to claim 1, characterized in that: The initial value of the effect coefficient of the auditory stimulation request of the driven vehicle on the driver's takeover quality is a 00 The initial value of the coefficient of the vibration stimulus request of the driven vehicle on the takeover mass is b 00 , the initial value of the effect coefficient of the visual stimulus request of the driven vehicle on the driver's takeover quality is c 00 .
4. The method for adjusting multi-modal stimulation of automatic driving manual takeover according to claim 1, characterized in that: In step 6, a, b, and c are updated by using a multivariate linear regression method and the formula P=aA+bB+cC.
5. An automatic driving manual takeover multimodal stimulation adjustment system, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the multi-modal stimulation adjustment method for manual takeover of autonomous driving described in any one of claims 1 to 4 are implemented.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-modal stimulation adjustment method for manual takeover of autonomous driving described in any one of claims 1 to 4 are implemented.
Citation Information
Cited By
A distraction task adaptive takeover pre-warning method and system for L3 level automatic driving
CN122761653A
