Key information acquisition method and device and intelligent driving equipment

Abstract

The invention provides a key information acquisition method and device and intelligent driving equipment, and the method comprises the steps: obtaining the biological feature information of a user in the intelligent driving equipment, the biological characteristic information of the user comprises one or more of the following items: expression information of the user, action information of the user, voice information of the user, eyeball focus information of the user or pupil change information of the user; obtaining driving environment information of the intelligent driving equipment; the key information in the driving environment information is determined according to the biological characteristic information of the user, the key information is used for assisting intelligent driving, the effectiveness of the collected key information can be improved, the situation of missing detection and false detection of obstacles is relieved, and planning and implementation of an automatic driving task are better facilitated.

Description

Key information collection method, device and intelligent driving equipment Technical Field

[0001] The present application relates to intelligent driving technology, which is applied to the field of key information collection for intelligent driving, and in particular to a key information collection method, device and intelligent driving equipment. Background Art

[0002] With the development of society, smart cars are gradually becoming part of people's daily lives, and intelligent driving is playing an increasingly important role. An intelligent driving system is a computer system that can use data collected by a perception system to identify and detect objects, generate results through calculation and analysis, and directly control the terminal or assist the human driver (or other decision-making body) in controlling the terminal. The perception subsystem of an intelligent driving system is typically implemented through a neural network model. During training, this neural network model requires pre-collection of driving environment information and is trained based on this driving environment information. This allows the model to learn to identify and detect objects in the driving environment information that may affect driving decisions.

[0003] This driving environment information can include various types of information, such as obstacles, traffic lights, lane markings, and traffic signs. Taking obstacle recognition as an example, the more types of obstacles input during training, the higher the trained model's obstacle recognition accuracy. Conversely, if some obstacle information is not used in the model training process, the model's recognition results for these obstacles will be inaccurate, easily leading to missed obstacle detections (for example, not recognizing an object as an obstacle) and false detections (incorrectly identifying one obstacle as another). When missed or false detections occur in intelligent driving scenarios, they can cause the intelligent driving terminal to fail to make correct decisions, seriously endangering driving safety. For example, if an obstacle is missed, the intelligent driving terminal may fail to perform obstacle avoidance operations, potentially resulting in a collision. Alternatively, if an obstacle is falsely detected, the intelligent driving terminal may perform an incorrect obstacle avoidance operation. Both of these operations will affect the comfort and safety of the vehicle. We call these rare obstacles or scenarios corner cases or critical information, and they are key issues that need to be addressed for the large-scale application of intelligent driving.

[0004] Summary of the Invention

[0005] This application discloses a key information collection method, device and intelligent driving equipment, which can determine the key information in the driving environment information in combination with the user's biometric characteristics, improve the effectiveness of the collected key information, alleviate the situation of missed and false detection of obstacles, and are more conducive to the planning and implementation of autonomous driving tasks.

[0006] In the first aspect, the present application provides a key information collection method for application in intelligent driving scenarios, the method comprising: obtaining biometric information of a user in an intelligent driving device; obtaining driving environment information of the intelligent driving device; and determining key information in the driving environment information based on the biometric information of the user, wherein the key information is used to assist intelligent driving.

[0007] In the above method, the user in the intelligent driving device may have corresponding reactions to the driving environment, and these reactions can be reflected through the user's biometric information. After obtaining the driving environment information, the key information in the driving environment information is determined through the user's biometric information. The key information may be rare obstacles, so that the user's reaction to the driving environment can participate in the screening of key information in the driving environment information. The effectiveness of the key information obtained is high, which alleviates the situation of missed detection of obstacles. For example, when the user's reaction to the driving environment at a certain moment is more obvious (such as eye gaze, surprised expression, etc.), it indicates that there is a high probability that there are rare obstacles that affect driving in the driving environment at that moment, that is, key information, so that the driving environment information at the current moment or the time period corresponding to the current moment can be saved in time to determine the key information.

[0008] In summary, this solution combines the user's biometrics to identify key information within the driving environment, improving the effectiveness of collected key information and enabling more timely acquisition of key information, reducing missed obstacle detections and facilitating the planning and implementation of autonomous driving missions. Furthermore, this solution can also increase the probability of detecting rare obstacles.

[0009] In one possible implementation, the key information in the driving environment information is determined based on the user's biometric information, including: when the user's expression information is preset expression information, the key information in the driving environment information is determined based on the user's expression information; or, when the user's action information is preset action information, the key information in the driving environment information is determined based on the user's action information; or, when the user's voice information is preset voice information, the key information in the driving environment information is determined based on the user's voice information.

[0010] In this method, the above-mentioned method can improve the user's participation in the key information collection process, and through the preset expressions, preset actions, and preset voices, the user's participation in the collection of key information is made more flexible.

[0011] In another possible implementation, after determining the key information in the driving environment information based on the user's biometric information, the method further includes: obtaining the user's description of the key information; storing the correspondence between the description and the key information, and the correspondence is used for model training.

[0012] In the above method, the description can be a label for the key information marked by the user, which is more conducive to the training of the model and makes the model more accurate.

[0013] In another possible implementation, obtaining the user's description of the key information includes: outputting a first prompt message, where the first prompt message is used to request a description of the key information; receiving a response message to the first prompt message, and obtaining the user's description of the key information based on the response message.

[0014] In another possible implementation, the method further includes: outputting the intelligent driving device's perception of the current driving environment; when the perception result is inconsistent with the result actually observed by the user, determining key information in the driving environment information based on the user's biometric information.

[0015] In the above method, the user's participation in the collection of key information is improved in this way, but it causes more interference to the user. However, by comparing the perception results with the results actually observed by the user, the effectiveness of the key information obtained is higher, avoiding the omission or false detection of key information. Furthermore, the probability of obtaining rare obstacles is increased.

[0016] In another possible implementation, the output of the intelligent driving device's perception result of the current driving environment includes: projecting the perception result through an augmented reality head-up display (AR-HUD); or displaying the perception result through a display screen; or broadcasting the perception result through voice.

[0017] In the above method, through the above manner, the way of outputting the perception results is very flexible and diverse.

[0018] In another possible implementation, the method further includes: outputting a second prompt message, where the second prompt message is used to notify the user that the intelligent driving device needs to obtain the user's biometric information.

[0019] In the above method, by outputting the second prompt information to inform the user that the intelligent driving device needs to obtain the user's biometric information, the user's privacy can be guaranteed.

[0020] In a second aspect, the present application provides an intelligent driving device, which includes: a computing platform, a first acquisition device and a second acquisition device, wherein the computing platform is used to: obtain biometric information of a user in the intelligent driving device through the first acquisition device, where the user's biometric information includes one or more of the following: the user's expression information, the user's movement information, the user's voice information, the user's eye focus information or the user's pupil change information; obtain driving environment information of the intelligent driving device through the second acquisition device; determine key information in the driving environment information based on the user's biometric information, and the key information is used to assist intelligent driving.

[0021] In one possible implementation, when determining the key information in the driving environment information based on the user's biometric information, the computing platform is used to: when the user's expression information is preset expression information, determine the key information in the driving environment information based on the user's expression information; or, when the user's action information is preset action information, determine the key information in the driving environment information based on the user's action information; or, when the user's voice information is preset voice information, determine the key information in the driving environment information based on the user's voice information.

[0022] In another possible implementation, the computing platform is further used to: obtain the user's description of the key information; and store a correspondence between the description and the key information, where the correspondence is used for model training.

[0023] In another possible implementation, when obtaining the user's description of the key information, the computing platform is used to: output a first prompt message, where the first prompt message is used to request a description of the key information; receive a response message to the first prompt message, and obtain the user's description of the key information based on the response message.

[0024] In another possible implementation, the computing platform is also used to: output the intelligent driving device's perception of the current driving environment; when the perception result is inconsistent with the result actually observed by the user, determine the key information in the driving environment information based on the user's biometric information.

[0025] In another possible implementation, when outputting the perception results of the intelligent driving device on the current driving environment, the computing platform is used to: project the perception results through an augmented reality head-up display (AR-HUD); or display the perception results through a display screen; or broadcast the perception results through voice.

[0026] In another possible implementation, the computing platform is further used to: output a second prompt message, where the second prompt message is used to notify the user that the intelligent driving device needs to obtain the user's biometric information.

[0027] Regarding the technical effects brought about by the second aspect or possible implementation methods, reference may be made to the introduction to the technical effects of the first aspect or corresponding implementation methods.

[0028] In the third aspect, the present application provides a key information collection device, which is an on-board computing platform or chip system. The key information collection device includes a processor and a memory. The memory and the processor are interconnected through a line. The processor is used to obtain a computer program stored in the memory. When the computer program is executed by the processor, it is used to implement the above-mentioned first aspect and the methods in various possible implementation methods.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a processor, the method in the above-mentioned first aspect and various possible implementation methods is implemented.

[0030] In a fifth aspect, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are run on a computer, the method in the above-mentioned first aspect and various possible implementation methods is implemented.

[0031] In a sixth aspect, the present application provides a chip, which includes a circuit, and the circuit is used to implement the method in the above-mentioned first aspect and various possible implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a schematic structural diagram of an intelligent driving device provided in an embodiment of the present application;

[0033] FIG2 is a schematic structural diagram of another intelligent driving device provided in an embodiment of the present application;

[0034] FIG3 is a schematic diagram of a driving control device of an intelligent driving device provided in an embodiment of the present application;

[0035] FIG4 is a flow chart of a key information collection method provided in an embodiment of the present application;

[0036] FIG5 is a schematic diagram of driving environment information of an intelligent driving device provided in an embodiment of the present application;

[0037] FIG6 is a schematic diagram of biometric information of a user provided in an embodiment of the present application;

[0038] FIG7 is a schematic diagram of outputting second prompt information provided by an embodiment of the present application;

[0039] FIG8 is a schematic diagram of key information provided by an embodiment of the present application;

[0040] FIG9 is a schematic diagram of first indication information and response information provided in an embodiment of the present application;

[0041] FIG10 is a schematic diagram of another first indication information and response information provided in an embodiment of the present application;

[0042] FIG11 is a schematic diagram of a perception result through AR-HUD projection provided by an embodiment of the present application;

[0043] FIG12 is a schematic diagram of a result of actual observation by a user provided in an embodiment of the present application;

[0044] FIG13 is a schematic structural diagram of a key information collection device provided in an embodiment of the present application;

[0045] FIG14 is a schematic diagram of the structure of an intelligent driving device provided in an embodiment of the present application;

[0046] FIG15 is a schematic structural diagram of another key information collection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of this application.

[0048] References to "one embodiment" or "some embodiments" etc. described in this application mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily all refer to the same embodiment, but rather mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprises", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized. It is understood explicitly and implicitly by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c; a and b; a and c; b and c; or a, b, and c. Among them, a, b, and c can be single or multiple.

[0050] In the description of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0051] In the description of this application, the terms "first," "second," "third," and "fourth" are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or apparatus.

[0052] Please refer to Figure 1, which is a schematic diagram of the structure of an intelligent driving device 100 provided in an embodiment of the present application. The intelligent driving device 100 includes a computing platform 101, a first acquisition device, and a second acquisition device. The first acquisition device can be a driver monitoring system (DMS) 103, and the second acquisition device can be a sensor 102. The computing platform 101 can be understood as a device with computing capabilities. The computing platform 101 is used to obtain biometric information of a user in the intelligent driving device through the first acquisition device / DMS 103. The user's biometric information includes one or more of the following: user's facial expression information, user's movement information, user's voice information, user's eye focus information, or user's pupil change information, and to obtain driving environment information of the intelligent driving device through the second acquisition device / sensor 102. Optionally, the sensor 102 can be a camera, an infrared sensor, a millimeter-wave radar, an ultrasonic radar, a lidar, or a visual sensor. Optionally, the number of sensors 102 can be one or more. When there are multiple sensors 102, the types of the multiple sensors can be different, for example, one sensor can be a camera and one sensor can be a lidar. The computing platform 101 can also be used to determine key information in the driving environment information based on the user's biometric information. Optionally, the DMS 103 can also include an expression recognition subsystem 1031, which is used to obtain the user's expression information; optionally, the DMS 103 can also include a motion recognition subsystem 1032, a voice recognition subsystem 1033 and an eye tracking subsystem 1034, the motion recognition subsystem 1032 is used to obtain the user's motion information, the voice recognition subsystem 1033 is used to obtain the user's voice information; the eye tracking subsystem 1034 is used to obtain the user's eye focus information or the user's pupil change information. Optionally, the motion recognition subsystem 1032, the voice recognition subsystem 1033 and the eye tracking subsystem 1034 may not be included in the DMS 103, but may be specifically included in the smart cockpit (in this case, the first acquisition device includes the DMS 103 and other subsystems in the smart cockpit for obtaining the user's biometric information).

[0053] Optionally, when the DMS 103 includes a motion recognition subsystem 1032, a speech recognition subsystem 1033, and an eye tracking subsystem 1034, the computing platform 101 being used to obtain biometric information of a user in the intelligent driving device can be understood as the computing platform 101 being used to obtain the user's expression information through the expression recognition subsystem 1031 in the DMS 103, obtain the user's motion information through the motion recognition subsystem 1032 in the DMS 103, obtain the user's speech information through the speech recognition subsystem 1033 in the DMS 103, and obtain the user's eye focus information or the user's pupil change information through the eye tracking subsystem 1034 in the DMS 103.

[0054] Optionally, when the motion recognition subsystem 1032, the voice recognition subsystem 1033, and the eye tracking subsystem 1034 are included in the smart cockpit, the computing platform 101 is used to obtain the biometric information of the user in the smart driving device, which can be understood as the computing platform 101 being used to obtain the user's facial expression information through the expression recognition subsystem 1031 in the DMS 103, obtain the user's motion information through the motion recognition subsystem 1032 in the smart cockpit, obtain the user's voice information through the voice recognition subsystem 1033 in the smart cockpit, and obtain the user's eye focus information or the user's pupil change information through the eye tracking subsystem 1034 in the smart cockpit. It should be noted that the smart driving device shown in Figure 1 can be used in the development, commissioning, or operation scenarios of smart driving systems.

[0055] It is understandable that the structure of the intelligent driving device in FIG1 is only an exemplary implementation in the embodiment of the present application, and the intelligent driving device in the embodiment of the present application may further include more components as needed.

[0056] Please refer to Figure 2, which further illustrates the structure of the intelligent driving device 100 based on Figure 1. The intelligent driving device 100 may include various subsystems, such as a travel system 202, a sensor system 204, a control system 206, one or more peripheral devices 208, a power supply 210, a computer system 212, and a user interface 216. Alternatively, the intelligent driving device 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the intelligent driving device 100 may be interconnected via wired or wireless connections.

[0057] The propulsion system 202 may include components that provide power and movement for the intelligent driving device 100. In one embodiment, the propulsion system 202 may include an engine 218, an energy source 219, a transmission 220, and wheels / tires 221. The engine 218 may be an internal combustion engine, an electric motor, an air compression engine, or other engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine 218 converts the energy source 219 into mechanical energy.

[0058] Examples of energy source 219 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 219 can also provide energy for other systems of the intelligent driving device 100.

[0059] Transmission 220 can transmit mechanical power from engine 218 to wheels 221. Transmission 220 can include a gearbox, a differential, and a drive shaft. In one embodiment, transmission 220 can also include other components, such as a clutch. The drive shaft can include one or more shafts that can be coupled to one or more wheels 221.

[0060] The sensor system 204 may include several sensors that sense information about the environment surrounding the intelligent driving device 100. For example, the sensor system 204 may include a positioning system 222 (the positioning system may be a GPS system, a BeiDou system, or other positioning systems), an inertial measurement unit (IMU) 224, a radar 226, a laser rangefinder 228, and a camera 230. The sensor system 204 may also include sensors for the internal systems of the intelligent driving device 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). This detection and recognition is a key function for the safe operation of the autonomous intelligent driving device 100.

[0061] The positioning system 222 may be used to estimate the geographic location of the intelligent driving device 100. The IMU 224 is used to sense the position and orientation changes of the intelligent driving device 100 based on inertial acceleration. In one embodiment, the IMU 224 may be a combination of an accelerometer and a gyroscope.

[0062] Radar 226 can use radio signals to sense objects in the surrounding environment of intelligent driving device 100. In some embodiments, in addition to sensing objects, radar 226 can also be used to sense the speed and / or direction of the objects. Radar 226 can be a laser radar, millimeter wave radar, or ultrasonic radar.

[0063] The laser rangefinder 228 may utilize laser light to sense objects in the environment in which the intelligent driving device 100 is located. In some embodiments, the laser rangefinder 228 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.

[0064] The camera 230 may be used to capture multiple images of the surrounding environment of the intelligent driving device 100. The camera 230 may be a still camera or a video camera. The number of cameras 230 may be one or more.

[0065] The control system 206 controls the operation of the intelligent driving device 100 and its components. The control system 206 may include various components, including a steering system 232 , a throttle 234 , a brake unit 236 , a perception system 240 , and a regulation system 242 .

[0066] The steering system 232 is operable to adjust the forward direction of the intelligent driving device 100. For example, in one embodiment, it can be a steering wheel system.

[0067] The throttle 234 is used to control the operating speed of the engine 218 and thus control the speed of the intelligent driving device 100 .

[0068] Braking unit 236 is used to control the deceleration of intelligent driving device 100. Braking unit 236 can use friction to slow down wheel 221. In other embodiments, braking unit 236 can convert the kinetic energy of wheel 221 into electric current. Braking unit 236 can also take other forms to slow the rotation speed of wheel 221 and thus control the speed of intelligent driving device 100.

[0069] The perception system 240 can be operated to process and analyze images captured by the camera 230 and reflected signals obtained by the radar 226 to identify objects and / or features in the environment surrounding the intelligent driving device 100. These objects and / or features may include obstacles, traffic signals, road boundaries, and obstacles. The perception system 240 can use object recognition, tracking algorithms, multimodal fusion algorithms, and other computer vision technologies. In some embodiments, the perception system 240 can be used to track objects, estimate the speed of objects, and so on. Optionally, the perception system 240 can be the computing platform 101 in Figure 1.

[0070] The control system 242 can be used to navigate the intelligent driving device 100, for example, to determine the driving route of the intelligent driving device 100, and can also perform positioning, mapping, planning, and control-by-wire. In some embodiments, the control system 242 can combine data from the positioning system 222 and one or more predetermined maps to determine the driving route for the intelligent driving device 100. In some embodiments, the control system 242 can use a simultaneous localization and mapping (SLAM) algorithm when mapping the environment.

[0071] Of course, in one example, the control system 206 may include additional or alternative components other than those shown and described, or may also include fewer than some of the components shown.

[0072] The intelligent driving device 100 interacts with external sensors, other vehicles, other computer systems, or users through the peripheral devices 208. The peripheral devices 208 may include a wireless communication system 246, an onboard computer 248, a microphone 250, and / or a speaker 252.

[0073] In some embodiments, the peripheral device 208 provides a means for the user of the intelligent driving device 100 to interact with the user interface 216. For example, the onboard computer 248 can provide information to the user of the intelligent driving device 100. The user interface 216 can also operate the onboard computer 248 to receive user input. The onboard computer 248 can be operated via a touch screen. In other cases, the peripheral device 208 can provide a means for the intelligent driving device 100 to communicate with other devices located in the vehicle. For example, the microphone 250 can receive audio (e.g., voice commands or other audio input) from the user of the intelligent driving device 100. Similarly, the speaker 252 can output audio to the user of the intelligent driving device 100.

[0074] The wireless communication system 246 can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 246 can use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 246 can use Wi-Fi to communicate with a wireless local area network (WLAN). In some embodiments, the wireless communication system 246 can use an infrared link, Bluetooth, or ZigBee to communicate directly with the device. The wireless communication system 246 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0075] The power supply 210 can provide power to the various components of the intelligent driving device 100. In one embodiment, the power supply 210 can be a rechargeable lithium-ion or lead-acid battery. One or more battery packs of such batteries can be configured as a power source to provide power to the various components of the intelligent driving device 100. In some embodiments, the power supply 210 and the energy source 219 can be implemented together.

[0076] Some or all functions of the intelligent driving device 100 are controlled by a computer system 212. The computer system 212 may include at least one processor 213 that executes instructions 215 stored in a non-transitory computer-readable medium, such as a memory 214. The computer system 212 may also be a plurality of computing devices that control individual components or subsystems of the intelligent driving device 100 in a distributed manner.

[0077] The processor 213 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor may be an application-specific integrated circuit (ASIC) or other hardware-based processor.

[0078] In various aspects described herein, the processor can be remote from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle while others are performed by a remote processor.

[0079] In some embodiments, the memory 214 may include instructions 215 (e.g., program logic) that may be executed by the processor 213 to perform various functions of the intelligent driving device 100, including those described above. The memory 214 may also include additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or sending control instructions to one or more of the travel system 202, the sensor system 204, the control system 206, and the peripheral devices 208.

[0080] In addition to instructions 215 , memory 214 may also store data and other information that may be used by the intelligent driving device 100 and computer system 212 during operation of the intelligent driving device 100 in autonomous, semi-autonomous, and / or manual modes.

[0081] User interface 216 is used to provide information to or receive information from a user of intelligent driving device 100. Optionally, user interface 216 may include one or more input / output devices within the set of peripheral devices 208, such as wireless communication system 246, onboard computer 248, microphone 250, and speaker 252.

[0082] The computer system 212 can control the functions of the intelligent driving device 100 based on input received from various subsystems (e.g., the travel system 202, the sensor system 204, and the control system 206) and from the user interface 216. For example, the computer system 212 can use input from the control system 206 to control the steering system 232 to avoid an obstacle detected by the sensor system 204. In some embodiments, the computer system 212 is operable to provide control over many aspects of the intelligent driving device 100 and its subsystems.

[0083] Optionally, one or more of the above components may be installed or associated separately from the intelligent driving device 100. For example, the memory 214 may be partially or completely separate from the intelligent driving device 100. The above components may be communicatively coupled together in a wired and / or wireless manner.

[0084] Optionally, the above components are only an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 2 should not be understood as a limitation to the embodiments of the present application.

[0085] The intelligent driving device 100 may be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawn mower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, cart, smart home device, etc., and the embodiments of the present application do not impose any particular limitation.

[0086] It can be understood that the structure of the intelligent driving device in Figure 2 is only an exemplary implementation in the embodiment of the present application. The intelligent driving device in the embodiment of the present application includes but is not limited to the above structure.

[0087] Please refer to Figure 3, which is a schematic diagram of a driving control device for an intelligent driving device provided in an embodiment of the present application. This device, as used in Figure 2 above, is equivalent to the computer system 212 shown in Figure 2 and may include a processor 213 coupled to a system bus 305. Processor 213 may be one or more processors, each of which may include one or more processor cores. Memory 214 may store relevant data information and is coupled to system bus 305. Computer system 212 also includes a display adapter 307, which may drive a display 309, which is coupled to system bus 305. System bus 305 is coupled to input / output (I / O) bus 313 via bus bridge 301. An I / O interface 315 is coupled to I / O bus 313. The I / O interface 315 communicates with various I / O devices, such as input device 317 (e.g., keyboard, mouse, touch screen, etc.) and media tray 321 (e.g., CD-ROM, multimedia interface, etc.). The transceiver 323 (can send and / or receive radio communication signals), the camera 355 (can capture scene and dynamic digital video images) and the external USB interface 325. Optionally, the interface connected to the I / O interface 315 can be a USB interface.

[0088] Processor 213 may be any conventional processor, including a reduced instruction set computer (RISC), a complex instruction set computer (CISC), or a combination thereof. Alternatively, the processor may be an ASIC. Alternatively, processor 213 may be a neural network processor, or a combination of a neural network processor and the conventional processors described above.

[0089] Alternatively, in various embodiments described herein, the computer system 212 may be located remotely from the intelligent driving device and may communicate wirelessly with the intelligent driving device. In other aspects, some of the processes described herein are executed on a processor within the intelligent driving device, while others are executed by a remote processor.

[0090] Computer system 212 can communicate with software deployment server 349 via network interface 329. Network interface 329 is a hardware network interface, such as a network card. Network 327 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Alternatively, network 327 can be a wireless network, such as a Wi-Fi network or a cellular network.

[0091] The transceiver 323 (which can send and / or receive radio communication signals) can use various wireless communication methods including but not limited to the second generation mobile communication networks (2G), 3G, 4G, 5G, etc., or DSRC technology, or long-term evolution vehicle-to-everything (LTE-V2X) technology, etc. Its main function is to receive information data sent by external devices and send the information data generated by the intelligent driving device when driving on the target road section back to the external device for storage and analysis.

[0092] The hard disk drive interface 331 is coupled to the system bus 305. The hard disk drive interface 331 is connected to the hard disk drive 333. The system memory 335 is coupled to the system bus 305. The data running in the system memory 335 may include the operating system 337 and application programs 343 of the computer system 212.

[0093] The system memory 335 is coupled to the system bus 305. For example, in the present application, the system memory 335 can be used to store the driving information of the vehicle passing through the target road section in a certain format.

[0094] Operating system (OS) 337 includes a shell 339 and a kernel 341. Shell 339 is an interface between the user and the operating system's kernel. Shell 339 is the outermost layer of the operating system. It manages the interaction between the user and the operating system: it waits for user input; interprets user input to the operating system; and processes various operating system outputs.

[0095] The kernel 341 consists of the parts of the operating system that manage memory, files, peripherals, and system resources. It interacts directly with the hardware and typically runs processes and provides inter-process communication, CPU time slice management, interrupts, memory management, I / O management, and more.

[0096] Applications 343 include autonomous driving-related programs 347, such as programs that manage the autonomous vehicle's interactions with obstacles on the road, control the autonomous vehicle's route or speed, and control the autonomous vehicle's interactions with other autonomous vehicles on the road. Applications 343 also reside on a system called a deploying server 349. In one embodiment, when autonomous driving-related programs 347 are needed, computer system 212 can download them from deploying server 349.

[0097] Sensor 353 is associated with computer system 212. Sensor 353 is used to detect the environment surrounding computer system 212. For example, sensor 353 can detect animals, cars, obstacles, and crosswalks. Furthermore, the sensor can also detect the environment surrounding these objects, such as the surrounding environment of the animals, such as other animals around the animals, weather conditions, and ambient light levels. Alternatively, if computer system 212 is located in a self-driving car, the sensor can be a camera, infrared sensor, lidar, visual sensor, chemical detector, microphone, etc. Alternatively, sensor 353 can be sensor 102 in FIG. 1 .

[0098] It can be understood that the driving control device of the intelligent driving device in Figure 3 is only an exemplary implementation in the embodiment of the present application. The driving control device applied to the intelligent driving device in the embodiment of the present application includes but is not limited to the above structure.

[0099] At present, the obstacle information collection method is also called the shadow mode collection method, that is, a variety of scenarios are deployed on the intelligent driving device to trigger the collection of obstacle information, for example, when the autonomous driving algorithm decision and the manual driving decision are inconsistent, the obstacle information collection is triggered; when the sensor perception results are inconsistent, the obstacle information collection is triggered; when the vehicle brakes suddenly or turns sharply, the obstacle information collection is triggered; however, the above-mentioned passive obstacle information collection method is prone to missed detection and false detection. In order to solve the above problems, the embodiments of the present application propose the following solutions.

[0100] It should be understood that the intelligent driving mentioned below may include intelligent assisted driving, automatic driving, and unmanned driving, etc.

[0101] Please refer to Figure 4, which is a flow chart of a key information collection method provided by an embodiment of the present application. Optionally, the method is applied to the device of Figure 1, Figure 2, or Figure 3, for example, to the intelligent driving device shown in Figure 1 above, wherein the computing platform can be used to support and execute the method flow steps S401 and S403 shown in Figure 4, and the sensor is used to support and execute the method flow step S402 shown in Figure 4. The method may include the following steps S401-S403, as follows:

[0102] Step S401: Obtain biometric information of the user in the intelligent driving device.

[0103] Optionally, the users in the intelligent driving device may include a driver, and the users in the intelligent driving device may also include a co-pilot.

[0104] Specifically, biometric information includes one or more of the following: user's facial expression information, user's motion information, user's voice information, user's eye focus information, or user's pupil change information. Optionally, the user's eye focus information can also be referred to as the user's eye gaze point information. The user's facial expression information can include one or more of the following: excitement, enjoyment, surprise, pain, fear, humiliation, disgust, or anger. The user's motion information can include one or more of the following: blinking twice in succession, opening the mouth wide, touching the nose with a hand, touching the ear with a hand, or touching the mouth with a hand. The user's voice information is used to describe information about the current driving environment. For example, it can include one or more of the following: the user saying "there's a car accident," the user saying "there's a landslide," the user saying "construction ahead," or the user saying "road repair ahead." The user's pupil change information can include pupil shrinkage or pupil enlargement.

[0105] Among them, the biometric information reflects the user's reaction to the driving environment information of the intelligent driving device, that is, the reaction can be the user's expression, action, voice, eye focus or pupil change, etc. In one example, please refer to Figure 5, which is a schematic diagram of the driving environment information of an intelligent driving device provided by an embodiment of the present application, and refer to Figure 6, which is a schematic diagram of the biometric information of a user provided by an embodiment of the present application. Under the driving environment information of Figure 5, the user says "Wow, wow, wow" in voice, and the user shows a surprised expression (for example, the user widens his eyes and opens his mouth wide), wherein the user's biometric information includes the user's voice information and the user's expression information, the user's voice information is "Wow, wow, wow", and the user's expression information is surprise.

[0106] Optionally, the computing platform is used to obtain the biometric information of the user in the intelligent driving device, which can be understood as the computing platform obtaining the user's expression information through the expression recognition subsystem in the DMS, obtaining the user's motion information through the motion recognition subsystem in the DMS, obtaining the user's voice information through the voice recognition subsystem in the DMS, and obtaining the user's eye focus information or the user's pupil change information through the eye tracking subsystem in the DMS.

[0107] Optionally, when the DMS includes an expression recognition subsystem, and a motion recognition subsystem, a voice recognition subsystem, and an eye tracking subsystem are included in the smart cockpit, the computing platform used to obtain the biometric information of the user in the smart driving device can be understood as the computing platform obtaining the user's expression information through the expression recognition subsystem in the DMS, obtaining the user's motion information through the motion recognition subsystem in the smart cockpit, obtaining the user's voice information through the voice recognition subsystem in the smart cockpit, and obtaining the user's eye focus information or the user's pupil change information through the eye tracking subsystem in the smart cockpit.

[0108] In a possible implementation, before obtaining the biometric information of the user in the intelligent driving device, the method further includes: outputting second prompt information, where the second prompt information is used to notify the user that the intelligent driving device needs to obtain the biometric information of the user.

[0109] Optionally, the second prompt information can be voice, text, or an identifier. In one example, the system voice broadcasts "We need to obtain your biometric information. Do you agree?" The content of the system voice broadcast is the second indication information. In another example, please refer to Figure 7, which is a schematic diagram of outputting the second prompt information proposed in an embodiment of the present application. "User Privacy Agreement" is displayed on the display screen. The user privacy agreement includes consent to obtain the user's biometric information. The user agreement is the second indication information.

[0110] In the above method, by outputting the second prompt information to inform the user that the intelligent driving device needs to obtain the user's biometric information, the user's privacy can be guaranteed.

[0111] Step S402: Acquire driving environment information of the intelligent driving device.

[0112] The computing platform obtains driving environment information of the smart device through sensors. Optionally, the sensor can be a camera, an infrared sensor, a millimeter-wave radar, an ultrasonic radar, a lidar, or a visual sensor. The number of sensors can be one or more. When there are multiple sensors, the types of the multiple sensors can be different, for example, one sensor can be a camera and one sensor can be a lidar. The driving environment information can include traffic lights, lane markings, traffic signs, animals, plants, vehicles, crosswalks, or pedestrians.

[0113] Optionally, while the intelligent driving device is in motion, the sensor continuously acquires information about the driving environment of the intelligent driving device. Optionally, the intelligent driving device may include a storage system configured to store the driving environment information of the intelligent driving device for a period of time. The period of time may be user-configured or protocol-specified. In one example, the storage system is configured to store three hours of driving environment information of the intelligent driving device.

[0114] Step S403: Determine key information in the driving environment based on the user's biometric information.

[0115] This critical information is used to assist driving. This critical information can be information that influences driving decisions. Optionally, this critical information can include, but is not limited to, rare obstacles. For example, a rare obstacle could be a tree blown down by a typhoon, an overturned truck, or two cars involved in a traffic accident. Optionally, this method of determining critical information in the driving environment based on the user's biometric information can be classified as Level 1.

[0116] Determining key information in the driving environment based on the user's biometric information can be understood as determining and saving the driving environment information at the current moment or within the time period corresponding to the current moment based on the user's biometric information, with the driving environment information including key information. For example, when a user's reaction to the driving environment at a certain moment is more obvious (e.g., eye contact, a surprised expression, etc.), it indicates that there is a high probability of a rare obstacle in the driving environment at that moment that may affect driving, i.e., key information. Therefore, the driving environment information at the current moment or within the time period corresponding to the current moment can be promptly saved, and the driving environment information including key information can be saved. In one example, at time T1, the user is gazing at a tree lying across the road, which was blown down by a typhoon. In this example, the user's biometric information is the user's eye focus information. Determining key information in the driving environment based on the user's eye focus information means determining and saving the driving environment information at time T1 or within the time period corresponding to time T1 based on the user's eye focus information. The driving environment information includes key information, which is the tree lying across the road.

[0117] Optionally, after determining key information in the driving environment based on the user's biometric information, the key information can be stored. The key information can be used for model training. Optionally, the trained model can be deployed on the intelligent driving device. Optionally, the model can be a three-dimensional target detection model, which can be implemented by a neural network model. The input of the neural network model can be the driving environment information of the intelligent driving device, for example, the driving environment information can be a point cloud, an image, or ranging information, and the output of the neural network can be the three-dimensional information of the obstacle, such as the position xyz, orientation angle, or length, width, height, etc. In one example, the key information is a tree lying across the road. This key information is used for model training, and then the trained model can be deployed on the intelligent driving device. In this way, iterative training and model deployment are repeated, and when the intelligent driving device encounters a "tree lying across the road", the trained model may be able to recognize this situation.

[0118] In one example, please refer to Figure 5, which is a schematic diagram of the driving environment information of an intelligent driving device provided by an embodiment of the present application. Please refer to Figure 6, which is a schematic diagram of the biometric information of a user provided by an embodiment of the present application. Under the driving environment information of Figure 5, the user says "Wow wow wow" in voice, and the user shows a surprised expression (for example, the user widens his eyes and opens his mouth), wherein the user's biometric information includes the user's voice information and the user's expression information, and the user's voice information is "Wow wow wow", and the user's expression information is surprise. Then under the driving environment information of Figure 5 and the user's biometric information of Figure 6, the key information in the driving environment can be determined based on the user's biometric information. Please refer to Figure 8, which is a schematic diagram of key information provided by an embodiment of the present application.

[0119] In this method, the key information in the driving environment information is determined through the user's biometric information, so that the user's response to the driving environment can participate in the screening of the key information in the driving environment information. The key information obtained is highly effective, which alleviates the situation of missed detection and false detection of obstacles, and is completely non-interfering to the driver.

[0120] Furthermore, after the trained model is deployed on the intelligent driving device, rare obstacles that the previous model could not handle are solved, but new rare obstacles may be encountered. Therefore, key information can continue to be obtained according to the method of this application, and iterative training and model deployment can be carried out. This cycle allows the model to handle more and more rare obstacles and the model processing capabilities to become more and more powerful.

[0121] In another possible implementation, when the user's facial expression information is preset facial expression information, the key information in the driving environment information is determined based on the user's facial expression information; or, when the user's action information is preset action information, the key information in the driving environment information is determined based on the user's action information; or, when the user's voice information is preset voice information, the key information in the driving environment information is determined based on the user's voice information.

[0122] Optionally, the preset expression information, preset action information, and preset voice information may be set by the user, or may be specified by the system or a protocol.

[0123] In one example, the preset action information is blinking twice continuously, and the user's action information is the driver blinking twice continuously. Based on the user's action information, it is determined that the current driving environment information contains key information. Optionally, after determining the key information, the current driving environment information, information collected by sensors in the intelligent driving device, the current status information of the intelligent driving device, or the algorithm working information in the intelligent driving device can be saved.

[0124] In this method, the above-mentioned method can improve the user's participation in the key information collection process, and through the preset expressions, preset actions, and preset voices, the user's participation in the collection of key information is made more flexible.

[0125] In another possible implementation, after determining the key information in the driving environment based on the user's biometric information, the method further includes: storing the user's description of the key information; storing the correspondence between the description and the key information, and the correspondence is used for model training.

[0126] The description can be voice, text, or an identifier, and the description includes scene information. For example, the description can be a label for key information, and the label is annotated by the user. The correspondence between the description and key information can be the training data of the model. Optionally, the model can be a three-dimensional object detection model. In one example, the user's description of the key information can be "There is a dog crossing the road"; in another example, the user's description of the key information can be "There was a car accident ahead."

[0127] Optionally, obtaining the user's description of the key information includes: outputting first prompt information, where the first prompt information is used to request a description of the key information; receiving response information to the first prompt information, and obtaining the user's description of the key information according to the response information.

[0128] Optionally, the first prompt information can be voice or text, and the response information can be voice or text. In one example, please refer to Figure 9, which is a schematic diagram of a first indication information and response information proposed in an embodiment of the present application. The system voice broadcasts "What just happened?", and the "What just happened" is the first prompt information. The user voice says "A car accident occurred." The "A car accident occurred" is the response information. In another example, please refer to Figure 10, which is a schematic diagram of another first indication information and response information proposed in an embodiment of the present application. "What just happened" is displayed on the display screen of the intelligent driving device. The "What just happened" is the first prompt information. The user enters "A car accident occurred" on the display screen, and the "A car accident occurred" is the response information.

[0129] In another possible implementation, the method further includes: outputting the intelligent driving device's perception result of the current driving environment; when the perception result is inconsistent with the user's actual observation result, determining key information in the driving environment information based on the user's biometric information.

[0130] Among them, the perception result of the intelligent driving device on the current driving environment can be determined by the intelligent driving device based on a perception algorithm.

[0131] Among them, outputting the intelligent driving device's perception results of the current driving environment includes: projecting the perception results through an augmented reality-head up display (AR-HUD); or displaying the perception results through a display screen; or broadcasting the perception results through voice. Optionally, when projecting the perception results through the AR-HUD, the activation of the AR-HUD can be determined by the user. Optionally, when projecting the perception results through the AR-HUD, the perception results can be projected onto the front windshield of the intelligent driving device.

[0132] When the perception result is inconsistent with the user's actual observation, a certain user action is triggered, allowing the intelligent driving device to determine that the perception result is inconsistent with the user's actual observation. For example, the user's action trigger can be staring at the missed or misdetected obstacle, or triggering it with an action (such as blinking twice); or the user can trigger it by announcing "missed or misdetected" by voice.

[0133] Optionally, when the perception result is inconsistent with the result of the user's actual observation, the key information in the driving environment information is determined based on the user's biometric information. This can be understood as follows: when the perception result is inconsistent with the result of the user's actual observation and the user's facial expression information is preset facial expression information, the key information in the driving environment information is determined based on the user's facial expression information; or, when the perception result is inconsistent with the result of the user's actual observation and the user's action information is preset action information, the key information in the driving environment information is determined based on the user's action information; or, when the perception result is inconsistent with the result of the user's actual observation and the user's voice information is preset voice information, the key information in the driving environment information is determined based on the user's voice information. Optionally, this method of determining key information can be divided into level three. Optionally, the higher the level, the higher the validity of the key information obtained, but the more disturbing it is to the driver.

[0134] In an example, please refer to Figure 11, which is a schematic diagram of a perception result projected through AR-HUD provided in an embodiment of the present application. Please refer to Figure 12, which is a schematic diagram of a user's actual observation result provided in an embodiment of the present application, wherein the perception result is a pedestrian, and the user's actual observation result is a pedestrian and a dog. The perception algorithm missed the dog, and the perception result is inconsistent with the user's actual observation result. For example, the preset action is that the driver blinks twice in succession, and the user's action information is blinking twice in succession. The key information in the driving environment information is determined based on the user's action information.

[0135] This approach enhances user engagement in the collection of key information, albeit with significant user disruption. However, by comparing the perception results with the user's actual observations, the key information obtained is more effective, avoiding missed or false detections. Furthermore, the probability of detecting rare obstacles is increased. To a certain extent, users become testers of the perception algorithm, performing quality checks and loopback corrections. Automakers could consider implementing incentive mechanisms to encourage user participation.

[0136] In the method described in FIG4 , the user in the driving device may have corresponding reactions to the driving environment, and these reactions can be reflected through the user's biometric information. After obtaining the driving environment information, the user's biometric information is used to determine key information in the driving environment information. This key information can be rare obstacles. This allows the user's reaction to the driving environment to participate in the screening of key information in the driving environment information. The obtained key information is highly effective, and the situation of missed obstacle detection is alleviated. For example, when the user's reaction to the driving environment at a certain moment is more obvious (such as eye contact, expression of surprise, etc.), it indicates that there is a high probability of rare obstacles affecting driving in the driving environment at that moment, i.e., key information. Therefore, the driving environment information at the current moment or the time period corresponding to the current moment can be saved in a timely manner to determine the key information. In summary, this solution combines the user's biometrics to determine the key information in the driving environment information, which can improve the effectiveness of the collected key information and obtain key information more timely, alleviating the situation of missed obstacle detection, which is beneficial to the planning and implementation of autonomous driving tasks. Furthermore, this solution can also increase the probability of obtaining rare obstacles. For example, for users, daily driving routes are relatively fixed (commuting, picking up children, etc.), and they may always encounter the same rare obstacles. By using the embodiments of the present application, these rare obstacles can be identified, making intelligent driving smoother and reducing risks.

[0137] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.

[0138] Please refer to FIG. 13 , which is a schematic diagram of the structure of a key information collection device 1300 provided in an embodiment of the present application. The key information collection device 1300 may be an on-board computing platform. The key information collection device 1300 may include a processing unit 1301 and a communication unit 1302 . The details of each unit are as follows:

[0139] The processing unit 1301 is used to perform data processing. The communication unit 1302 can implement corresponding communication functions. The communication unit 1302 can also be called a communication interface or a communication module.

[0140] Optionally, the key information collection device 1300 may further include a storage unit, which may be used to store instructions and / or data. The processing unit 1301 may read the instructions and / or data in the storage module to implement the aforementioned method embodiment.

[0141] The key information collection device 1300 can be used to perform the actions performed by the intelligent driving device in the above method embodiments. The key information collection device 1300 can be an intelligent driving device or a component that can be configured in an intelligent driving device. The processing unit 1301 is used to perform processing-related operations of the intelligent driving device in the above method embodiments. The communication unit 1302 is used to perform communication-related operations of the intelligent driving device in the above method embodiments.

[0142] Optionally, the communication unit 1302 may include a sending unit and a receiving unit. The sending unit is configured to perform the sending operation in the above method embodiment. The receiving unit is configured to perform the receiving operation in the above method embodiment.

[0143] It should be noted that the key information collection device 1300 may include a sending unit but not a receiving unit. Alternatively, the key information collection device 1300 may include a receiving unit but not a sending unit. The specific implementation depends on whether the above-mentioned solution executed by the key information collection device 1300 includes both sending and receiving actions.

[0144] Optionally, the key information collection device 1300 is used to perform the actions performed by the intelligent driving device in the embodiment shown in FIG4 above. For details, please refer to the relevant introduction of the embodiment shown in FIG4 above, which will not be expanded in detail here. For example, the key information collection device 1300 is used to perform the following scheme:

[0145] Processing unit 1301 is configured to obtain biometric information of a user in the intelligent driving device, where the biometric information of the user includes one or more of the following: facial expression information of the user, movement information of the user, voice information of the user, eye focus information of the user, or pupil change information of the user;

[0146] The processing unit 1301 is configured to obtain driving environment information of the intelligent driving device;

[0147] The processing unit 1301 is used to determine key information in the driving environment information based on the user's biometric information, where the key information is used to assist intelligent driving.

[0148] In one possible implementation, the processing unit 1301 is used to determine the key information in the driving environment information based on the user's facial expression information when the user's facial expression information is preset facial expression information; or, the processing unit 1301 is used to determine the key information in the driving environment information based on the user's action information when the user's action information is preset action information; or, the processing unit 1301 is used to determine the key information in the driving environment information based on the user's voice information when the user's voice information is preset voice information.

[0149] In another possible implementation, the processing unit 1301 is further used to obtain the user's description of the key information; the storage unit is used to store the correspondence between the description and the key information, and the correspondence is used for model training.

[0150] In another possible implementation, the communication unit 1302 is used to output a first prompt message, where the first prompt message is used to request a description of the key information; the communication unit 1302 is used to receive a response message to the first prompt message, and obtain the user's description of the key information based on the response message.

[0151] In another possible implementation, the communication unit 1302 is further used to output the perception result of the intelligent driving device on the current driving environment; the processing unit 1301 is further used to determine the key information in the driving environment information based on the user's biometric information when the perception result is inconsistent with the result actually observed by the user.

[0152] In another possible implementation, the communication unit 1302 is configured to project the perception result through an augmented reality head-up display (AR-HUD); or display the perception result through a display screen; or announce the perception result through voice.

[0153] In another possible implementation, the communication unit 1302 is further configured to output a second prompt message, where the second prompt message is configured to notify the user that the intelligent driving device needs to obtain the user's biometric information.

[0154] It should be noted that the implementation and beneficial effects of each unit may also correspond to the corresponding description of the method embodiment shown in FIG. 4 .

[0155] It should be understood that the specific process of each module executing the above corresponding process has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0156] The processing unit 1301 in the above embodiment can be implemented by at least one processor or processor-related circuits. The communication unit 1302 can be implemented by a transceiver or transceiver-related circuits. The communication unit 1302 can also be referred to as a communication module or communication interface. The storage module can be implemented by at least one memory.

[0157] Please refer to FIG. 14 , which is a schematic diagram of the structure of an intelligent driving device 1400 provided in an embodiment of the present application. The intelligent driving device 1400 includes a computing platform 1401 , a first acquisition device 1402 , and a second acquisition device 1403 . The computing platform 1401 is used to:

[0158] Acquiring, through the first acquisition device 1402, biometric information of a user in the intelligent driving device, the biometric information of the user including one or more of the following: facial expression information of the user, action information of the user, voice information of the user, eye focus information of the user, or pupil change information of the user;

[0159] Acquiring the driving environment information of the intelligent driving device through the second acquisition device 1403;

[0160] Key information in the driving environment information is determined based on the user's biometric information, and the key information is used to assist intelligent driving.

[0161] The first acquisition device 1402 may be a DMS, and the second acquisition device 1403 may be a sensor.

[0162] In one possible implementation, when determining the key information in the driving environment information based on the user's biometric information, the computing platform 1401 is used to: when the user's expression information is preset expression information, determine the key information in the driving environment information based on the user's expression information; or, when the user's action information is preset action information, determine the key information in the driving environment information based on the user's action information; or, when the user's voice information is preset voice information, determine the key information in the driving environment information based on the user's voice information.

[0163] In another possible implementation, the computing platform 1401 is further used to: obtain the user's description of the key information; and store a correspondence between the description and the key information, where the correspondence is used for model training.

[0164] In another possible implementation, when obtaining the user's description of the key information, the computing platform 1401 is used to: output a first prompt message, which is used to request a description of the key information; receive response information to the first prompt message, and obtain the user's description of the key information based on the response information.

[0165] In another possible implementation, the computing platform 1401 is also used to: output the perception result of the intelligent driving device on the current driving environment; when the perception result is inconsistent with the result actually observed by the user, determine the key information in the driving environment information based on the user's biometric information.

[0166] In another possible implementation, when outputting the perception result of the intelligent driving device on the current driving environment, the computing platform 1401 is used to: project the perception result through an augmented reality head-up display AR-HUD; or display the perception result through a display screen; or broadcast the perception result through voice.

[0167] In another possible implementation, the computing platform 1401 is further used to: output a second prompt message, where the second prompt message is used to notify the user that the intelligent driving device needs to obtain the user's biometric information.

[0168] It should be noted that the implementation and beneficial effects of each unit may also correspond to the corresponding description of the method embodiment shown in FIG4 .

[0169] Please refer to Figure 15, which is a key information collection device 1500 provided in an embodiment of the present application. The key information collection device 1500 can be a vehicle-mounted computing platform. The key information collection device 1500 includes at least one processor 1501 and a communication interface 1503, and optionally, also includes a memory 1502. The processor 1501, memory 1502 and communication interface 1503 are interconnected through a bus 1504.

[0170] Memory 1502 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). Memory 1502 is used for storing computer programs and data. Communication interface 1503 is used to receive and send data.

[0171] The processor 1501 may be one or more central processing units (CPUs). When the processor 1501 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0172] The processor 1501 in the key information collection device 1500 is used to read the computer program or instructions stored in the memory 1502 and perform the following operations:

[0173] Processor 1501 is configured to obtain biometric information of a user in the intelligent driving device, where the biometric information of the user includes one or more of the following: facial expression information of the user, movement information of the user, voice information of the user, eye focus information of the user, or pupil change information of the user;

[0174] Processor 1501, configured to obtain driving environment information of the intelligent driving device;

[0175] Processor 1501 is used to determine key information in the driving environment information based on the user's biometric information, where the key information is used to assist intelligent driving.

[0176] In one possible implementation, the processor 1501 is used to determine the key information in the driving environment information based on the user's facial expression information when the user's facial expression information is preset facial expression information; or, the processor 1501 is used to determine the key information in the driving environment information based on the user's action information when the user's action information is preset action information; or, the processor 1501 is used to determine the key information in the driving environment information based on the user's voice information when the user's voice information is preset voice information.

[0177] In another possible implementation, the processor 1501 is further configured to obtain the user's description of the key information; and the storage unit is configured to store a correspondence between the description and the key information, where the correspondence is used for model training.

[0178] In another possible implementation, the processor 1501 is used to output a first prompt message through the communication interface 1503, where the first prompt message is used to request a description of the key information; the processor 1501 is used to receive response information to the first prompt message through the communication interface 1503, and obtain the user's description of the key information based on the response information.

[0179] In another possible implementation, the processor 1501 is further used to output the perception result of the intelligent driving device on the current driving environment through the communication interface 1503; the processor 1501 is further used to determine the key information in the driving environment information based on the user's biometric information when the perception result is inconsistent with the result actually observed by the user.

[0180] In another possible implementation, the processor 1501 is configured to project the perception result through an augmented reality head-up display (AR-HUD) via the communication interface 1503; or display the perception result through a display screen; or announce the perception result through voice.

[0181] In another possible implementation, the processor 1501 is further used to output a second prompt message through the communication interface 1503, where the second prompt message is used to notify the user that the intelligent driving device needs to obtain the user's biometric information.

[0182] It should be noted that the implementation and beneficial effects of each operation may also correspond to the corresponding description of the method embodiment shown in FIG. 4 .

[0183] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program or instruction. When the computer program or instruction runs on a processor, it implements the method performed by the intelligent driving device in the above method embodiment.

[0184] An embodiment of the present application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed on a processor, the method performed by the intelligent driving device in the above method embodiment is implemented.

[0185] An embodiment of the present application also provides a chip, which includes a circuit, and the circuit is used to implement the method executed by the intelligent driving device in the above method embodiment.

[0186] In the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

Claims

1. A key information collection method, applied in intelligent driving scenarios, characterized in that: include: Acquire biometric information of a user in the intelligent driving device, where the biometric information of the user includes one or more of the following: facial expression information of the user, action information of the user, voice information of the user, eye focus information of the user, or pupil change information of the user; Acquiring driving environment information of the intelligent driving device; Key information in the driving environment information is determined according to the biometric information of the user, and the key information is used to assist intelligent driving.

2. The method according to claim 1, characterized in that The determining of key information in the driving environment information according to the biometric information of the user includes: When the user's expression information is preset expression information, determining key information in the driving environment information according to the user's expression information; or, When the user's motion information is preset motion information, determining key information in the driving environment information according to the user's motion information; or, When the voice information of the user is preset voice information, key information in the driving environment information is determined according to the voice information of the user.

3. The method according to claim 1 or 2, characterized in that: After determining the key information in the driving environment information according to the biometric information of the user, the method further includes: Obtaining a description of the key information by the user; The correspondence between the description and the key information is stored, and the correspondence is used for model training.

4. The method according to claim 3, characterized in that: The obtaining the user's description of the key information includes: Outputting first prompt information, where the first prompt information is used to request a description of the key information; Receive response information to the first prompt information, and obtain the user's description of the key information according to the response information.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: Outputting the perception result of the intelligent driving device on the current driving environment; When the perception result is inconsistent with the result actually observed by the user, key information in the driving environment information is determined according to the biometric information of the user.

6. The method according to claim 5, characterized in that The outputting of the perception result of the intelligent driving device on the current driving environment includes: Projecting the perception result through an augmented reality head-up display (AR-HUD); or Displaying the perception result via a display screen; or The perception result is reported by voice.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Output second prompt information, where the second prompt information is used to notify the user that the intelligent driving device needs to obtain the user's biometric information.

8. An intelligent driving device, characterized in that: The intelligent driving device comprises: a computing platform, a first acquisition device and a second acquisition device, wherein the computing platform is used to: Acquiring, by the first acquisition device, biometric information of a user in the intelligent driving device, the biometric information of the user including one or more of the following: facial expression information of the user, action information of the user, voice information of the user, eye focus information of the user, or pupil change information of the user; Acquiring the driving environment information of the intelligent driving device through the second acquisition device; Key information in the driving environment information is determined according to the biometric information of the user, and the key information is used to assist intelligent driving.

9. The intelligent driving device according to claim 8, characterized in that: When determining key information in the driving environment information according to the biometric information of the user, the computing platform is used to: When the user's expression information is preset expression information, determining key information in the driving environment information according to the user's expression information; or, When the user's motion information is preset motion information, determining key information in the driving environment information according to the user's motion information; or, When the voice information of the user is preset voice information, key information in the driving environment information is determined according to the voice information of the user.

10. The intelligent driving device according to claim 8 or 9, characterized in that: The computing platform is also used to: Obtaining a description of the key information by the user; The correspondence between the description and the key information is stored, and the correspondence is used for model training.

11. The intelligent driving device according to claim 10, characterized in that: When obtaining the user's description of the key information, the computing platform is used to: Outputting first prompt information, where the first prompt information is used to request a description of the key information; Receive response information to the first prompt information, and obtain the user's description of the key information according to the response information.

12. The intelligent driving device according to claim 8 or 9, characterized in that: The computing platform is also used to: Outputting the perception result of the intelligent driving device on the current driving environment; When the perception result is inconsistent with the result actually observed by the user, key information in the driving environment information is determined according to the biometric information of the user.

13. The intelligent driving device according to claim 12, characterized in that: When outputting the perception result of the intelligent driving device on the current driving environment, the computing platform is used to: Projecting the perception result through an augmented reality head-up display (AR-HUD); or Displaying the perception result via a display screen; or The perception result is reported by voice.

14. The intelligent driving device according to any one of claims 8 to 13, characterized in that: The computing platform is also used to: Output second prompt information, where the second prompt information is used to notify the user that the intelligent driving device needs to obtain the user's biometric information.

15. A key information collection device, characterized in that: It comprises a processor and a memory, wherein the memory and the processor are interconnected via a line, and the processor is used to obtain a computer program stored in the memory. When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

16. The key information collection device according to claim 15, characterized in that: The key information collection device is a vehicle-mounted computing platform.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, which, when executed on a processor, implements the method according to any one of claims 1 to 7.

18. A computer program product, characterized in that The computer program product includes a computer program or instructions, and when the computer program or instructions are executed on a computer, the method according to any one of claims 1 to 7 is implemented.

Images