Vehicle interaction method and device, equipment and storage medium

Abstract

The invention discloses a vehicle interaction method and device, equipment and a storage medium, and relates to the technical field of vehicles. The method comprises the following steps: acquiring eye movement information and / or semantic information of a user driving a vehicle; determining correspondingly triggered interaction operation according to the eye movement information and / or the semantic information, responding to the interaction operation and generating projection information incident to the human eyes; and when it is detected that the vehicle is in a non-linear driving state, the projection information is hidden or shrunk to be displayed in the corner area. Through the technical means, the problem that the eyes of a driver leave the front of the vehicle when the driver interacts with the vehicle in the prior art is solved, the driving risk is reduced, and safe driving of the vehicle is guaranteed.

Description

Technical Field

[0001] This application relates to the technical field of vehicles, and particularly to a vehicle interaction method, device, equipment, and storage medium. Background Art

[0002] With the rapid development of vehicle technology, vehicles are equipped with a variety of driving functions, such as navigation, answering calls, and playing music. When driving a vehicle, the driver can interact with the vehicle to use various driving functions provided by the vehicle, greatly improving the driving experience of the driver when driving the vehicle.

[0003] In the prior art, interaction with the vehicle can be achieved through contact operations or voice operations. However, when using contact operations, the driver's eyes will leave the front of the vehicle, resulting in a relatively high driving risk. And voice operations can only perform simple interaction operations. When it comes to complex interactions that require selection, the driver's eyes will also leave the front of the vehicle, unable to ensure safe driving. Summary of the Invention

[0004] This application provides a vehicle interaction method, device, equipment, and storage medium to solve the problem that the driver's eyes will leave the front of the vehicle when interacting with the vehicle in the prior art, reduce the driving risk, and ensure the safe driving of the vehicle.

[0005] In a first aspect, this application provides a vehicle interaction method, including:

[0006] Obtain the eye movement information and / or semantic information of the user driving the vehicle;

[0007] Determine the corresponding triggered interaction operation according to the eye movement information and / or semantic information, and in response to the interaction operation, generate projection information incident on the human eye;

[0008] In the case where it is detected that the vehicle state is non-straight driving, hide or reduce the display of the projection information in a corner area.

[0009] Through the above technical means, throughout the process from inputting the interaction operation to viewing the response result of the interaction operation, the user's line of sight can not leave the front of the vehicle, reducing the driving risk and ensuring the safe driving of the vehicle. When the user needs to highly concentrate on driving the vehicle, the projection information is reduced or hidden to avoid the projection information interfering with the user's view of the road conditions ahead and distracting the user's concentration, further improving the safe driving of the vehicle.

[0010] Optionally, the obtaining the eye movement information and / or voice information of the user driving the vehicle includes:

[0011] Obtain the eye image of the user collected by a camera, perform eye movement recognition processing on the eye image to obtain eye movement information; and / or,

[0012] The voice information of the user collected by the microphone is subjected to speech recognition processing and semantic understanding processing to obtain semantic information.

[0013] Optionally, the generating projection information incident on the human eye in response to the interaction operation includes:

[0014] Generating a to-be-displayed screen in response to the interaction operation, and projecting the to-be-displayed screen on a first preset area of the front windshield of the vehicle through a head-up display; or,

[0015] Generating a to-be-displayed screen in response to the interaction operation, and projecting the to-be-displayed screen on a second preset area of the AR glasses.

[0016] Optionally, after the generating projection information incident on the human eye in response to the interaction operation, it further includes:

[0017] Determining a display viewing distance according to the moving speed of the vehicle, and adjusting the position of the projection information according to the display viewing distance.

[0018] Through the above technical means, the projection information can be adjusted to the staying point of the user's line of sight on the front windshield, so that the user can observe the road conditions ahead when viewing the projection information, which is beneficial to improving the safety of vehicle driving.

[0019] Optionally, after the generating projection information incident on the human eye in response to the interaction operation, it further includes:

[0020] Monitoring the distance between the vehicle and the vehicle in front, and comparing the distance with the display viewing distance;

[0021] In the case where the distance is less than the display viewing distance, hiding or reducing the display of the projection information in the corner area.

[0022] Through the above technical means, when the distance between the vehicle and the vehicle in front is less than the safe distance, the projection information can be hidden or reduced in the corner area, so that the user can highly concentrate on driving the vehicle to maintain a safe distance from the vehicle in front, avoid colliding with the vehicle in front, and improve the safety of driving.

[0023] Optionally, the method further includes:

[0024] Obtaining an image in front of the vehicle collected by the AR glasses, and performing obstacle detection on the image in front of the vehicle;

[0025] In the case of detecting an obstacle, marking the corresponding obstacle position on the front windshield of the vehicle based on the depth of the obstacle in the image in front of the vehicle.

[0026] Through the above technical means, the user can be prompted to avoid the obstacles ahead in time when driving a vehicle, reducing the risk of collision between the vehicle and the obstacles, which is beneficial to improving the safety of vehicle driving.

[0027] Optionally, before obtaining the eye movement information and / or semantic information of the user driving the vehicle, it further includes:

[0028] Obtain the biometric information of the user collected by the camera, and identify the identity information of the user based on the biometric information;

[0029] Configure the display mode of the projection information according to the display mode saved in association with the identity information.

[0030] Through the above technical means, the projection information can be configured to a display mode that conforms to the user's habits, bringing a better driving experience to the user.

[0031] In a second aspect, the present application provides a vehicle interaction device, including:

[0032] An interaction data acquisition module, configured to acquire the eye movement information and / or semantic information of the user driving the vehicle;

[0033] An interaction operation response module, configured to determine the corresponding triggered interaction operation according to the eye movement information and / or semantic information, and generate projection information incident on the human eye in response to the interaction operation;

[0034] A first display adjustment module, configured to hide or reduce the display of the projection information in a corner area when it is detected that the vehicle state is non-linear driving.

[0035] In a third aspect, the present application provides a vehicle interaction device, including:

[0036] One or more processors; a memory, storing one or more programs, when the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the vehicle interaction method as described in the first aspect.

[0037] In a fourth aspect, the present application provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute the vehicle interaction method as described in the first aspect when executed by a computer processor.

[0038] In this application, when the user is driving a vehicle, the eye movement information and / or semantic information of the user is obtained, the corresponding triggered interaction operation is determined according to the eye movement information and / or semantic information, the projection information incident on the human eye is generated based on the interaction operation, and when it is detected that the vehicle state is non-straight driving, the projection information is hidden or reduced and displayed in the corner area. Through the above technical means, the user can input interaction operations by turning the line of sight and making voice, and project the response results of the interaction operations within the line of sight range directly in front of the vehicle when the user is driving the vehicle, so that the user's line of sight does not need to leave the front of the vehicle during the whole process from inputting the interaction operation to viewing the response results of the interaction operation, solving the problem that the driver's eyes will leave the front of the vehicle when interacting with the vehicle in the prior art, reducing the driving risk and ensuring the safe driving of the vehicle. Moreover, when the vehicle is in a non-straight driving state, it indicates that the current road condition is complex and the user needs to highly concentrate on driving the vehicle. Therefore, the projection information is reduced or hidden to avoid the projection information interfering with the user's view of the road condition ahead and distracting the user's concentration, further improving the safe driving of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of a vehicle interaction method provided by an embodiment of the present application;

[0040] Figure 2 is a first schematic diagram of the driver's front field of view provided by an embodiment of the present application;

[0041] Figure 3 is a second schematic diagram of the driver's front field of view adopted by an embodiment of the present application;

[0042] Figure 4 is a third schematic diagram of the driver's front field of view provided by an embodiment of the present application;

[0043] Figure 5 is a schematic diagram of the user's line of sight provided by an embodiment of the present application;

[0044] Figure 6 is a fourth schematic diagram of the driver's front field of view provided by an embodiment of the present application;

[0045] Figure 7 is a schematic structural diagram of a vehicle interaction device provided by an embodiment of the present application;

[0046] Figure 8 is a schematic structural diagram of a vehicle interaction device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] To make the objectives, technical solutions and advantages of this application clearer, the following provides a more detailed description of specific embodiments of this application with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application, rather than limiting this application. Additionally, it should be noted that for ease of description, only parts related to this application rather than all content are shown in the drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operations are completed, but it can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0048] The terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same category, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / ", generally represents an "or" relationship between the associated objects before and after.

[0049] In relatively common existing implementation methods, the user driving the vehicle can input operation instructions to the vehicle through contact operations. The contact operations can be touching the in-vehicle screen or turning a knob, etc. Since devices or components such as the in-vehicle screen and the knob are located in the middle of the vehicle, that is, when the user interacts with the vehicle through contact operations, the user's line of sight will leave the front of the vehicle. In addition, the user can also input operation instructions to the vehicle through voice operations. The voice operations can be that the user controls the vehicle to play music or display a map, etc. Most current voice operations are triggered by keywords, and the preset keywords of the vehicle are limited, that is, the voice operations can only perform simple interaction operations. When it comes to complex interactions that require selection, the user's line of sight will leave the front of the vehicle. For example, when the vehicle displays a map and the user inputs a destination by voice, if the search engine searches for multiple destinations, the user needs to view and select which destination specifically. Therefore, whether it is a contact operation or a voice operation, there is a driving risk that the user's line of sight will leave the front of the vehicle during driving. When an unexpected situation occurs in front of the vehicle, the user may not have time to react, and the driving safety of the vehicle is relatively low.

[0050] To solve the above problems, the present embodiment provides a vehicle interaction method to trigger interactive operations based on the user's eye movement information and / or semantic information, and project the response results of the interactive operations into the user's line of sight in front of the vehicle when driving the vehicle, ensuring that the user's line of sight does not leave the front of the vehicle when driving the vehicle, thereby reducing driving risks and ensuring safe driving of the vehicle.

[0051] The vehicle interaction method provided in this embodiment may be executed by a vehicle interaction device, which may be implemented by software and / or hardware, and may be composed of two or more physical entities or one physical entity. For example, the vehicle interaction device may be a vehicle or a processor of a vehicle.

[0052] The vehicle interaction device is installed with at least one type of operating system, wherein the operating system includes but is not limited to Android system, Linux system and Windows system. The vehicle interaction device can install at least one application based on the operating system, and the application can be an application that comes with the operating system or an application downloaded from a third-party device or server. In this embodiment, the vehicle interaction device has at least an application that can execute the vehicle interaction method.

[0053] For ease of understanding, this embodiment is described by taking a vehicle as an example of a subject that executes the vehicle interaction method.

[0054] Figure 1 is a flow chart of a vehicle interaction method provided by an embodiment of the present application. Figure 1 As shown, the steps of the vehicle interaction method include:

[0055] S110: Acquire eye movement information and / or semantic information of a user driving a vehicle.

[0056] Among them, eye movement information refers to the information of the user's eye movement. The eye movement information can reflect the movement of the user's line of sight. Based on the eye movement information, the fixation point of the user's line of sight can be determined, and based on the fixation point of the line of sight, the interaction operation corresponding to the user's input can be determined. Exemplarily, an eye image of the user collected by a camera is obtained, and eye movement recognition processing is performed on the eye image to obtain eye movement information. In this embodiment, the camera can be a vehicle-mounted camera or a front camera of a smart phone. The vehicle-mounted camera or the smart phone is installed on one side of the steering wheel and the lens is directed towards the user's eyes to capture an eye image including the user's eyes. If the user wears AR glasses when driving a vehicle, the camera can also be a front camera of the AR glasses. The camera can continuously capture the user's eye images at a preset frame rate to generate an image sequence, perform frame-by-frame recognition on the image sequence to determine the eye positions in each eye image, determine the eye movement trajectory based on the eye positions in each eye image in the image sequence, and determine the fixation point of the user's line of sight based on the eye movement trajectory. When recognizing the eye position in the eye image, the eye in the eye image can be recognized based on a pre-trained neural network model to obtain the eye position in the eye image. In this embodiment, the eye movement information of the user is analyzed through the eye image collected by the camera to obtain the eye movement information without disturbing the user's driving of the vehicle. While determining the interaction operation input by the user through the eye movement information, it is ensured that the user can concentrate on driving the vehicle, improving driving safety.

[0057] In this embodiment, semantic information refers to the semantic information of the user's speech. The interaction operation input by the user can be analyzed based on the semantic information in the user's speech. Exemplarily, the speech information of the user collected by a microphone is obtained, and speech recognition processing and semantic understanding processing are performed on the speech information to obtain semantic information. Among them, the microphone can be a vehicle-mounted microphone or a microphone of a smart phone. The sound pickup directions of the vehicle-mounted microphone and the microphone of the smart phone are directed towards the user's mouth to clearly collect the speech information issued by the user. If the user wears AR glasses when driving a vehicle, the microphone can also be a microphone of the AR glasses. The microphone real-time collects the user's speech information, performs speech recognition processing on the speech information to obtain speech text information, and performs semantic understanding processing on the speech text information to obtain semantic information. Among them, the speech information can be subjected to speech recognition processing or semantic understanding processing through a pre-trained neural network model. In this embodiment, the interaction operation input by the user is accurately obtained through semantic analysis of the user's speech, which is beneficial to improving the user's driving experience of the vehicle.

[0058] S120. Determine the corresponding triggered interaction operation according to the eye movement information and / or semantic information, and generate projection information incident on the human eye in response to the interaction operation.

[0059] Among them, the interaction operation can be understood as an operation in which the user inputs to control the vehicle to execute corresponding instructions. For example, an operation to control the vehicle to play music or an operation to control the vehicle to display a map, etc. The projection information is the response result corresponding to the interaction operation, and this response result is incident on the user's eyes in a projection manner. In this embodiment, the response result can be projected onto the lens of the AR glasses to be incident on the user's eyes, or projected onto the front windshield of the vehicle through a head-up display to be incident on the user's eyes. However, no matter which projection method is used, in the user's field of view when driving the vehicle and looking straight ahead at the vehicle, the response result can be seen displayed on the front windshield of the vehicle, so that the user does not need to move the line of sight away from the due front of the vehicle during the driving process, ensuring the safe driving of the vehicle. For the convenience of description and understanding, in this embodiment, the projection information being displayed on the front windshield is used to generally describe the two projection methods of the response result being projected onto the front windshield and the lens of the AR glasses.

[0060] Exemplarily, if the eye movement information or semantic information is obtained alone currently, the interaction operation input by the user can be determined based on the obtained eye movement information or semantic information, and the vehicle executes corresponding instructions based on this interaction operation. For example, when the semantic information obtained alone is "locate point A", the vehicle determines that the interaction operation input by the user based on this semantic information is to display the navigation route from the vehicle to point A. In response to this interaction operation, the vehicle starts to search for the address of point A and determines the navigation route from the current position of the vehicle to point A, and projects the navigation route in the map as the projection information, so that the navigation route in the map is displayed on the front windshield of the vehicle. Figure 2 is the first schematic diagram of the driver's front field of view provided by the embodiment of the present application. As Figure 2 shown, the map 11 is semi-transparently displayed on the front windshield 10 and is located directly in front of the driver's seat of the vehicle, so that the user can see the navigation route 12 displayed in the map 11 when viewing the road conditions ahead, realizing that the user's eyes do not need to leave the front of the vehicle during the process from searching for the navigation route to viewing the navigation route, reducing the driving risk and ensuring the safe driving of the vehicle.

[0061] Furthermore, when the vehicle obtains the eye movement information alone, the stopping point of the user's line of sight is determined based on the eye movement information, and the corresponding triggered interaction operation is determined based on the position of the stopping point on the front windshield and the current picture displayed on the front windshield. For example, when the vehicle obtains the semantic information "locate point A", since point A is a large area, that is, the vehicle can search for relevant points A1, A2, and A3. Figure 3 is the second schematic diagram of the driver's front field of view passed by the embodiment of the present application. As Figure 3As shown in the figure, after the vehicle searches for points A1, A2, and A3 related to point A, a semi-transparent selection window 13 is displayed in the area on the front windshield 10 that is directly in front of the driver's seat of the vehicle. Options 13 corresponding to points A1, A2, and A3 are displayed in the selection window 13. At this time, eye movement information is obtained separately. If it is determined based on the eye movement information that the fixation point of the user's line of sight 15 is on the option 13 of point A1, it is determined that the current interaction operation input by the user is to display the navigation route from the vehicle to point A1. In response to this interaction operation, the vehicle starts to search for the address of point A1 and determines the navigation route from the current position to point A1, and takes the navigation route in the map as projection information and projects it so that the navigation route in the map is displayed on the front windshield. The display effect of the final navigation route can be referred to Figure 2 . Similarly, when the user inputs an interaction operation through eye movement information, the user's line of sight is still fixed on the road conditions ahead, enabling the user to keep their eyes on the front of the vehicle when inputting a relatively complex interaction operation, reducing the driving risk and ensuring the safe driving of the vehicle.

[0062] If the vehicle obtains both eye movement information and semantic information at the same time, it determines whether to trigger a corresponding interaction operation based on the eye movement information or based on the semantic information based on the projection information currently displayed on the front windshield. For example, when there is no projection information displayed on the front windshield, there is no option that matches the fixation point of the user's field of view corresponding to the eye movement information. Therefore, the interaction operation triggered can be determined based on the semantic information. After displaying information on the front windshield based on this interaction operation, the eye movement information is obtained again to determine the interaction operation input by the user for the information displayed on the front windshield. On the contrary, when various options are displayed on the front windshield, such as function options or location options, etc., the semantic information cannot determine the option selected by the user. In this case, the interaction operation triggered can be determined based on the eye movement information.

[0063] It should be noted that in the case of obtaining eye movement information separately, if there is no projection information displayed on the front windshield, it can also be determined that the eye movement information is invalid, that is, no response is made to this eye movement information. To save the energy consumption of the vehicle, the camera can be started to capture eye movement images only when there is projection information displayed on the front windshield. Otherwise, the camera is in a sleep state in other cases, avoiding the vehicle from capturing invalid eye movement images and recognizing invalid eye movement images, which is beneficial to optimizing the operating performance of the vehicle.

[0064] Furthermore, when generating projection information that is incident on the human eye in response to an interaction operation, a display screen to be displayed can be generated in response to the interaction operation, and the display screen to be displayed is projected onto a first preset area of the front windshield of the vehicle through a head-up display. Among them, the display screen to be displayed is the response result of the interaction operation, and the first preset area can be understood as the area directly in front of the driver's seat of the vehicle. For example, the first preset area can be Figure 2 or Figure 3The display area of the map 11 or the option window 13 in the middle. The map 11 and the option window are the to-be-displayed pictures generated after responding to the interaction operation. Refer to Figure 2 In response to the interaction operation of displaying the navigation route of the vehicle to point A and generating a map 11 including the navigation route 12, the vehicle projects the map 11 onto the first preset area of the front windshield through the head-up display, so that when the user sees the map 11 projected on the front windshield, the user can see the road conditions ahead, reducing the driving risk and ensuring the safe driving of the vehicle.

[0065] If the user wears AR glasses, a to-be-displayed picture is generated in response to the interaction operation, and the to-be-displayed picture is projected onto the second preset area of the AR glasses. It can be understood that when the user sits in the driver's seat and looks ahead at the vehicle, the second preset area of the lens of the AR glasses worn by the user substantially coincides with the first preset area of the front windshield. Refer to Figure 2 Since the lens of the AR glasses is transparent, the map 11 is displayed in a semi-transparent form in the second preset area of the lens, and the picture within the user's field of view is Figure 2 substantially the same, which is equivalent to the user seeing the map 11 displayed on the front windshield. When the user sees the map 11 through the AR glasses, the user can see the road conditions ahead, reducing the driving risk and ensuring the safe driving of the vehicle.

[0066] S130. When it is detected that the vehicle state is non-straight driving, the information incident on the human eye is hidden or reduced and displayed in the corner area.

[0067] Exemplarily, since the user is easily disturbed by the external driving environment during driving, when encountering non-straight driving such as sudden acceleration (such as during overtaking), braking or turning of the vehicle, the projected information can be hidden or reduced and displayed in the corner area, so that the user can highly concentrate on driving the vehicle. Among them, it can be determined whether the vehicle is in a state of sudden acceleration based on the front and rear speeds of the vehicle. For example, when the difference between the speed at the previous moment and the speed at the previous moment exceeds a certain threshold, it is determined that the vehicle is in a state of sudden acceleration; it can be determined whether the vehicle is in a braking state based on the braking device of the vehicle; it can be determined whether the vehicle is in a turning state based on the steering wheel of the vehicle. It should be noted that when the turning angle is small, it can be confirmed that the vehicle is in a straight driving state (for example, the user controls the vehicle to keep going straight by slightly turning the steering wheel). Further, when the vehicle returns to the straight driving state, the hidden projected information can be redisplayed or the projected information reduced in the corner area can be enlarged and displayed in the area in front of the driver's seat.

[0068] Figure 4 This is the third schematic diagram of the driver's front view provided by the embodiment of the present application. As Figure 4 shown, when the vehicle state is non-straight driving, the map 11 originally displayed in the area in front of the driver's seat ( Figure 2The shown map 11 is reduced and displayed at the lower left corner of the front windshield 10 to avoid distracting the user's driving concentration. When the vehicle state returns to straight driving, the map 11 displayed at the lower left corner is enlarged and displayed in the area in front of the driver's seat. At this time, the user's field of vision is as Figure 2 shown. Since the map 11 can be automatically displayed in the area in front of the driver's seat without additional input operations by the user, it is beneficial to improve the user experience.

[0069] In an embodiment, when there is projection information displayed on the front windshield, the display viewing distance can be determined according to the moving speed of the vehicle, and the position of the projection information can be adjusted according to the display viewing distance. Among them, the display viewing distance can be understood as the distance between the stop point of the user's line of sight on the lane and the vehicle. Figure 5 is a schematic diagram of the user's line of sight provided by an embodiment of the present application. As Figure 5 shown, the first line of sight 16 is the user's line of sight when the vehicle is moving at a high speed, point C is the stop point of the first line of sight on the lane, the second line of sight 17 is the user's line of sight when the vehicle is moving at a low speed, and point B is the stop point of the second line of sight on the lane. From Figure 5 it can be seen that when the vehicle is moving at a high speed, the user focuses on the distance, and when the vehicle is moving at a low speed, the user's line of sight focuses on the near. That is, the greater the moving speed of the vehicle, the greater the display viewing distance. The display viewing distance corresponding to the current user's line of sight can be determined based on the moving speed of the vehicle. The display viewing distances of the first line of sight 16 and the second line of sight 17 are different, so the heights of the stop points of the first line of sight 16 and the second line of sight 17 on the front windshield are different. The height of the stop point of the user's line of sight on the front windshield can be determined based on the display viewing distance of the current user's line of sight, and the projection height of the projection information can be adjusted to the height of the stop point of the user's line of sight on the front windshield. In this embodiment, the stop point of the user's line of sight on the front windshield is deduced through the moving speed of the vehicle, so as to adjust the projection information to the stop point of the user's line of sight on the front windshield, so that the user can observe the road conditions ahead when viewing the projection information, which is beneficial to improving the safety of vehicle driving.

[0070] In this embodiment, when the distance between the vehicle and the vehicle in front is relatively close, the user needs to focus highly on driving the vehicle to avoid colliding with the vehicle in front. However, the projection information will distract the user's attention while driving. To address this, this embodiment proposes that it is possible to determine whether the projection information will interfere with the user's driving based on the distance between the vehicle and the vehicle in front, and when it is determined that the projection information interferes with the user's driving, the projection information is hidden or reduced to a corner area, improving the user's focus on driving and the safety of vehicle driving. The specific implementation process is as follows: Monitor the distance between the vehicle and the vehicle in front, and compare the distance with the display viewing distance; in the case where the distance is less than the display viewing distance, the projection information is hidden or reduced and displayed in the corner area. It can be understood that when the vehicle is moving, a certain safety distance needs to be maintained from the vehicle in front. The safety distance is different depending on the moving speed of the vehicle. For example, when the moving speed of the vehicle is relatively fast, the safety distance is larger, and when the moving speed of the vehicle is relatively slow, the safety distance is smaller. In this embodiment, the display viewing distance is set as the safety distance, so that when the distance between the vehicle and the vehicle in front is less than the display viewing distance, the projection information is hidden or reduced to the corner area, enabling the user to focus highly on driving the vehicle to maintain a safe distance from the vehicle in front and avoid colliding with the vehicle in front, improving driving safety. Moreover, when the distance between the vehicle and the vehicle in front is less than the display viewing distance and the vehicle in front and the projection information overlap within the user's field of view, after hiding or reducing the projection information to the corner area, it is possible to prevent this information from interfering with the user's view of the driving condition of the vehicle in front, further improving driving safety. When the distance between the vehicle and the vehicle in front is greater than or equal to the display viewing distance, the distance between the vehicle and the vehicle in front is relatively far and the collision risk is relatively low, and the corresponding information can continue to be displayed on the front windshield.

[0071] In one embodiment, when the user wears AR glasses, the rear-view camera of the AR glasses can be used to capture an image in front of the vehicle, and based on the image in front of the vehicle, the user is prompted to avoid the obstacles in front in a timely manner when driving the vehicle, reducing the risk of the vehicle colliding with the obstacles, which is beneficial to improving the safety of vehicle driving. The specific implementation process is as follows: Obtain the image in front of the vehicle collected by the AR glasses, and perform obstacle detection on the image in front of the vehicle; in the case where an obstacle is detected, mark the corresponding obstacle position on the front windshield of the vehicle based on the depth of the obstacle in the image in front of the vehicle. Among them, the rear-view camera of the AR glasses can be a depth camera or a binocular camera. The image in front of the vehicle captured by the AR glasses is subjected to obstacle detection through an obstacle detection model, and the depth of the obstacle is determined in the case where an obstacle is detected. The depth of the obstacle can represent the distance between the AR glasses and the obstacle. Therefore, based on the depth of the obstacle and the pixel position of the obstacle in the image in front of the vehicle, the position information of the obstacle within the display area of the AR glasses is determined, and the obstacle is marked within the display area based on this position information, so that the user can see the marked obstacle when looking at the front windshield. Figure 6 It is the fourth schematic diagram of the driver's forward field of view provided by the embodiments of the present application. As Figure 6As shown in the figure, when an obstacle 18 is detected based on the image in front of the vehicle, the obstacle 18 is marked by a circle 19 based on the position information of the obstacle within the display area of the AR glasses, so as to prompt the user to avoid the obstacle in front when driving the vehicle, reduce the risk of collision between the vehicle and the obstacle, and contribute to improving the safety of vehicle driving. At the same time, the depth of the obstacle can be displayed beside the obstacle 18, so that the user can better control the driving situation of the vehicle.

[0072] In this embodiment, when the depth of the obstacle is less than the display viewing distance, the corresponding position of the obstacle is marked on the front windshield of the vehicle based on the depth of the obstacle in the image in front of the vehicle. Similarly, in this embodiment, the display viewing distance is set as the safe distance between the obstacle and the vehicle, so that when the distance between the obstacle and the vehicle is less than the safe distance, the corresponding position of the obstacle is marked on the front windshield to prompt the user to avoid the obstacle in time, and to prevent the mark of the obstacle from staying on the front windshield for a long time and affecting the display of other information.

[0073] In one embodiment, since different drivers have different heights and driving habits, their requirements for the display mode of projection information are different. For example, the display height, size, and style of the information on the front windshield can all be collectively referred to as the display mode of the information. In response to this, this embodiment proposes that after the user gets in the vehicle, the identity information of the user is recognized, and the projection information is configured into a display mode that conforms to the user's habits based on the identity information of the user, bringing a better driving experience to the user. The specific implementation process is as follows: obtain the biometric information of the user collected by the camera, and recognize the identity information of the user based on the biometric information; configure the display mode of the projection information according to the display mode associated with and saved with the identity information. Among them, the biometric information can be information such as face, fingerprint, voice, etc. that can represent the identity of the user. In this embodiment, the biometric information is taken as an example of the face for description. After the user gets in the vehicle, the face image of the user is captured by the camera, and face recognition is performed on the face image to determine the identity information of the user. The vehicle stores the identity information and the associated display mode, and the display mode can be manually input by the user when registering the identity information, or can be automatically set by the vehicle based on historical usage. The display mode associated with and saved can be obtained based on the current user's identity information, and the height, size, style, etc. of the projection information are set based on this display mode to be more in line with the user's driving habits.

[0074] In summary, the vehicle interaction method provided by the embodiments of the present application obtains the user's eye movement information and / or semantic information when the user is driving a vehicle, determines the corresponding triggered interaction operation according to the eye movement information and / or semantic information, generates projection information incident on the human eye based on the interaction operation, and when it is detected that the vehicle state is non-straight driving, hides or reduces the display of the projection information in the corner area. Through the above technical means, the user can input interaction operations by turning the line of sight and speaking, and project the response result of the interaction operation within the line of sight range directly in front of the vehicle when the user is driving the vehicle, so that the user's line of sight does not need to leave the front of the vehicle during the whole process from inputting the interaction operation to viewing the response result of the interaction operation, solving the problem that the driver's eyes will leave the front of the vehicle when interacting with the vehicle in the prior art, reducing the driving risk and ensuring the safe driving of the vehicle. Moreover, when the vehicle is in a non-straight driving state, it indicates that the current road condition is complex and the user needs to focus highly on driving the vehicle. Therefore, the projection information is reduced or hidden to avoid the projection information interfering with the user's viewing of the road condition ahead and distracting the user's attention, further improving the safe driving of the vehicle.

[0075] Based on the above embodiments, Figure 7 is a schematic structural diagram of a vehicle interaction device provided by an embodiment of the present application. Refer to Figure 7 , the vehicle interaction device provided in this embodiment specifically includes: an interaction data acquisition module 21, an interaction operation response module 22, and a first display adjustment module 23.

[0076] Among them, the interaction data acquisition module 21 is configured to acquire the eye movement information and / or semantic information of the user driving the vehicle;

[0077] The interaction operation response module 22 is configured to determine the corresponding triggered interaction operation according to the eye movement information and / or semantic information, and generate projection information incident on the human eye in response to the interaction operation;

[0078] The first display adjustment module 23 is configured to hide or reduce the display of the projection information in the corner area when it is detected that the vehicle state is non-straight driving.

[0079] Based on the above embodiments, the interaction data acquisition module 21 includes: an eye movement information acquisition unit configured to acquire the eye image of the user collected by a camera, perform eye movement recognition processing on the eye image, and obtain the eye movement information; and / or, a semantic information acquisition unit configured to collect the voice information of the user through a microphone, perform speech recognition processing and semantic understanding processing on the voice information, and obtain the semantic information.

[0080] Based on the above embodiments, the interaction operation response module 22 includes: a first display unit configured to generate a to-be-displayed screen in response to an interaction operation and project the to-be-displayed screen onto a first preset area of the front windshield of the vehicle through a head-up display; or, a second display unit configured to generate a to-be-displayed screen in response to an interaction operation and project the to-be-displayed screen onto a second preset area of the AR glasses.

[0081] Based on the above embodiments, the vehicle interaction device includes: a second display adjustment module configured to determine a display viewing distance according to the moving speed of the vehicle and adjust the position of the projection information according to the display viewing distance after generating projection information incident on the human eye in response to an interaction operation.

[0082] Based on the above embodiments, the vehicle interaction device includes a third display adjustment module, and the third display adjustment module includes: a distance monitoring unit configured to monitor the distance between the vehicle and the vehicle in front after generating projection information incident on the human eye in response to an interaction operation and compare the distance with the display viewing distance; a display adjustment unit configured to hide or reduce the display of the projection information in a corner area when the distance is less than the display viewing distance.

[0083] Based on the above embodiments, the vehicle interaction device includes an obstacle marking module, and the obstacle marking module includes: an obstacle detection unit configured to acquire an image in front of the vehicle collected by the AR glasses and perform obstacle detection on the image in front of the vehicle; an obstacle marking unit configured to mark the corresponding obstacle position on the front windshield of the vehicle based on the depth of the obstacle in the image in front of the vehicle when an obstacle is detected.

[0084] Based on the above embodiments, the vehicle interaction device includes a display mode configuration module, and the display mode configuration module includes: an identity information recognition unit configured to acquire biometric information of a user collected by a camera and recognize the identity information of the user based on the biometric information; a display mode configuration unit configured to configure the display mode of the projection information according to the display mode associated with and saved with the identity information.

[0085] As described above, the vehicle interaction device provided by the embodiment of the present application obtains the eye movement information and / or semantic information of the user when the user is driving the vehicle, determines the corresponding triggered interaction operation according to the eye movement information and / or semantic information, generates projection information incident on the human eye based on the interaction operation, and hides or reduces the display of the projection information in the corner area when it is detected that the vehicle state is non-straight driving. Through the above technical means, the user can input interaction operations by turning the line of sight and making a voice, and project the response result of the interaction operation within the line of sight range directly in front of the vehicle when the user is driving the vehicle, so that the user's line of sight does not need to leave the front of the vehicle during the whole process from inputting the interaction operation to viewing the response result of the interaction operation, solving the problem that the driver's eyes will leave the front of the vehicle when interacting with the vehicle in the prior art, reducing the driving risk, and ensuring the safe driving of the vehicle. Moreover, when the vehicle is in a non-straight driving state, it indicates that the current road condition is complex and the user needs to highly concentrate on driving the vehicle. Therefore, the projection information is reduced or hidden to avoid the projection information interfering with the user's view of the road condition ahead and distracting the user's concentration, further improving the safe driving of the vehicle.

[0086] The vehicle interaction device provided by the embodiment of the present application can be used to execute the vehicle interaction method provided by the above embodiment, and has corresponding functions and beneficial effects.

[0087] Figure 8 It is a schematic structural diagram of a vehicle interaction device provided by an embodiment of the present application. Refer to Figure 8 , the vehicle interaction device includes: a processor 31, a memory 32, a communication device 33, an input device 34, and an output device 35. The number of processors 31 in the vehicle interaction device can be one or more, and the number of memories 32 in the vehicle interaction device can be one or more. The processor 31, memory 32, communication device 33, input device 34, and output device 35 of the vehicle interaction device can be connected through a bus or other means.

[0088] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the vehicle interaction method of any embodiment of the present application (for example, the interaction data acquisition module 21, the interaction operation response module 22, and the first display adjustment module 23 in the vehicle interaction device). The memory 32 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the device, etc. In addition, the memory 32 can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some instances, the memory can further include a memory remotely set relative to the processor, and these remote memories can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0089] The communication device 33 is used for data transmission.

[0090] The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 32, that is, implements the above-mentioned vehicle interaction method.

[0091] The input device 34 can be used to receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the device. The output device 35 can include display devices such as a display screen.

[0092] The above-provided vehicle interaction device can be used to execute the vehicle interaction method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0093] An embodiment of the present application further provides a storage medium containing computer-executable instructions. The computer-executable instructions are used to execute a vehicle interaction method when executed by a computer processor. The vehicle interaction method includes: acquiring eye movement information and / or semantic information of a user driving a vehicle; determining a corresponding triggered interaction operation according to the eye movement information and / or semantic information, and generating projection information incident on the human eye in response to the interaction operation; when it is detected that the vehicle state is non-straight driving, hiding or reducing the display of the projection information in a corner area.

[0094] Storage medium - Any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. Additionally, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system that is connected to the first computer system via a network (such as the Internet). The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (such as in different computer systems connected via a network). The storage medium may store program instructions executable by one or more processors (e.g., embodied as a computer program).

[0095] Of course, for a storage medium containing computer-executable instructions provided by an embodiment of the present application, the computer-executable instructions are not limited to the vehicle interaction method as described above, and can also perform related operations in the vehicle interaction method provided by any embodiment of the present application.

[0096] The vehicle interaction device, storage medium, and vehicle interaction equipment provided in the above embodiments can execute the vehicle interaction method provided by any embodiment of the present application. For technical details not described in detail in the above embodiments, reference can be made to the vehicle interaction method provided by any embodiment of the present application.

[0097] The above is only the preferred embodiment of the present application and the technical principles applied. The present application is not limited to the specific embodiments here, and various obvious changes, re-adjustments, and substitutions that can be made by those skilled in the art will not depart from the protection scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments. Without departing from the concept of the present application, more other equivalent embodiments may be included, and the scope of the present application is determined by the scope of the claims.

Claims

1. A vehicle interaction method, characterized in that, Including: Obtaining the eye movement information and / or semantic information of the user driving the vehicle; Determining the corresponding triggered interaction operation according to the eye movement information and / or semantic information, and generating projection information incident on the human eye in response to the interaction operation; When it is detected that the vehicle state is non-straight driving, hiding or reducing the display of the projection information in the corner area.

2. The vehicle interaction method according to claim 1, wherein The obtaining of the eye movement information and / or voice information of the user driving the vehicle includes: Obtaining the eye image of the user collected by the camera, performing eye movement recognition processing on the eye image to obtain eye movement information; and / or, The voice information of the user collected by the microphone, performing speech recognition processing and semantic understanding processing on the voice information to obtain semantic information.

3. The vehicle interaction method according to claim 1, wherein The generating of the projection information incident on the human eye in response to the interaction operation includes: Generating a to-be-displayed screen in response to the interaction operation, and projecting the to-be-displayed screen on a first preset area of the front windshield of the vehicle through a head-up display; or, Generating a to-be-displayed screen in response to the interaction operation, and projecting the to-be-displayed screen on a second preset area of the AR glasses.

4. The vehicle interaction method according to claim 1, wherein After the generating of the projection information incident on the human eye in response to the interaction operation, it further includes: Determining a display viewing distance according to the moving speed of the vehicle, and adjusting the position of the projection information according to the display viewing distance.

5. The vehicle interaction method according to claim 4, wherein After the generating of the projection information incident on the human eye in response to the interaction operation, it further includes: Monitoring the distance between the vehicle and the vehicle in front, and comparing the distance with the display viewing distance; When the distance is less than the display viewing distance, hiding or reducing the display of the projection information in the corner area.

6. The vehicle interaction method according to claim 1, wherein The method further includes: Obtaining an image in front of the vehicle collected by the AR glasses, and performing obstacle detection on the image in front of the vehicle; When an obstacle is detected, marking the corresponding obstacle position on the front windshield of the vehicle based on the depth of the obstacle in the image in front of the vehicle.

7. The vehicle interaction method according to claim 1, wherein Before the obtaining of the eye movement information and / or semantic information of the user driving the vehicle, it further includes: Obtaining the biometric information of the user collected by the camera, and identifying the identity information of the user based on the biometric information; Configuring the display mode of the projection information according to the display mode associated with and saved for the identity information.

8. A vehicle interaction device, characterized in that, Including: An interaction data acquisition module configured to obtain the eye movement information and / or semantic information of the user driving the vehicle; An interaction operation response module configured to determine the corresponding triggered interaction operation according to the eye movement information and / or semantic information, and generate projection information incident on the human eye in response to the interaction operation; A first display adjustment module configured to hide or reduce the display of the projection information in the corner area when it is detected that the vehicle state is non-straight driving.

9. A vehicle interaction device, characterized in that, Including: One or more processors; A memory storing one or more programs, when the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the vehicle interaction method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the vehicle interaction method as described in any one of claims 1-7.

Images