Method and system for assisted driving

Abstract

A method and system for assisting driving, which are run in a car machine system and smart glasses. First, the car machine system acquires the cabin model of the vehicle and the field of view image of the smart glasses, then through feature point matching, the field of view image is compared with the cabin model to determine the spatial position information of the driving assistance image. Subsequently, the car machine system generates the driving assistance image according to the field of view image, the cabin model and the spatial position information, and transmits the image to the smart glasses, so that the driver can obtain more intuitive driving assistance information through the smart glasses, and improve the driving safety and driving experience.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and more particularly to a method and system for assisting driving. Background Technology

[0002] With the continuous advancement of vehicle technology, not only has power efficiency been improved, but safety protection has also been strengthened. In addition to advanced active safety technologies, the application of panoramic imaging systems is moving further towards the vision of zero accidents.

[0003] However, even with increasingly sophisticated vehicle features, the design of the vehicle structure can still affect the driver's visibility. For example, the A-pillar may obstruct the view while driving or turning, or blind spots may be created when reversing; these factors can still lead to accidents. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method and system for assisting driving, which can display information about the surrounding environment outside the vehicle that needs to be paid attention to during the driving process via smart glasses, thereby assisting in safe driving.

[0005] An embodiment of the present invention provides a method for assisting driving, executed in a vehicle infotainment system and smart glasses. The method includes: the vehicle infotainment system acquiring a cabin model corresponding to the vehicle and a field-of-view image from the smart glasses; the vehicle infotainment system performing feature point matching between the field-of-view image and the cabin model to determine spatial location information for generating an assisted driving image; the vehicle infotainment system generating the assisted driving image based on the field-of-view image, the cabin model, and the spatial location information; and the vehicle infotainment system transmitting the assisted driving image to the smart glasses, so that the smart glasses display the assisted driving image.

[0006] An embodiment of the present invention also provides an assisted driving system, including a vehicle infotainment system and smart glasses, wherein the vehicle infotainment system includes a controller and a memory for storing a computer program, and when the computer program is executed by the controller, the controller implements the assisted driving method.

[0007] Compared to existing technologies, the assisted driving method and system provided by this invention can be combined with smart glasses to display assisted driving images on the smart glasses worn by the driver. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of the architecture of an assisted driving system according to an embodiment of the present invention.

[0009] Figure 2 This is a flowchart of an assisted driving method according to an embodiment of the present invention.

[0010] Figure 3 This is a flowchart of an assisted driving method according to another embodiment of the present invention.

[0011] Figure 4 This is a flowchart of an assisted driving method according to another embodiment of the present invention.

[0012] Explanation of main component symbols Smart Glasses 110 Car infotainment system 120 Processor 111 Controller 121 Memory 112, 122 Input devices 113, 123 Output devices 114, 124 Sensor units 115, 125 Communication interfaces 116, 126 Steps S201~S205, S301~S305, S401~S403 The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0013] To facilitate understanding and implementation of this invention by those skilled in the art, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that this invention provides many applicable inventive concepts, which can be implemented in various specific forms. Those skilled in the art can utilize the details described in these or other embodiments, as well as any structural, logical, and electrical variations thereof, to implement the invention without departing from its spirit and scope.

[0014] This specification provides different embodiments to illustrate the technical features of different implementations of the invention. The configuration of elements in the embodiments is for illustrative purposes only and is not intended to limit the invention. Furthermore, the repetition of some reference numerals in the embodiments is for simplification and does not imply any correlation between different embodiments. The same element numbers used in the illustrations and specification represent the same or similar components. The illustrations in this specification are simplified and not drawn to scale.

[0015] Furthermore, in describing some embodiments of the present invention, the specification describes the method and / or procedure of the present invention in a specific order of steps. However, since the method and procedure are not necessarily performed according to the specific order of steps described, they are not limited to the specific order of steps. Those skilled in the art will understand that other orders are also possible implementations. Therefore, the specific order of steps described in the specification is not intended to limit the scope of the patent application. Moreover, the scope of the present invention for the method and / or procedure is not limited to the order of execution steps written therein, and those skilled in the art will understand that adjusting the order of execution steps does not depart from the spirit and scope of the present invention.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Some embodiments of the invention are described in detail below with reference to the accompanying drawings.

[0017] Please see Figure 1 The diagram shows a schematic of the architecture of an assisted driving system according to an embodiment of the present invention. The system 100 includes smart glasses 110 and a vehicle infotainment system 120. The smart glasses 110 are worn by the driver, and the vehicle infotainment system 120 is installed in the vehicle. The smart glasses 110 and the vehicle infotainment system 120 can communicate via any wired network, wireless network, or a combination thereof. The vehicle infotainment system 120 can be used in, but is not limited to, autonomous vehicles, semi-autonomous vehicles, and non-autonomous vehicles. Those skilled in the art will understand that… Figure 1 The smart glasses 110 and vehicle infotainment system 120 shown may also include more or fewer components than shown, or combine some of their components, or have different component arrangements.

[0018] Smart glasses 110 include, but are not limited to, mixed reality glasses, augmented reality glasses, and virtual reality glasses.

[0019] The smart glasses 110 include a processor 111, a memory 112, an input device 113, an output device 114, a sensor unit 115, and a communication interface 116.

[0020] The processor 111 may include one or more processing cores. The processor 111 connects to various components within the smart glasses 110 using various interfaces and lines, and performs various functions and processes data of the smart glasses 110 by running or executing instruction sets, computer programs or code stored in memory 112, and by calling data stored in memory 112.

[0021] Input device 113 is used to provide input to the user and generate signal input related to function control, including but not limited to a camera device, microphone, etc. Output device 114 may include devices such as a display, speaker, vibrator, etc. The display may include an imaging screen to reflect an image projection source to the user's eyes, or the display may use light waves to guide the image projection source to the user's eyes.

[0022] Sensor unit 115 is used to acquire motion parameters of smart glasses 110 to determine the pose of smart glasses 110. For example, sensor unit 115 may include inertial measurement unit (IMU) sensors (including accelerometers and gyroscopes) to acquire three-degree-of-freedom attitude data of smart glasses 110 as motion parameters of smart glasses 110. Based on the three-degree-of-freedom attitude angle related information, the rotation direction and rotation angle of smart glasses 110 can be determined, while based on the three-degree-of-freedom axial acceleration information, the translational distance of smart glasses 110 in each rotation direction can be determined.

[0023] Communication interface 116 is used to communicate with external devices or systems, such as Figure 1 The vehicle infotainment system 120 is included. The communication interface 116 may optionally include standard wired or wireless interfaces, such as Bluetooth or Wi-Fi.

[0024] The vehicle infotainment system 120 includes a controller 121, a memory 122, an input device 123, an output device 124, a sensor unit 125, and a communication interface 126.

[0025] The controller 121 is the control center of the vehicle infotainment system 120. It connects to various parts of the system via various interfaces and lines. By running and storing instruction sets, computer programs, or code in the memory 122, and by calling data stored in the memory 122, it executes various functions of the vehicle infotainment system 120 and processes data, thereby achieving overall vehicle control and information processing. In one embodiment, the controller 121 may include one or more processing units, such as a microcontroller or an electronic control unit. For example, the controller 121 may include a first microcontroller (…). Figure 1 (Not shown in the image) is used to process and control data and functions of the advanced driver assistance system; second microcontroller ( Figure 1 (Not shown in the image) is used to process and control data and functions for autonomous driving; the third microcontroller ( Figure 1 (Not shown) A braking mechanism used to control the vehicle body, which may include, for example, a chassis motor controller, seat controller, air conditioning controller, battery controller, door and window controller, and lighting controller.

[0026] The memory 122 is used to store computer programs and various data. The memory 122 includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc.; the data storage area can store data or information created based on the use of the vehicle system 120, etc.

[0027] Input device 123 is used to receive input signals and user input information. For example, a microphone for receiving voice input, a touch screen for displaying information to the driver or passenger and / or receiving input.

[0028] Output device 124 is used to display data and information. Examples include monitors, touch screens, television screens, head-up displays, other display types or combinations thereof, speakers, and dashboards.

[0029] Sensor unit 125 may include one or more sensors for acquiring sensing data from the one or more sensors. The one or more sensors are disposed in the interior and / or exterior areas of the vehicle. Examples of the one or more sensors include position sensors, radar sensors, ultrasonic sensors, light sensors, IMU sensors, surround-view cameras, infrared cameras, steering sensors, vibration sensors, speed sensors, etc.

[0030] The communication interface 126 is used to communicate with external devices via a network. The communication interface 126 can receive data and instructions from external devices, and can also send data, instructions, and information to the external devices. In one example, the external device can be an electronic device such as a mobile phone, tablet computer, or smart glasses 110.

[0031] Please see Figure 2 The diagram shows a flowchart of an assisted driving method according to an embodiment of the present invention. This method can be executed by smart glasses in the assisted driving system and the vehicle's infotainment system. Before executing the assisted driving method, the smart glasses and the vehicle's infotainment system can first establish a connection and pair via Bluetooth.

[0032] Step S201: The vehicle system obtains the cabin model corresponding to the vehicle.

[0033] In one embodiment, the cabin model can be a 3D cabin model obtained by three-dimensional modeling of the cabin, or it can be a cabin model obtained by capturing images of the cabin and stitching the captured images together. For example, multiple cabin images can be obtained by taking panoramic photos of the vehicle's interior space from multiple shooting angles, and then the multiple cabin images can be combined into an animation to obtain an animated cabin model. This cabin model can be pre-built and stored in the vehicle's infotainment system's memory.

[0034] In step S202, the vehicle system acquires the field-of-view image from the smart glasses.

[0035] In one embodiment, the driver's field of vision is captured by the camera device of the smart glasses and transmitted to the vehicle system via the communication interface of the smart glasses, such as Wi-Fi.

[0036] In step S203, the vehicle system performs feature point matching between the field of view image and the vehicle cabin model to determine the spatial location information for generating the assisted driving image.

[0037] In one embodiment, image feature extraction algorithms such as SIFT, ORB, and SURF can be used to match key points of the field of view image with the vehicle cabin model, and determine the spatial position information of the generated assisted driving image mapped to the vehicle cabin model.

[0038] In step S204, the vehicle system generates assisted driving images for display on the smart glasses based on the field of view image, the vehicle cabin model, and spatial location information.

[0039] In one embodiment, the vehicle infotainment system can perform neural network-like operations. After the neural network is trained, it can identify at least one region of the vehicle cabin model. For example, the neural network can be used to identify the vehicle body structure region and the glass material region of the vehicle cabin model. The vehicle body structure region, such as the A-pillar, B-pillar, or C-pillar, usually obstructs the actual view outside the vehicle; the glass material region, such as the window, windshield, or rearview mirror, usually displays the actual view outside the vehicle.

[0040] The vehicle infotainment system utilizes a neural network-like approach to identify at least one region of the vehicle cabin model, essentially locating that region. Therefore, the system can determine whether the driver's field of vision includes any vehicle structure areas that might obstruct their view, based on the field-of-view image and the vehicle cabin model. When it determines that the driver's field of vision includes such areas, such as the A-pillar, B-pillar, or C-pillar, it acquires the corresponding external image based on spatial location information to generate an assisted driving image that includes a perspective image of that vehicle structure area.

[0041] Specifically, the vehicle system can perform image processing on the field of view image and the external image to overlay the external image corresponding to the vehicle structure area onto the vehicle structure area in the field of view image to achieve a perspective effect.

[0042] In step S205, the vehicle system transmits the assisted driving image to the smart glasses for display.

[0043] Please see Figure 3 The diagram shows a flowchart of an assisted driving method according to another embodiment of the present invention. This method can be executed by smart glasses in the assisted driving system and the vehicle's infotainment system. Before executing the assisted driving method, the smart glasses and the vehicle's infotainment system can first establish a connection and pair via Bluetooth.

[0044] Step S301: The vehicle system determines the direction the driver intends to travel.

[0045] In one embodiment, the vehicle infotainment system can determine the driver's intended driving direction based on position information, gear shift information, and steering wheel rotation angle. Gear shift information includes shifting from park to drive, from drive to reverse, and from park to reverse. Specifically, shifting from park to drive indicates the vehicle is moving forward from a stationary state; shifting from drive to reverse indicates the vehicle is moving backward from a forward position; and shifting from park to reverse indicates the vehicle is moving backward from a stationary state. The vehicle infotainment system can calculate the vehicle's azimuth angle based on the position information, gear shift information, and steering wheel rotation angle, thereby determining the driver's intended driving direction.

[0046] In another embodiment, the vehicle system can determine the driver's intended driving direction based on the status of the turn signals and the direction of wheel rotation.

[0047] In another embodiment, the vehicle system can also determine the driver's intended direction of travel based on images of the surrounding environment captured by the surround-view camera. For example, it can analyze lane markings based on the images to determine whether to change lanes left or right, or compare the relative positions of the vehicle with intersection signs or other vehicles to determine the driver's intended direction of travel.

[0048] Step S302: The vehicle system acquires the blind spot image corresponding to the desired driving direction.

[0049] In step S303, the vehicle system determines whether the driver's line of sight is consistent with the intended driving direction.

[0050] In one embodiment, the vehicle system can determine the driver's line of sight by comparing the field of vision image from the smart glasses, the center point of the field of vision image, and the vehicle cabin model.

[0051] Step S304: When the driver's line of sight is consistent with the desired driving direction, the vehicle system generates an assisted driving image based on the field of vision image of the smart glasses, the vehicle cabin model and spatial position and blind spot image. The assisted driving image includes a perspective image of the vehicle body structure area corresponding to the blind spot image.

[0052] In one embodiment, the vehicle system can simultaneously compare the field-of-view image with feature points of the vehicle cabin model, as well as the field-of-view image with feature points of the blind spot image to generate the perspective image.

[0053] For example, when the vehicle's infotainment system determines that the vehicle is switching from a parked state to moving forward, it can acquire images of the blind spot ahead and generate a driver assistance image that includes a perspective image of the lower edge of the windshield and the dashboard.

[0054] Step S305: When the driver's line of sight is not consistent with the desired driving direction, the vehicle system determines the vehicle structure area where the blind spot image fits based on the field of vision image and the cabin model to generate an assisted driving image.

[0055] For example, when the vehicle's infotainment system determines that the vehicle is switching from forward to backward movement, the driver typically looks at the rearview mirror rather than behind when reversing. In this situation, the smart glasses' field of vision will include the rearview mirror. The infotainment system can then capture the blind spot image and match it to the rearview mirror's position within the field of vision to generate an assisted driving image.

[0056] In one embodiment, the vehicle system can also be combined with ADAS to add a prompt box to remind the driver to pay attention to objects approaching the vehicle in the assisted driving image.

[0057] Please see Figure 4 The diagram shows a flowchart of an assisted driving method according to another embodiment of the present invention. This method can be executed by smart glasses in the assisted driving system and the vehicle's infotainment system. Before executing the assisted driving method, the smart glasses and the vehicle's infotainment system can first establish a connection and pair via Bluetooth.

[0058] In step S401, the vehicle system determines whether there is a second vehicle ahead based on the field of view image of the smart glasses.

[0059] In step S402, the vehicle system determines whether the driver intends to change lanes or overtake.

[0060] In one embodiment, the vehicle system can comprehensively determine whether the driver intends to change lanes or overtake based on the lane line analysis results of the external image, the status of the turn signals, and whether the vehicle speed has changed.

[0061] Step S403: When the vehicle system determines that there is an intention to change lanes or overtake, it acquires the forward-looking image of the second vehicle and overlays the forward-looking image onto the second vehicle in the field of view image to generate an assisted driving image.

[0062] In one embodiment, the vehicle system can acquire forward-looking images of a second vehicle via vehicle-to-vehicle communication or vehicle networking.

[0063] In summary, the assisted driving method and system of the present invention can assist the driver in safe driving by displaying assisted driving images with a see-through effect through smart glasses worn by the driver.

[0064] It is worth noting that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A driving assistance method, implemented in an in-vehicle infotainment system and smart glasses, characterized in that, The method includes: The vehicle system acquires the cabin model corresponding to the vehicle and the field of view image of the smart glasses; The vehicle system performs feature point matching between the field of view image and the vehicle cabin model to determine the spatial location information for generating the assisted driving image. The vehicle system generates the assisted driving image based on the field-of-view image, the vehicle cabin model, and the spatial location information; and The vehicle system transmits the assisted driving images to the smart glasses so that the smart glasses can display the assisted driving images.

2. The assisted driving method as described in claim 1, characterized in that, The vehicle cabin model includes a vehicle body structure area and a glass material area.

3. The assisted driving method as described in claim 2, characterized in that, The step of generating the assisted driving image based on the field of view image, the vehicle cabin model, and the spatial position information further includes: The vehicle infotainment system determines, based on the field-of-view image and the vehicle cabin model, whether the driver's field of vision includes the vehicle body structure area; and When it is determined that the driver's field of vision includes the vehicle structure area, the vehicle system obtains the external image corresponding to the vehicle structure area based on the spatial location information to generate the assisted driving image.

4. The assisted driving method as described in claim 3, characterized in that, The step of obtaining the external image corresponding to the vehicle body structure region based on the spatial location information to generate the assisted driving image further includes: The exterior image corresponding to the vehicle body structure area is overlaid onto the vehicle body structure area in the field of view image to achieve a perspective effect.

5. The assisted driving method as described in claim 1, characterized in that, The method further includes: The vehicle system determines the driver's desired driving direction; The vehicle system acquires blind spot images corresponding to the desired driving direction; The vehicle system determines whether the driver's line of sight is consistent with the intended driving direction; When the driver's line of sight is determined to be consistent with the intended driving direction, the vehicle system generates the assisted driving image based on the field of vision image of the smart glasses, the vehicle cabin model, the spatial position information, and the blind spot image. The assisted driving image includes a perspective image of the vehicle structure area corresponding to the blind spot image; and When it is determined that the driver's line of sight is inconsistent with the intended driving direction, the vehicle system determines the vehicle structure area where the blind spot image fits based on the field of vision image and the vehicle cabin model to generate the auxiliary image.

6. The assisted driving method as described in claim 1, characterized in that, The method further includes: The vehicle infotainment system detects objects approaching the vehicle via the ADAS system; The vehicle infotainment system adds a notification box to the object in the assisted driving image to alert the driver of the vehicle.

7. The assisted driving method as described in claim 1, characterized in that, The method further includes: The vehicle system determines whether there is a second vehicle ahead based on the field of view image of the smart glasses; The vehicle infotainment system determines whether the driver intends to change lanes or overtake. When it is determined that the driver intends to change lanes or overtake, the forward-view image of the second vehicle is acquired, and the forward-view image is superimposed on the location of the second vehicle in the field of view image to generate the assisted driving image.

8. The assisted driving method as described in claim 2, characterized in that, The glass material areas include the windows, windshield, and rearview mirrors.

9. The assisted driving method as described in claim 2, characterized in that, The vehicle body structure area includes the A-pillar, B-pillar, and C-pillar.

10. A driver assistance system, characterized in that, include In-vehicle infotainment system; and Smart glasses, wherein the vehicle system includes a controller and a memory, the memory being used to store a computer program, which, when executed by the controller, causes the controller to implement the assisted driving method as described in any one of claims 1 to 9.

Images