Auxiliary method and system for augmented reality head-up display intelligent rearview system

Abstract

The invention belongs to the technical field of intelligent driving, and provides an auxiliary method and system for an augmented reality head-up display intelligent rearview system. The auxiliary method comprises the following steps: acquiring image data of a driver through an image acquisition module; acquiring position information and an attitude angle of the image acquisition module through the position acquisition module; determining coordinates of eyeballs in the image based on the image data of the driver; based on the position information, the attitude angle and the resolution of the image acquisition module, converting the coordinates of the eyeball in the image into a three-dimensional space to obtain the coordinates of the eyeball in the three-dimensional space; the coordinates of the eyeballs in the three-dimensional space are converted into a vehicle coordinate system, and the positions of the eyeballs in the vehicle coordinate system are obtained; the position of the eyeball in the vehicle coordinate system is sent to an AR-HUD intelligent rearview system; and head-up display is carried out through the AR-HUD intelligent rearview system. According to the invention, the computing resources of the AR-HUD intelligent rearview system can be saved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to an auxiliary method and system for an augmented reality head-up display intelligent rear-view system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the context of technological development, head-up display (HUD) was originally designed to help pilots better perceive their surroundings. With the popularization of technology, this head-up display technology has also been used in the automotive field. The development of HUD in the automotive industry has evolved from C-HUD to W-HUD, and is currently moving towards augmented reality head-up display (AR-HUD).

[0004] Augmented reality head-up display (AR-HUD) technology can combine virtual information with actual road conditions, allowing drivers to obtain information more intuitively.

[0005] Rearview mirrors are common accessories in cars, mainly used to observe the situation behind the vehicle. However, traditional rearview mirrors are gradually replaced by smart rearview mirrors. Smart rearview mirrors have multiple functions and are gradually becoming popular in the market.

[0006] When AR-HUD is applied to the field of smart rearview mirrors, it is necessary to track the driver's eye position to adjust the displayed image. However, the calculation process of eye tracking consumes a lot of computing resources. Summary of the invention

[0007] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides an auxiliary method and system for an augmented reality head-up display intelligent rearview system. The present invention places the implementation and calculation of eye tracking on the intelligent rearview mirror, and calculates the angle and position of the eye tracking camera through the built-in gyroscope, thereby determining the driver's eye position, which can save computing resources of the AR-HUD intelligent rearview system.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A first aspect of the present invention provides an assisting method for an augmented reality head-up display intelligent rearview system.

[0010] An auxiliary method for an augmented reality head-up display intelligent rear-view system, comprising:

[0011] Acquire the image data of the driver through the image acquisition module;

[0012] Acquire the position information and attitude angle of the image acquisition module through the position acquisition module;

[0013] Based on the image data of the driver, the coordinates of the eyeball in the image are determined; based on the position information, the attitude angle and the resolution of the image acquisition module, the coordinates of the eyeball in the image are converted into three-dimensional space to obtain the coordinates of the eyeball in the three-dimensional space; the coordinates of the eyeball in the three-dimensional space are converted into a vehicle coordinate system to obtain the position of the eyeball in the vehicle coordinate system; the position of the eyeball in the vehicle coordinate system is sent to the AR-HUD intelligent rearview system;

[0014] Head-up display is provided through AR-HUD intelligent rear-view system.

[0015] Further, the coordinates of the eyeball in the image are determined based on the image data of the driver; based on the position information, the attitude angle and the resolution of the image acquisition module, the coordinates of the eyeball in the image are converted into three-dimensional space to obtain the coordinates of the eyeball in the three-dimensional space; the following formula is used to express it:

[0016]

[0017] Among them, (X, Y, Z) represents the coordinates of the eyeball in three-dimensional space, (u, v) represents the coordinates of the eyeball in the image, (M, N) represents the resolution of the image acquisition module, (x0, y0, z0) represents the position information of the image acquisition module, γ represents the attitude angle of the image acquisition module, f represents the focal length of the image acquisition module, and Z c Indicates the distance from the eyeball to the image acquisition module.

[0018] Furthermore, the image acquisition module and the position acquisition module are both arranged on the central rearview mirror.

[0019] Furthermore, the method of acquiring the image data of the driver through the image acquisition module includes: acquiring the image information of the driver through the image acquisition module, and converting the image information into image data.

[0020] Furthermore, after the image data is obtained, it is necessary to perform denoising and contrast enhancement processing on the image data.

[0021] Furthermore, after preprocessing the image data, a deep learning model is used to perform face detection and locate the face area; based on the face area, eye features are extracted; based on the eye features, a machine learning model is used to locate the eye area; based on the eye area, an iris recognition algorithm is used to locate the eyeball.

[0022] A second aspect of the present invention provides an auxiliary system for an augmented reality head-up display intelligent rearview system.

[0023] An auxiliary system for an augmented reality head-up display intelligent rear-view system, comprising:

[0024] An image acquisition module, used for acquiring image data of the driver;

[0025] A position acquisition module is used to obtain the position information and attitude angle of the image acquisition module;

[0026] The control module is connected to the image acquisition module and the position acquisition module, and is used to determine the coordinates of the eyeball in the image based on the image data of the driver; based on the position information, the attitude angle and the resolution of the image acquisition module, the coordinates of the eyeball in the image are converted into three-dimensional space to obtain the coordinates of the eyeball in the three-dimensional space; the coordinates of the eyeball in the three-dimensional space are converted into the vehicle coordinate system to obtain the position of the eyeball in the vehicle coordinate system; the position of the eyeball in the vehicle coordinate system is sent to the AR-HUD intelligent rearview system;

[0027] AR-HUD intelligent rear-view system, used for head-up display.

[0028] A third aspect of the present invention provides a computer-readable storage medium.

[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the auxiliary method of the augmented reality head-up display intelligent rearview system as described in the first aspect above.

[0030] A fourth aspect of the present invention provides a computer device.

[0031] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the auxiliary method of the augmented reality head-up display intelligent rearview system as described in the first aspect above are implemented.

[0032] A fifth aspect of the present invention provides a computer program product or a computer program.

[0033] The present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the auxiliary method of the augmented reality head-up display intelligent rearview system as described in the first aspect above.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention places the implementation and calculation of eye tracking on the smart rearview mirror, calculates the angle and position of the eye tracking camera through the built-in gyroscope, and thus determines the driver's eye position, which can save the computing resources of the AR-HUD smart rearview system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0037] Figure 1 is a flow chart of an auxiliary method of an augmented reality head-up display intelligent rear-view system shown in the present invention;

[0038] Figure 2 is a structural diagram of an auxiliary system of an augmented reality head-up display intelligent rear-view system shown in the present invention;

[0039] Among them, 1. Image acquisition module, 2. Position acquisition module, 3. Control module, 4. AR-HUD intelligent rear-view system. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0041] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0042] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0043] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0044] Embodiment 1

[0045] like Figure 1 As shown, this embodiment provides an auxiliary method for an augmented reality head-up display intelligent rearview system. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0046] Acquire the image data of the driver through the image acquisition module;

[0047] Acquire the position information and attitude angle of the image acquisition module through the position acquisition module;

[0048] Based on the image data of the driver, the coordinates of the eyeball in the image are determined; based on the position information, the attitude angle and the resolution of the image acquisition module, the coordinates of the eyeball in the image are converted into three-dimensional space to obtain the coordinates of the eyeball in the three-dimensional space; the coordinates of the eyeball in the three-dimensional space are converted into a vehicle coordinate system to obtain the position of the eyeball in the vehicle coordinate system; the position of the eyeball in the vehicle coordinate system is sent to the AR-HUD intelligent rearview system;

[0049] Head-up display is provided through AR-HUD intelligent rear-view system.

[0050] The present embodiment is described in detail below:

[0051] Step 1: Acquire the driver's image data through the image acquisition module;

[0052] Specifically, the image information of the driver is obtained through the image acquisition module, and the image information is converted into image data;

[0053] After obtaining the image data, it is necessary to perform denoising and contrast enhancement on the image data;

[0054] After image data preprocessing, a deep learning model (such as a convolutional neural network) is used to detect faces and locate the face area;

[0055] Based on the face area, the features of the eye area (such as color, shape, texture, etc.) are used for feature extraction. According to the shape of the pupil, a circular template can be used to dynamically search the position of the pupil from left to right and from top to bottom in the image window. The template matching algorithm determines the position of the target by calculating the similarity between the template and the matching area, and uses the most similar position as the matching point. Machine learning models (such as support vector machines, random forests, neural networks, etc.) are used to classify and regress the extracted eye features to locate the eye area; based on the eye area, an iris recognition algorithm is used to locate the eyeball.

[0056] Step 2: Obtain the position information and attitude angle of the image acquisition module through the position acquisition module;

[0057] The location information includes the longitude, latitude, altitude and attitude angle of the rearview mirror where the image acquisition module is located. The attitude angle includes the pitch angle, yaw angle and roll angle.

[0058] Step 3: Based on the image data of the driver, determine the coordinates of the eyeball in the image; based on the position information, the attitude angle and the resolution of the image acquisition module, convert the coordinates of the eyeball in the image into three-dimensional space to obtain the coordinates of the eyeball in the three-dimensional space; convert the coordinates of the eyeball in the three-dimensional space into the vehicle coordinate system to obtain the position of the eyeball in the vehicle coordinate system; send the position of the eyeball in the vehicle coordinate system to the AR-HUD intelligent rearview system;

[0059] In some embodiments, the eye position calculation is achieved by the following process:

[0060] Assume that the coordinates of the eyeball position captured by the image acquisition module 1 in the image are (u, v), and the resolution of the camera is (M, N).

[0061] Assume that the position information of the image acquisition module 2 is (x0, y0, z0) and the attitude angle is γ.

[0062] According to the pinhole camera model, the position of the eyeball in three-dimensional space (X, Y, Z) can be calculated by the following formula:

[0063]

[0064] Where f is the focal length of the camera, Z c Indicates the distance from the eyeball to the image acquisition module (which can be obtained through methods such as depth sensor or binocular vision).

[0065] Due to the influence of the attitude angle, the calculated (X, Y, Z) needs to be transformed to obtain the position of the eyeball in the vehicle coordinate system.

[0066] Step 4: Use AR-HUD intelligent rear-view system to display the head-up image.

[0067] At present, the position of the eyeball is usually fixed, while the position of the smart rearview mirror may change as the driver adjusts the angle of the rearview mirror. The present invention places the implementation and calculation of eye tracking on the smart rearview mirror, calculates the position of the eye tracking camera through the built-in gyroscope, and thus determines the driver's eyeball position, which can save the computing resources of the AR-HUD smart rearview system.

[0068] Embodiment 2

[0069] This embodiment provides an auxiliary system for an augmented reality head-up display intelligent rearview system.

[0070] like Figure 2 As shown, an auxiliary method and system for an augmented reality head-up display intelligent rear-view system, comprising:

[0071] Image acquisition module 1: It is installed on the central rearview mirror and uses a high-resolution camera, an infrared camera, etc. to capture the driver's image information.

[0072] Position acquisition module 2: It is also installed on the central rearview mirror and uses positioning technologies such as GPS, gyroscope, and acceleration sensor to obtain the precise position information of the image acquisition module. This information includes the longitude, latitude, altitude, and attitude angle (such as pitch angle, yaw angle, and roll angle) of the rearview mirror.

[0073] Control module 3: It is electrically connected to image acquisition module 1 and position acquisition module 2, and is used to process and analyze image data, identify the position of the driver's eyeballs, and calculate the specific position of the eyeballs in the vehicle based on the position information. Then, this information is sent to AR-HUD intelligent rearview system 4 via CAN bus.

[0074] In some embodiments, the image acquisition module 1 captures the driver's image information and converts it into digital image data. The control module 3 pre-processes the digital image data, including denoising, contrast enhancement, etc., to improve image quality.

[0075] Based on digital image data, image processing algorithms (such as edge detection, contour extraction, etc.) are used to identify the driver's facial features and then locate the eye position.

[0076] In some embodiments, the position acquisition module 2 is used to obtain the precise position information of the image acquisition module 1, including longitude, latitude, altitude and attitude angle. The control module 3 converts the position information into coordinates in the vehicle coordinate system for subsequent calculations.

[0077] In some embodiments, the eye position calculation is achieved by the following process:

[0078] Assume that the coordinates of the eyeball position captured by the image acquisition module 1 in the image are (u, v), and the resolution of the camera is (M, N).

[0079] Assume that the position information of the image acquisition module 2 is (x0, y0, z0) and the attitude angle is γ.

[0080] According to the pinhole camera model, the position of the eyeball in three-dimensional space (X, Y, Z) can be calculated by the following formula:

[0081]

[0082] Where f is the focal length of the camera, Z c Indicates the distance from the eyeball to the image acquisition module (which can be obtained through methods such as depth sensor or binocular vision).

[0083] Due to the influence of the attitude angle, the calculated (X, Y, Z) needs to be transformed to obtain the position of the eyeball in the vehicle coordinate system.

[0084] The control module sends the position of the eyeball in the vehicle coordinate system to the AR-HUD intelligent rearview system 4 through the CAN bus. The AR-HUD intelligent rearview system 4 dynamically adjusts the content and position of the projection based on the received eyeball position information to form a virtual-real fusion effect and enhance the driver's perception and interaction capabilities.

[0085] In practical applications, it is necessary to calibrate the image acquisition module 1 and the position acquisition module 2 to improve the measurement accuracy.

[0086] In the process of eye position calculation, many factors need to be considered, such as image distortion, sensor error, lighting conditions, etc., to ensure the accuracy of the calculation results.

[0087] Relevant protocols and standards must be followed during CAN bus communication to ensure data reliability and real-time performance.

[0088] Through the above process, the image acquisition module, the position acquisition module and the control module of the present invention work together to achieve accurate acquisition and transmission of the driver's eye position information, provide strong support for the application of the AR-HUD intelligent rearview system, improve the utilization rate of the rearview mirror, and save the computing resources of the AR-HUD intelligent rearview system.

[0089] Embodiment 3

[0090] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the auxiliary method of the augmented reality head-up display intelligent rearview system as described in the above-mentioned embodiment 1 are implemented.

[0091] Embodiment 4

[0092] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the auxiliary method of the augmented reality head-up display intelligent rearview system as described in the first embodiment above are implemented.

[0093] Embodiment 5

[0094] This embodiment provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the auxiliary method of the augmented reality head-up display intelligent rearview system described in the first embodiment.

[0095] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0096] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0099] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An auxiliary method for an augmented reality head-up display intelligent rear-view system, characterized in that: include: Acquire the image data of the driver through the image acquisition module; Acquire the position information and attitude angle of the image acquisition module through the position acquisition module; Based on the image data of the driver, the coordinates of the eyeball in the image are determined; based on the position information, the attitude angle and the resolution of the image acquisition module, the coordinates of the eyeball in the image are converted into three-dimensional space to obtain the coordinates of the eyeball in the three-dimensional space; the coordinates of the eyeball in the three-dimensional space are converted into a vehicle coordinate system to obtain the position of the eyeball in the vehicle coordinate system; the position of the eyeball in the vehicle coordinate system is sent to the AR-HUD intelligent rearview system; Head-up display is provided through AR-HUD intelligent rear-view system.

2. The auxiliary method of the augmented reality head-up display intelligent rear-view system according to claim 1, characterized in that: The method comprises: determining the coordinates of the eyeball in the image based on the image data of the driver; converting the coordinates of the eyeball in the image into three-dimensional space based on the position information, the attitude angle and the resolution of the image acquisition module to obtain the coordinates of the eyeball in the three-dimensional space; It is expressed by the following formula: Among them, (X, Y, Z) represents the coordinates of the eyeball in three-dimensional space, (u, v) represents the coordinates of the eyeball in the image, (M, N) represents the resolution of the image acquisition module, (x0, y0, z0) represents the position information of the image acquisition module, γ represents the attitude angle of the image acquisition module, f represents the focal length of the image acquisition module, and Z c Indicates the distance from the eyeball to the image acquisition module.

3. The auxiliary method of the augmented reality head-up display intelligent rear-view system according to claim 1, characterized in that: The image acquisition module and the position acquisition module are both arranged on the central rearview mirror.

4. The auxiliary method of the augmented reality head-up display intelligent rear-view system according to claim 1, characterized in that: The method for obtaining the image data of the driver through the image acquisition module includes: obtaining the image information of the driver through the image acquisition module, and converting the image information into image data.

5. The auxiliary method of the augmented reality head-up display intelligent rear-view system according to claim 4, characterized in that: After obtaining the image data, it is necessary to perform denoising and contrast enhancement processing on the image data.

6. The auxiliary method of the augmented reality head-up display intelligent rear-view system according to claim 5, characterized in that: After image data preprocessing, a deep learning model is used to detect faces and locate the face area; based on the face area, eye features are extracted; Based on the eye features, a machine learning model is used to locate the eye area; based on the eye area, an iris recognition algorithm is used to locate the eyeball.

7. An auxiliary system for an augmented reality head-up display intelligent rear-view system, characterized in that: include: An image acquisition module, used for acquiring image data of the driver; A position acquisition module is used to obtain the position information and attitude angle of the image acquisition module; The control module is connected to the image acquisition module and the position acquisition module, and is used to determine the coordinates of the eyeball in the image based on the image data of the driver; based on the position information, the attitude angle and the resolution of the image acquisition module, the coordinates of the eyeball in the image are converted into three-dimensional space to obtain the coordinates of the eyeball in the three-dimensional space; the coordinates of the eyeball in the three-dimensional space are converted into the vehicle coordinate system to obtain the position of the eyeball in the vehicle coordinate system; the position of the eyeball in the vehicle coordinate system is sent to the AR-HUD intelligent rearview system; AR-HUD intelligent rear-view system, used for head-up display.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the auxiliary method of the augmented reality head-up display intelligent rearview system as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the auxiliary method of the augmented reality head-up display intelligent rearview system as described in any one of claims 1-6 are implemented.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the steps in the auxiliary method of the augmented reality head-up display intelligent rearview system according to any one of claims 1 to 6 are implemented.

Images