Alignment Method and Alignment Device for a Display Device, and In-Vehicle Display System

Abstract

The present invention discloses an alignment method and alignment device for a display device, and a vehicle-mounted display system. The alignment method includes: determining a line-of-sight focusing area of a user in a target vehicle; acquiring a captured image of an original scene by a first camera, and extracting entity identification points in the captured image; determining a two-dimensional image mapping relationship between the captured image and a virtual image in the current field of view of the user through pre-obtained calibration parameters, where the virtual image is an image to be projected by the display device; projecting the entity identification points into the virtual image according to the two-dimensional image mapping relationship, where virtual identification objects in the virtual image are aligned with entity reference objects in the line-of-sight focusing area. The present invention solves the technical problem in the related art that it is impossible to enable the driver and passengers to obtain a good "immersive" augmented reality experience with good virtual-real alignment while the line of sight moves, and reduces the usage interest of the driver and passengers.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to an alignment method and an alignment device for a display device, and a vehicle-mounted display system. Background Art

[0002] With the rise of the "intelligent" wave, technologies such as computer vision, artificial intelligence, virtual / augmented reality have been widely applied in the fields of vehicle driving and safety. Through a head-up display (HUD), users do not need to lower their heads to view instrument information, thus always maintaining a head-up state, reducing potential safety hazards caused by ignoring the rapid changes in the external environment during the process of lowering and raising the head, and greatly improving the driving experience of the vehicle. In 2012, researchers added navigation information to the HUD, enabling the vehicle to obtain prompts for the driving direction at the next moment based on its own position, and introducing the concept of augmented reality (AR) into the HUD in a milestone manner, enabling the HUD to prompt information about the real scene in the virtual image and enriching the application scenarios of the HUD.

[0003] In recent years, the HUD has gradually evolved towards an immersive augmented reality head-up display (AR-HUD). The immersive AR-HUD requires that the virtual image in the user's eyes be related to the real scene covered by the virtual image in the user's eyes, such as highlighting the lane line boundary, prompting the planned path, and prompting the safe driving distance, etc., greatly improving the user experience. However, similarly, the requirements for virtual-real alignment and calibration of the immersive AR-HUD have also increased accordingly. Currently, the main defects of the virtual-real alignment and calibration technology of the immersive AR-HUD include: (1) In vehicle-mounted applications, the movement of the driver's head cannot be restricted. Therefore, the geometric relationship between the virtual image projected by the AR-HUD and the user's viewing angle cannot be fixed like other AR applications such as AR helmets and AR glasses; (2) In vehicle-mounted applications, the vehicle is moving at a high speed. Even a very small time delay will be amplified due to the vehicle speed, resulting in a decrease in alignment accuracy. Therefore, developing the virtual-real alignment technology and its calibration algorithm for the AR-HUD that follows the line of sight movement and improving the algorithm performance are the keys to enabling the large-scale commercialization of the AR-HUD.

[0004] In related technologies, common virtual-real alignment technologies include two types: Prior Art One. Through the three-dimensional calibration reprojection technology, the three-dimensional external scenery of the vehicle can be reprojected to the human eye and truncated on the virtual image plane of the AR-HUD. However, since this technology has very high requirements for sensors and computing power, it cannot be commercially used on a large scale. Moreover, in order to reduce costs, most of the external scenery of the vehicle is agreed upon at a certain depth. Therefore, when the scenery outside this depth is reprojected to the virtual image of the AR-HUD, the size of the virtual image is inconsistent with the size of the real scenery it covers, resulting in visual differences for the driver; Prior Art Two. Through the two-dimensional alignment technology, the corresponding virtual image content is directly "pasted" according to the fixed scene seen from the driver's perspective during calibration. Since the calibration of this technology lacks quantitative indicators and requires that the position and attitude of the human eye and the virtual image plane of the AR-HUD should be the same as those during calibration, it cannot be compatible with the requirement of "following the driver's line of sight moving". The applicable scenario of this technology is too single, only applicable to the environment of marking the ground plane, unable to adapt to uphill and downhill, and also unable to mark scenery such as pedestrians, vehicles, and signal systems that are higher than the ground plane.

[0005] In response to the above problems, no effective solution has been proposed yet. Summary of the Invention

[0006] An embodiment of the present invention provides an alignment method, an alignment device, and a vehicle-mounted display system for a display device, so as to at least solve the technical problem in related technologies that it is impossible to enable the driver and passengers to obtain a good virtual-real alignment "immersive" augmented reality experience while the line of sight moves, reducing the user interest of the driver and passengers.

[0007] According to one aspect of the embodiment of the present invention, an alignment method for a display device is provided, including: determining a line-of-sight focusing area of a user in a target vehicle, where at least one physical reference object is included in the line-of-sight focusing area; acquiring a captured image of an original scene by a first camera, and extracting physical identification points in the captured image; determining a two-dimensional image mapping relationship between the captured image and a virtual image in the current field of view of the user through pre-obtained calibration parameters, where the virtual image is an image to be projected by the display device; and projecting the physical identification points into the virtual image according to the two-dimensional image mapping relationship, where a virtual identification object in the virtual image is aligned with the physical reference object in the line-of-sight focusing area.

[0008] Optionally, the step of determining the line-of-sight focusing area of the user in the target vehicle includes: determining a focused line of sight of the user in front of the target vehicle based on the coordinates of the line-of-sight projection object of the user; performing an intersection process on the focused line of sight and the plane where the ground in front of the target vehicle is located to obtain a gaze position point of the user; and determining the line-of-sight focusing area of the user in the target vehicle based on the gaze position point.

[0009] Optionally, according to the position states of the display device and the user's eyes, the calibration parameters are determined by a virtual-real alignment calibration method in combination with a human eye simulation auxiliary camera.

[0010] Optionally, the alignment method further includes: when the positions of the display device and the user's eyes are maintained at fixed positions, the calibration parameters are determined by a virtual-real opening and closing calibration method in combination with the human eye simulation auxiliary camera at the fixed positions; or when the positions of the display device and / or the user's eyes change, the virtual-real opening and closing calibration method is repeated by adjusting the device with a changed position to multiple different calibration positions to determine the normalization parameter calibration correspondence, and the calibration parameters are determined according to the actual position of the device and the normalization parameter calibration correspondence.

[0011] Optionally, the actual positions of the display device and the user's eyes are obtained through a driver monitoring system.

[0012] Optionally, the calibration parameters include: a first two-dimensional image mapping relationship between a first image collected by a first camera and a second image collected by the human eye simulation auxiliary camera, and a second two-dimensional image mapping relationship between a third image collected by the human eye simulation auxiliary camera and a virtual calibration pattern.

[0013] Optionally, determining the calibration parameters by a virtual-real alignment calibration method in combination with a human eye simulation auxiliary camera includes: by controlling the turning on and off of the display device, in combination with a physical calibration pattern and a virtual calibration pattern, determining the calibration parameters according to the images captured by the human eye simulation auxiliary camera and the first camera.

[0014] Optionally, by controlling the turning on and off of the display device, in combination with a physical calibration pattern and a virtual calibration pattern, determining the calibration parameters according to the images captured by the human eye simulation auxiliary camera and the first camera includes: after turning off the display device, in combination with the physical calibration pattern, determining a third two-dimensional image mapping relationship between a first image collected by the first camera and a second image collected by the human eye simulation auxiliary camera; after turning on the display device, in combination with the virtual calibration pattern, determining a fourth two-dimensional image mapping relationship between a third image collected by the human eye simulation auxiliary camera and the virtual calibration pattern; and characterizing the third two-dimensional image mapping relationship and the fourth two-dimensional image mapping relationship as the calibration parameters.

[0015] Optionally, the physical calibration pattern is placed in front of the target vehicle, and the virtual calibration pattern is displayed on the virtual image plane projected by the display device, where the physical calibration pattern includes at least one physical feature point, and the virtual calibration pattern includes at least one virtual feature point.

[0016] Optionally, the alignment method further includes: the physical calibration pattern is replaced by an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.

[0017] Optionally, by controlling the turning on and off of the display device, combining the equivalent physical calibration pattern and the virtual calibration pattern, and determining the calibration parameters according to the images captured by the human eye simulation auxiliary camera and the first camera, including: after turning off the display device, restoring the virtual calibration pattern equivalent to the ground through the equivalent physical calibration pattern and projecting it onto the fourth image generated by the first camera, and the human eye simulation auxiliary camera collecting a fifth image of the equivalent physical calibration pattern to determine the fifth two-dimensional image mapping relationship between the fourth image and the fifth image; after turning on the display device, combining the virtual calibration pattern to determine the sixth two-dimensional image mapping relationship between the sixth image collected by the human eye simulation auxiliary camera and the virtual calibration pattern; characterizing the fifth two-dimensional image mapping relationship and the sixth two-dimensional image mapping relationship as the calibration parameters.

[0018] Optionally, restoring the virtual calibration pattern equivalent to the ground through the equivalent physical calibration pattern and projecting it onto the fourth image generated by the first camera includes: determining the ground clearance value and the preset scaling ratio of the equivalent physical calibration pattern according to the first spatial pose of the first camera, the human eye simulation auxiliary camera, and the equivalent physical calibration pattern obtained by calibration; according to the ground clearance value and the preset scaling ratio, combining the principle of light propagation to restore the virtual calibration pattern equivalent to the ground of the equivalent physical calibration pattern and calculating the second spatial pose of the virtual calibration pattern; projecting the virtual calibration pattern onto the imaging plane of the first camera in combination with the second spatial pose to generate the fourth image.

[0019] Optionally, the normalization parameter calibration correspondence is a mapping relationship between any normalization parameter and the corresponding calibration parameter, where the normalization parameter is the proportion of the position movement amount of the display device and / or the human eye simulation auxiliary camera in the variable stroke.

[0020] Optionally, the devices whose positions change include: the display device and / or the human eye simulation auxiliary camera.

[0021] Optionally, when the position of the display device and / or the user's eyes changes, the virtual-real opening / closing calibration method is repeated by adjusting the device with the changed position to multiple different calibration positions, and the normalization parameter calibration correspondence is determined. The calibration parameter is determined according to the actual position of the device and the normalization parameter calibration correspondence, including: adjusting the device with the changed position to at least two different calibration positions in the variable space, repeating the virtual-real opening / closing calibration method, and determining the normalization parameter calibration correspondence; determining the normalized position parameter on the variable space based on the actual position of the device, and determining the calibration parameter according to the normalized position parameter and the normalization parameter calibration correspondence.

[0022] Optionally, when the variable space is a variable linear travel, the alignment method includes: adjusting the device to at least two different calibration positions within the variable linear travel, repeating the virtual-real opening / closing calibration method, and determining the normalization parameter calibration correspondence, where the at least two different calibration positions include: the highest display reference position and the lowest display reference position of the device in the variable linear travel.

[0023] Optionally, when the variable space is a variable planar travel, the alignment method includes: adjusting the device to at least three different calibration positions within the variable planar travel, repeating the virtual-real opening / closing calibration method, and determining the normalization parameter calibration correspondence, where the at least three different calibration positions include: three limit reference positions of the device that are not collinear at the same time.

[0024] Optionally, when the variable space is a variable three-dimensional space travel, the alignment method includes: adjusting the device to at least four different calibration positions within the variable three-dimensional space travel, repeating the virtual-real opening / closing calibration method, and determining the normalization parameter calibration correspondence, where the at least four different calibration positions include: four limit reference positions of the device that are not coplanar at the same time.

[0025] Optionally, adjusting the device with the changed position to at least two different calibration positions in the variable space, repeating the virtual-real opening / closing calibration method, and determining the normalization parameter calibration correspondence includes: adjusting the device with the changed position to at least two different calibration positions in the variable space, and repeating the virtual-real opening / closing calibration method at the different calibration positions to obtain a set of calibration parameters, where each calibration parameter in the set of calibration parameters corresponds one-to-one to each calibration position; determining a set of normalization parameters according to all the calibration positions and the variable space; and fitting and determining the normalization parameter calibration correspondence by combining the set of calibration parameters and the set of normalization parameters.

[0026] Optionally, the alignment method further includes: mounting the human eye simulation auxiliary camera by a robotic arm to adjust the position of the human eye simulation auxiliary camera, and providing corresponding three-dimensional normalized coordinates.

[0027] Optionally, the alignment method further includes: projecting the physical identification points into the virtual image by combining the two-dimensional image mapping relationship and the attitude of the target vehicle.

[0028] According to another aspect of the embodiments of the present invention, there is also provided an alignment device for a display device, including: a first determination unit configured to determine a line-of-sight focusing area of a user in a target vehicle, where at least one physical reference object is included in the line-of-sight focusing area; a first acquisition unit configured to acquire a captured image of an original scene by a first camera, and extract physical identification points in the captured image; a first determination unit configured to determine a two-dimensional image mapping relationship between the captured image and a virtual image in the current field of view of the user through pre-obtained calibration parameters, where the virtual image is an image to be projected by the display device; an alignment unit configured to project the physical identification points into the virtual image according to the two-dimensional image mapping relationship, where virtual identification objects in the virtual image are aligned with physical reference objects in the line-of-sight focusing area.

[0029] Optionally, the first determination unit includes: a first determination module configured to determine a focused line of sight of the user in front of the target vehicle based on the coordinates of the line-of-sight projection subject of the user; a fixation point determination module configured to perform an intersection process on the focused line of sight and a plane where the ground in front of the target vehicle is located to obtain a fixation position point of the user; a second determination module configured to determine the line-of-sight focusing area of the user in the target vehicle based on the fixation position point.

[0030] Optionally, according to the position states of the display device and the user's eyes, the calibration parameters are determined by combining a human eye simulation auxiliary camera through a virtual-real alignment calibration method.

[0031] Optionally, the alignment device further includes: a third determination unit configured to, when the positions of the display device and the user's eyes are kept at a fixed position, determine the calibration parameters by combining the human eye simulation auxiliary camera through a virtual-real opening / closing calibration method at this fixed position; or a fourth determination unit configured to, when the positions of the display device and / or the user's eyes change, repeat the virtual-real opening / closing calibration method by adjusting the device with a changed position to multiple different calibration positions and determine a normalized parameter calibration correspondence relationship, and determine the calibration parameters according to the actual position of the device and the normalized parameter calibration correspondence relationship.

[0032] Optionally, the actual positions of the display device and the user's eyes are obtained through a driver monitoring system.

[0033] Optionally, the calibration parameters include: a first two-dimensional image mapping relationship between a first image collected by a first camera and a second image collected by the human eye simulation auxiliary camera, and a second two-dimensional image mapping relationship between a third image collected by the human eye simulation auxiliary camera and a virtual calibration pattern.

[0034] Optionally, the third determination unit includes: a third determination module, configured to determine the calibration parameters according to images captured by the human eye simulation auxiliary camera and the first camera by controlling the turning on and off of the display device and combining a physical calibration pattern and a virtual calibration pattern.

[0035] Optionally, the third determination module includes: a first determination sub-module, configured to, after turning off the display device, determine a third two-dimensional image mapping relationship between a first image collected by the first camera and a second image collected by the human eye simulation auxiliary camera by combining the physical calibration pattern; a second determination sub-module, configured to, after turning on the display device, determine a fourth two-dimensional image mapping relationship between a third image collected by the human eye simulation auxiliary camera and the virtual calibration pattern by combining the virtual calibration pattern; and a third determination sub-module, configured to characterize the third two-dimensional image mapping relationship and the fourth two-dimensional image mapping relationship as the calibration parameters.

[0036] Optionally, the alignment device further includes: a replacement unit, configured to replace the physical calibration pattern with an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.

[0037] Optionally, the third determination module further includes: a fourth determination sub-module, configured to, after turning off the display device, restore a virtual calibration pattern equivalent to the ground and project it onto a fourth image generated by the first camera, and determine a fifth two-dimensional image mapping relationship between the fourth image and a fifth image of the equivalent physical calibration pattern collected by the human eye simulation auxiliary camera; a fifth determination sub-module, configured to, after turning on the display device, determine a sixth two-dimensional image mapping relationship between a sixth image collected by the human eye simulation auxiliary camera and the virtual calibration pattern by combining the virtual calibration pattern; and a sixth determination sub-module, configured to characterize the fifth two-dimensional image mapping relationship and the sixth two-dimensional image mapping relationship as the calibration parameters.

[0038] According to another aspect of the embodiments of the present invention, a vehicle-mounted display system is further provided, including: a driver monitoring system for tracking the gaze focus area of a user in a target vehicle, where at least one physical reference object is included in the gaze focus area; a driving recorder for photographing the gaze focus area to obtain a photographed image; a vehicle-mounted controller connected to the driver monitoring system and the driving recorder respectively, and executing the alignment method for a display device described in any one of the above; a head-up display for projecting a virtual image to a preset position directly in front of the user, where a virtual identifier in the virtual image is aligned with the physical reference object in the gaze focus area.

[0039] According to another aspect of the embodiments of the present invention, a vehicle-mounted control device is further provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the alignment method for a display device described in any one of the above by executing the executable instructions.

[0040] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the alignment method for a display device described in any one of the above.

[0041] In the embodiments of the present invention, first, the gaze focus area of a user in a target vehicle is determined, where at least one physical reference object is included in the gaze focus area. A photographed image of the original scene collected by a first camera is obtained, and physical identification points in the photographed image are extracted. Through pre-obtained calibration parameters, a two-dimensional image mapping relationship between the photographed image and a virtual image in the current field of view of the user is determined, where the virtual image is an image to be projected by a display device. According to the two-dimensional image mapping relationship, the physical identification points are projected into the virtual image, where the virtual identifier in the virtual image is aligned with the physical reference object in the gaze focus area. In this embodiment, after pre-calibrating to obtain calibration parameters, the actual scene of the gaze focus area outside the vehicle is photographed by a photographing device, and the physical identification points of the photographed picture content are projected onto the virtual image, so that the virtual identifier in the virtual image in the user's field of view at the current moment is aligned with the physical reference object in the gaze focus area, and the virtual image in the user's eyes is associated with the real scene covered by the virtual image in the user's eyes, achieving an "immersive" augmented reality experience effect of virtual-real alignment, improving the user's experience and usage interest, and thus solving the technical problem in the related art that the driver and passengers cannot obtain a good "immersive" augmented reality experience of virtual-real alignment while their gazes are moving, reducing the usage interest of the driver and passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings described herein are used to provide a further understanding of the present invention, and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation of the present invention. In the drawings:

[0043] Figure 1 is a hardware structural block diagram of a computer terminal (or mobile device) for implementing an alignment method for a display device according to an embodiment of the present invention;

[0044] Figure 2 is a flowchart of an alternative alignment method for a display device according to an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of a scenario of an alternative calibration method for a display device according to an embodiment of the present invention Figure 1 ;

[0046] Figure 4 is a schematic diagram of the basic principle of pinhole imaging in three-dimensional vision according to an embodiment of the present invention;

[0047] Figure 5 is a schematic diagram of a scenario of an alternative calibration method for a display device according to an embodiment of the present invention Figure 2 ;

[0048] Figure 6 is a schematic diagram of a scenario of an alternative calibration method for a display device according to an embodiment of the present invention Figure 3 ;

[0049] Figure 7 is an alternative alignment effect diagram according to an embodiment of the present invention;

[0050] Figure 8 is a schematic diagram of an alignment device for a display device according to an embodiment of the present invention. Detailed implementation manners

[0051] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0052] It should be noted that the terms "first", "second", etc. in the description, claims and the above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0053] First, some nouns or terms that appear in the process of describing the embodiments of the present application are applicable to the following explanations:

[0054] Augmented reality head-up display (AR-HUD): The display is located in front of the instrument panel in the driver's line of sight, or the display covers the windshield. The projection device displays the virtual image content on the virtual image surface of the augmented reality head-up display, and the virtual image content is related to the real scene covered by the virtual image, such as prompts, markings, transparency, etc.

[0055] Driver monitor system (DMS): Detects driver behavior through visual methods, such as closing eyes, blinking, gaze direction, head movement, etc.

[0056] Augmented reality (AR): Generates virtual objects that do not exist in the real environment through computer graphics technology and visualization technology, and "embeds" them into the real environment through display technology to merge with it.

[0057] Virtual-real registration: Aligns the virtual image generated in augmented reality with the real scene.

[0058] Following the line of sight movement: It means that when the driver's line of sight moves, the present invention can calculate in real time the position where the virtual image of the AR-HUD should be located, so that in the driver's line of sight, the virtual image can always be aligned with the real scene.

[0059] The following embodiments of the present invention can be applied to various vehicles equipped with augmented reality head-up displays. The types of vehicles include but are not limited to: bicycles, cars, driving simulators, AR windows, etc. It can also be applied to various AR projection devices whose geometric poses with the user are not fixed, such as, naked-eye AR, etc.

[0060] In the present invention, a virtual-real alignment technology for following the driver's line of sight in an AR-HUD and its calibration algorithm are proposed. After calibration according to the calibration steps in the present invention, the driver monitoring system can calculate the fixation point of the driver at the current moment. The in-vehicle computer calculates the image of the real scene outside the vehicle seen by the driver at the current moment in the driving recorder, and further projects the image in the driving recorder onto the virtual image of the AR-HUD according to the alignment method proposed in the embodiments of the present invention, so that the virtual image of the AR-HUD in the driver's field of view at the current moment can completely cover the real scene outside the vehicle, achieving an "immersive" augmented reality experience of virtual-real alignment. The present invention proposes a virtual-real alignment model that follows the driver's line of sight, which can balance the requirements of computing power, cost, and user experience, and does not require a three-dimensional reconstruction process nor complex camera internal and external parameter calibration, and the real-time performance on mainstream in-vehicle computers can reach the frame rate of the camera.

[0061] Embodiment 1

[0062] According to an embodiment of the present invention, an embodiment of an alignment method for a display device is further provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0063] The method embodiment provided by the first embodiment of the present application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing an alignment method for a display device is shown. As Figure 1 shown, the computer terminal 10 (or mobile device 10) may include one or more (shown as 102a, 102b,..., 102n in the figure) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may further include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0064] It should be noted that one or more of the above-mentioned processors 102 and / or other data processing circuits can generally be referred to as "data processing circuits" herein. The data processing circuit can be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit can be a single independent processing module, or be incorporated in whole or in part into any one of other elements in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit is a kind of processor control (such as the selection of a variable resistance terminal path connected to an interface).

[0065] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage devices corresponding to the alignment method for a display device in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the vulnerability detection method of the above-mentioned application program. The memory 104 can include high-speed random access memory, and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 can further include a memory remotely disposed relative to the processor 102, and these remote memories can be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, enterprise intranets, local area networks, mobile communication networks, and combinations thereof.

[0066] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network can include the wireless network provided by the communication provider of the computer terminal 10. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0067] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables a user to interact with the user interface of the computer terminal 10 (or mobile device).

[0068] According to the embodiments of the present invention, an embodiment of an alignment method for a display device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0069] Figure 2 is a flowchart of an optional alignment method for a display device according to an embodiment of the present invention. As Figure 2 shown, the method includes the following steps:

[0070] Step S202, determining a line-of-sight focus area of a user in a target vehicle, where at least one physical reference object is included in the line-of-sight focus area;

[0071] Step S204, acquiring a captured image of the first camera for collecting the original scene, and extracting physical identification points in the captured image;

[0072] Step S206, determining a two-dimensional image mapping relationship between the captured image and a virtual image in the current field of view of the user through pre-obtained calibration parameters, where the virtual image is an image to be projected by the display device;

[0073] Step S208, projecting the physical identification points onto the virtual image according to the two-dimensional image mapping relationship, where virtual identification objects in the virtual image are aligned with physical reference objects in the line-of-sight focus area.

[0074] Through the above steps, it is possible to first determine the line-of-sight focus area of the user in the target vehicle, where at least one physical reference object is included in the line-of-sight focus area, acquire a captured image of the first camera for collecting the original scene, and extract physical identification points in the captured image. Through pre-obtained calibration parameters, determine the two-dimensional image mapping relationship between the captured image and a virtual image in the current field of view of the user, where the virtual image is an image to be projected by the display device. According to the two-dimensional image mapping relationship, project the physical identification points onto the virtual image, where virtual identification objects in the virtual image are aligned with physical reference objects in the line-of-sight focus area. In this embodiment, after pre-calibrating to obtain calibration parameters, the real scene of the line-of-sight focus area outside the vehicle is photographed by the photographing device, and the physical identification points of the photographed picture content are projected onto the virtual image, so that the virtual identification objects in the virtual image in the user's field of view at the current moment are aligned with the physical reference objects in the line-of-sight focus area, and the virtual image in the user's eyes is associated with the real scene covered by the virtual image in the user's eyes, achieving the "immersive" augmented reality experience effect of virtual-real alignment, improving the user's experience and interest in use, and thus solving the technical problem in the related art that the driver and passengers cannot obtain a good "immersive" augmented reality experience of virtual-real alignment while the line of sight moves, reducing the driver and passengers' interest in use.

[0075] In the embodiment of the present invention, a vehicle is used as an example for illustrative purposes. Among them, the AR-HUD is installed in front of the instrument panel behind the steering wheel directly in front of the driver, and the virtual image screen is projected onto the virtual image plane where the AR-HUD is located. The driving recorder (denoted as Cam adas)It is installed at the interior rearview mirror and faces the forward direction of the vehicle.

[0076] Optionally, the alignment method of the display device can be applied to an in-vehicle augmented reality system, and the display device is an augmented reality head-up display AR-HUD.

[0077] The embodiments of the present invention will be described in detail below in conjunction with each step.

[0078] Step S202: Determine the line-of-sight focus area of the user in the target vehicle, where at least one physical reference object is included in the line-of-sight focus area.

[0079] In the embodiments of the present invention, the target vehicle includes but is not limited to: bicycles, cars, driving simulators, AR windows, etc. The line-of-sight focus area is the area mainly viewed in the real scene outside the vehicle that can be observed in the user's field of vision. The physical reference objects include but are not limited to: lane lines, road signs, pedestrians, vehicles ahead, etc., which are reference objects that can assist the user in driving decisions in any scene.

[0080] Optionally, the step of determining the line-of-sight focus area of the user in the target vehicle includes: determining the focused line of sight of the user in front of the target vehicle based on the coordinates of the line-of-sight projection object of the user; performing an intersection process on the focused line of sight and the plane where the ground in front of the target vehicle is located to obtain the user's gaze position point; and determining the line-of-sight focus area of the user in the target vehicle based on the gaze position point.

[0081] Optionally, a driver monitoring system (DMS) is used to track the coordinates of the line-of-sight projection object of the user, and the driver monitoring system calculates the eye coordinates of the user at the current moment (i.e., the coordinates of the line-of-sight projection object). Then, the focused line of sight from the user's eyes to the ground or physical reference object in front of the target vehicle can be determined to obtain the user's gaze position point. The embodiments of the present application do not limit the method of obtaining the line of sight and the viewing point from the collected line-of-sight projection object of the user. The line of sight and the viewing point of the user can be determined based on the corneal pupil model or deep learning. Similarly, the embodiments of the present application do not limit the method of determining the line-of-sight focus area in the user's field of vision based on the gaze position point. Traditional geometric methods or deep learning can be used.

[0082] Step S204: Obtain the captured image of the original scene collected by the first camera, and extract the entity identification points in the captured image.

[0083] In an embodiment of the present invention, the entity identification point is a pixel point on the entity reference object. Taking the driving scenario as an example, the entity reference object can refer to the entities on the road, including but not limited to: lane lines, road signs, pedestrians, vehicles ahead, etc., which are reference objects that can assist users in making driving decisions in any scenario. Since the actual reference object contains rich scene information, user decisions often rely on the actual reference object in the scene, and actively extracting the contained scene information provides the possibility for immersive guidance.

[0084] Step S206, determine the two-dimensional image mapping relationship between the captured image and the virtual image in the user's current field of view through the pre-obtained calibration parameters, where the virtual image is the image to be projected by the display device.

[0085] Optionally, in this embodiment, the calibration parameters can be determined by the virtual-real alignment calibration method in combination with the eye-simulating auxiliary camera according to the position states of the display device and the user's eyes. Specifically, during the calibration process, the eye-simulating auxiliary camera is used to simulate the position of the human eye, which can accurately quantify the coordinates of the user's eyes in space, improve the calibration efficiency and ensure the calibration accuracy, realizing the industrial process of calibration. If direct eye calibration is used, the user cannot be fixed at the calibration position, and the calibration space range will be limited, affecting the accuracy of the calibration parameters.

[0086] In the actual usage scenario, different users have different height requirements for the display device, and the height of the display device is not fixed. Similarly, during use, the user's line of sight does not remain unchanged. Changes in sitting posture, head movement, scene jolting, etc. can all cause changes in the position and line of sight of the user's eyes. In this embodiment, by determining the correspondence between the calibration parameters of the user's eyes and the display device at any position, the virtual image of the display device can always be kept in a virtual-real alignment state with the scene real image following the movement of the line of sight, without being terminated due to changes in the user's sitting posture, head movement, and display device adjustment.

[0087] Another option is that the alignment method further includes: when the positions of the display device and the user's eyes are kept at a fixed position, determining the calibration parameters by the virtual-real opening and closing calibration method in combination with the eye-simulating auxiliary camera at this fixed position; or when the positions of the display device and / or the user's eyes change, adjusting the device with the changed position to multiple different calibration positions and repeating the virtual-real opening and closing calibration method to determine the normalized parameter calibration correspondence, and determining the calibration parameters according to the actual position of the device and the normalized parameter calibration correspondence.

[0088] Optionally, the device with the changed position includes: the display device and / or the eye-simulating auxiliary camera.

[0089] Optionally, the actual positions of the display device and the user's eyes are obtained through the driver monitoring system.

[0090] Optionally, the calibration parameters include: a first two-dimensional image mapping relationship between a first image captured by a first camera and a second image captured by a human eye simulation auxiliary camera, and a second two-dimensional image mapping relationship between a third image captured by the human eye simulation auxiliary camera and a virtual calibration pattern. Specifically, the present application does not limit the order of determining the calibration parameters, and they can be calculated sequentially or in parallel.

[0091] In an embodiment of the present invention, the calibration parameters are obtained in advance through a calibration operation. Figure 3 FIG. is a schematic diagram of a scenario of an optional calibration method for a display device according to an embodiment of the present invention. As Figure 3 shown, for calibration, it is necessary to prepare: one physical calibration pattern (1), whose form is not limited (for example, a calibration board, a calibration cloth, a ground spray-painted calibration pattern, etc., and the style of the calibration pattern and the content displayed on the pattern are not specifically limited), one virtual calibration pattern (2), whose resolution is the same as the resolution of the AR-HUD virtual image and can be displayed by the AR-HUD, and one auxiliary calibration camera (in this embodiment, it refers to a human eye simulation auxiliary camera, denoted as Cam eye ), where the style of the calibration pattern includes but is not limited to: checkerboard, dot grid, graphic code (one-dimensional code, two-dimensional code), etc. Cam eye is installed at the human eye position to simulate the human eye. A physical calibration pattern (1) is laid on the ground in front of the vehicle. The feature points in the physical calibration pattern (1) can be distinguished in the auxiliary calibration camera Cam eye and the main calibration camera Cam adas (in this embodiment, it refers to the first camera, which is used to capture the real scene outside the vehicle and can be located at the position where the driving recorder is installed, and its type includes but is not limited to: visible light camera, infrared camera, etc.), and the feature points in the virtual calibration pattern (2) can be distinguished in Cam eye .

[0092] The calibration methods in this embodiment include multiple situations. Each calibration method will be described separately below, and the following calibration methods in this embodiment can be combined with each other.

[0093] (1) The first calibration method: display device opening and closing calibration

[0094] Optionally, the calibration parameters are determined by a virtual-real alignment calibration method in combination with the human eye simulation auxiliary camera, including: by controlling the opening and closing of the display device, combining the physical calibration pattern and the virtual calibration pattern, and determining the calibration parameters according to the images captured by the human eye simulation auxiliary camera and the first camera.

[0095] The display device in this embodiment may include but is not limited to: an AR-HUD device. In this embodiment, the virtual image plane of the AR-HUD device is used for illustration.

[0096] Optionally, by controlling the turning on and off of the display device, combining the physical calibration pattern and the virtual calibration pattern, the calibration parameters are determined according to the images captured by the human eye simulation auxiliary camera and the first camera, including: after turning off the display device, combining the physical calibration pattern, determining the third two-dimensional image mapping relationship between the first image collected by the first camera and the second image collected by the human eye simulation auxiliary camera; after turning on the display device, combining the virtual calibration pattern, determining the fourth two-dimensional image mapping relationship between the third image collected by the human eye simulation auxiliary camera and the virtual calibration pattern; representing the third two-dimensional image mapping relationship and the fourth two-dimensional image mapping relationship as calibration parameters.

[0097] Optionally, the physical calibration pattern is placed in front of the target vehicle, and the virtual calibration pattern is displayed on the virtual image plane projected by the display device, where the physical calibration pattern includes at least one physical feature point, and the virtual calibration pattern includes at least one virtual feature point.

[0098] As Figure 3 shown, the AR-HUD virtual image plane can be turned off first, and Cam adas and Cam eye are used to photograph the calibration pattern (1) (i.e., the physical calibration pattern, located on the ground). According to the feature points of the photographed physical calibration pattern (1), the two-dimensional image transformation relationship H adas2eye (i.e., the third two-dimensional image mapping relationship) between the two is calculated; then the AR-HUD virtual image plane is turned on, and Cam eye is used to photograph the virtual calibration pattern (2) (i.e., the virtual image plane displayed by the display device, which is in front of the front windshield and perpendicular to the ground). According to the feature points of the photographed virtual calibration pattern (2) (i.e., the fourth feature points of the third image), the two-dimensional image transformation relationship H eye2image (i.e., the fourth two-dimensional image transformation relationship) between the feature points obtained by photographing the virtual calibration pattern (2) and the feature points finally displayed on the AR-HUD virtual image plane (i.e., the third feature points in the virtual image plane) is calculated. Finally, two sets of calibration parameters are obtained: the two-dimensional image transformation relationship H adas2eye and the two-dimensional image transformation relationship H eye2image . The embodiments of the present application do not limit the extraction of feature points and the method of determining the two-dimensional image transformation relationship based on feature points. Traditional methods can be used, for example, extracting the description information of feature points, Ransac to eliminate incorrect matches, and solving the transformation relationship by equations. Depth estimation transformation relationships can also be used, such as HomographyNet. In addition, the physical calibration pattern and the virtual calibration pattern are used separately when obtaining the two sets of calibration parameters, and the feature points do not need to correspond one by one, so the styles of the two do not need to be the same.

[0099] In the alignment step after obtaining the two sets of calibration parameters, Cam adasCollect the scene image, extract the pixel points of the reference object in the scene image, such as the pixel points where the lane lines are located, denoted as p, through the two-dimensional image transformation relationship H adas2eye and the two-dimensional image transformation relationship H eye2image , calculate the illuminated pixel points p' in the AR-HUD = H eye2image *H eye2image *p. The illuminated pixel points are the virtual lane lines displayed on the display device. That is, under the field of view of the user's current position, the virtual lane lines displayed on the display device cover the real white lane lines.

[0100] In the embodiments of the present application, through the virtual calibration pattern and the physical calibration pattern, the two-dimensional image pixel correspondence relationship between the image collected by the driving recorder under the user's field of view and the virtual image displayed on the display device can be established without three-dimensional reconstruction, which can be achieved only by turning on and off the display device. It is simple to operate, improves the calculation efficiency and calibration speed, and greatly reduces the requirement for hardware computing power. In addition, the human eye simulation auxiliary camera is installed at the human eye position for simulating the human eye, which helps to realize the industrialization of the calibration process and enriches the usage scenarios.

[0101] (2) The second calibration method, equivalent page calibration

[0102] In the embodiments of the present invention, if the real ground is used as the real scene area covered by the AR-HUD during the calibration operation, in actual situations, the real ground covered is about 30-100 meters away from the vehicle head. Therefore, the requirements for the space of the calibration site and the size of the physical calibration pattern (1) will be extremely high. In order to further adapt to the actual situations of different calibration scenarios, the embodiments of the present application shorten the space distance required for the physical calibration pattern (1) during the calibration process through the equivalent physical calibration pattern.

[0103] Optionally, the alignment method further includes: the physical calibration pattern is replaced by an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.

[0104] Figure 4 is a schematic diagram of the basic principle of pinhole imaging in three-dimensional vision according to an optional embodiment of the present invention. As Figure 4 shown, when the object (1) with a larger size at a farther object distance is parallel to the object (2) with a smaller size at a closer object distance, and their object height ratio is consistent with the object distance ratio, through the pinhole projection center, the images (1) and (2) on the same imaging plane are the same. In other words, according to the basic principle of pinhole imaging in three-dimensional vision, when the larger object at a farther object distance is parallel to the smaller object at a closer object distance, and their object height ratio is consistent with the object distance ratio, the images formed by the pinhole optical system are the same. Therefore, according to this principle, according to the actual situations of different calibration sites, the distance from the physical calibration pattern (1) to the calibration vehicle during the calibration of the AR-HUD can be shortened.

[0105] Optionally, by controlling the turning on and off of the display device, combining the equivalent physical calibration pattern and the virtual calibration pattern, the calibration parameters are determined according to the images captured by the human eye simulation auxiliary camera and the first camera, including: after turning off the display device, restoring the virtual calibration pattern equivalent to the ground through the equivalent physical calibration pattern and projecting it onto the fourth image generated by the first camera, and the human eye simulation auxiliary camera collecting the fifth image of the equivalent physical calibration pattern to determine the fifth two-dimensional image mapping relationship between the fourth image and the fifth image; after turning on the display device, combining the virtual calibration pattern, determining the sixth two-dimensional image mapping relationship between the sixth image collected by the human eye simulation auxiliary camera and the virtual calibration pattern; and characterizing the fifth two-dimensional image mapping relationship and the sixth two-dimensional image mapping relationship as calibration parameters.

[0106] Optionally, restoring the virtual calibration pattern equivalent to the ground through the equivalent physical calibration pattern and projecting it onto the fourth image generated by the first camera includes: determining the ground clearance value and the preset scaling ratio of the equivalent physical calibration pattern according to the first spatial pose of the first camera, the human eye simulation auxiliary camera, and the equivalent physical calibration pattern obtained by calibration; according to the ground clearance value and the preset scaling ratio, combining the principle of light propagation to restore the virtual calibration pattern equivalent to the ground and calculate the second spatial pose of the virtual calibration pattern; and projecting the virtual calibration pattern onto the imaging plane of the first camera to generate the fourth image in combination with the second spatial pose.

[0107] Specifically, on the basis of the first calibration method, according to the principle of pinhole imaging, the physical calibration pattern is replaced by an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera. Figure 5 It is a schematic diagram of the scenario of an optional calibration method for a display device according to an embodiment of the present invention Figure 2 , such as Figure 5 shown, in the embodiment of the present invention, an "equivalent physical calibration pattern (3)" is introduced to replace the physical calibration pattern (1). In the field of view of Cam eye , an equivalent physical calibration pattern (3) that is higher than and parallel to the physical calibration pattern (1) is used to replace the physical calibration pattern (1). The scaling ratio between the equivalent physical calibration pattern (3) and the physical calibration pattern (1) is equivalent to the ratio of the ground clearance of the equivalent physical calibration pattern (3) to the ground clearance of Cam eye , and can also be equivalent to the ratio of the horizontal distance from the physical calibration pattern (1) to Cam eye to the horizontal distance from the equivalent physical calibration pattern (3) to Cam eye . The arrangement and requirements of the remaining calibration devices are the same as Figure 3Based on the principle of pinhole imaging, the calibration working distance can be greatly shortened, the requirements for the calibration site can be reduced, and the size of the equivalent physical calibration pattern (3) can be reduced relative to the physical calibration pattern (1).

[0108] Since the equivalent physical calibration pattern (3) is only adapted to Cam eye Field of view, its projection relationship does not satisfy Cam adas The field of view of Cam can be calculated separately eye ,Cam adas The pose relationship between the three and the physical calibration pattern (3) is calculated, and the equivalent virtual calibration pattern (1) is restored according to the height above the ground of the equivalent physical calibration pattern (3), and then reprojected to Cam adas , where reprojection refers to the process of eye The pixel points of the equivalent physical calibration pattern (3) in the field of view are projected onto the Cam adas The pixel points that generate new images in the field of view. Figure 5 As shown, the first camera Cam can be calibrated independently eye And human eye simulation auxiliary camera Cam adas The camera internal parameters (i.e. parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc.) are not limited in this application. eye ,Cam adas and the spatial pose (i.e., the first spatial pose) of the equivalent physical calibration pattern (3), the spatial pose not only provides the positional relationship between the devices, but also can provide the height h of the equivalent physical calibration pattern (3) from the ground and the preset zoom ratio information; according to the height h of the equivalent physical calibration pattern (3) from the ground (in this embodiment, it refers to the height value of the equivalent physical calibration pattern from the ground) and the principle of light propagation along a line, the spatial pose (in this embodiment, it refers to the second spatial pose) of the virtual calibration pattern (1) located on the ground is calculated, and finally the feature points of the virtual calibration pattern (1) are reprojected to the Cam adas , a new image is generated (i.e., a fourth image is generated by projecting onto the imaging surface of the first camera).

[0109] The alignment steps after obtaining the two sets of calibration parameters are consistent with the steps included in the first calibration method and will not be repeated. In this embodiment, by combining the geometric relationship and the pinhole imaging principle, an equivalent calibration plane is introduced to calculate the virtual ground and reproject it, which can greatly shorten the calibration working distance and reduce the requirements for the calibration site.

[0110] (3) The third calibration method: variable space calibration

[0111] When the positions of the display device and the user's eyes are kept at fixed positions, the first calibration method and the second calibration method can be used to obtain calibration parameters at the fixed positions.

[0112] However, the user's line of sight does not remain unchanged. Changes in sitting posture, head movement, scene jolting, etc. can all cause changes in the position of the user's eyes and the line of sight. When the position of the display device and / or the user's eyes changes, by adjusting the device whose position has changed to multiple different calibration positions, the first calibration method and the second calibration method are repeated at the multiple different calibration positions to determine the normalization parameter calibration correspondence, and the calibration parameters are determined according to the actual position of the device and the normalization parameter calibration correspondence.

[0113] Optionally, the normalization parameter calibration correspondence is the mapping relationship between any normalization parameter and the corresponding calibration parameter. Among them, the normalization parameter is the proportion of the position movement amount of the display device and / or the human eye simulation auxiliary camera in the variable stroke.

[0114] Optionally, when the position of the display device and / or the user's eyes changes, by adjusting the device whose position has changed to multiple different calibration positions, the virtual-real opening and closing calibration method is repeated to determine the normalization parameter calibration correspondence, and the calibration parameters are determined according to the actual position of the device and the normalization parameter calibration correspondence, including: adjusting the device whose position has changed to at least 2 different calibration positions in the variable space to repeat the virtual-real opening and closing calibration method to determine the normalization parameter calibration correspondence; determining the normalization position parameter on the variable space based on the actual position of the device, and determining the calibration parameters according to the normalization position parameter and the normalization parameter calibration correspondence.

[0115] In this embodiment, the types included in the variable space include but are not limited to: variable linear stroke, variable planar stroke, variable three-dimensional space stroke. In this embodiment, the changeable position (stroke) is represented by a normalization parameter. Through the corresponding relationships of several pre-calibrated groups of "normalization parameter - calibration parameter", the calibration parameters corresponding to any normalization parameter can be fitted. For the device whose position has changed, the method for obtaining the stroke information in this application embodiment is not limited. For example, it can be detected and provided by an external sensor.

[0116] The minimum value of the above-mentioned "several" is that if the changeable position is approximately linear (such as the up and down adjustment of the virtual image plane of the AR-HUD), it should be greater than or equal to 2, and these several groups of corresponding relationships do not coincide at the same point; if the changeable position is approximately a planar array (such as the movement of the human eye in the DMS camera), it should be greater than or equal to 3, and these several groups of corresponding relationships do not coincide on the same line; further, if the changeable position is approximately a three-dimensional range (such as the movement of the simulated human eye camera in the eye box), it should be greater than or equal to 4, and these several groups of corresponding relationships do not coincide on the same plane.

[0117] Optionally, when the variable space is a variable linear stroke, the alignment method includes: adjusting the device to at least two different calibration positions within the variable linear stroke, repeating the virtual-real opening and closing calibration method, and determining the normalization parameter calibration correspondence, where the at least two different calibration positions include: the highest display reference position and the lowest display reference position of the device within the variable linear stroke.

[0118] Optionally, adjusting the device with a changed position to at least two different calibration positions within the variable space, repeating the virtual-real opening and closing calibration method, and determining the normalization parameter calibration correspondence includes: adjusting the device with a changed position to at least two different calibration positions within the variable space, and repeating the virtual-real opening and closing calibration method at different calibration positions to obtain a set of calibration parameters, where each calibration parameter in the set of calibration parameters corresponds one-to-one to each calibration position; determining a set of normalization parameters based on all calibration positions and the variable space; and fitting and determining the normalization parameter calibration correspondence by combining the set of calibration parameters and the set of normalization parameters.

[0119] Taking a mobile AR-HUD as an example, the AR-HUD can adjust the virtual image plane in N different strokes (where N >= 2, indicating that there should be at least two strokes, the highest and the lowest). After that, calibration work is carried out to obtain N sets of calibration parameters. The N sets of calibration parameters respectively correspond to N values between 0 and 1 of the normalized stroke of the AR-HUD virtual image plane, and the normalization parameter calibration correspondence can be determined. Thereafter, taking the normalized stroke of the AR-HUD virtual image plane as a single independent variable (that is, the AR-HUD can move up and down within a certain range, and the movement ratio is the single independent variable. For example, if the movement range is 10 mm, the movement ratio of moving 1 mm is 0.1 (i.e., the single independent variable is 0.1)), the calibration parameters corresponding to the position when the AR-HUD virtual image plane moves to this position can be obtained by combining the normalization parameter calibration correspondence. Similarly, when the user's eyes move on the variable linear stroke, the movement of the human eye is simulated by an auxiliary camera for the human eye, and the calibration parameters corresponding to any position are determined by the above method.

[0120] Figure 6 It is a schematic diagram of a scenario of an optional calibration method for a display device according to an embodiment of the present invention Figure 3 , as Figure 6 shown, taking the movement of the AR-HUD virtual image plane as an example, in the embodiment of the present invention, the position of the display device can be adjusted within the variable linear stroke. When adjusted to the highest, it is the highest display reference position, and when adjusted to the lowest, it is the lowest display reference position. The AR-HUD virtual image plane is adjusted up and down by N = 2 extreme strokes, that is, adjusted to the highest display reference position and the lowest display reference position. When the AR-HUD virtual image plane is adjusted to the highest, that is, when the normalized stroke value is 1, after closing the AR-HUD virtual image plane, use Cam adas and Cameye Take a picture of the calibration pattern (1) of No. 1, and calculate the two-dimensional image transformation relationship H between the characteristic points of the photographed physical calibration pattern (1) of No. 1 (1) adas2eye , then turn on the virtual image plane of the AR-HUD and use Cam eye Take a picture of the virtual image plane of the HUD of No. 1 (that is, the virtual image plane displayed by the display device, which is in front of the front windshield and perpendicular to the ground). According to the characteristic points of the photographed virtual image plane of the HUD of No. 1, calculate the two-dimensional image transformation relationship H between the characteristic points obtained by photographing the virtual image plane of the HUD of No. 1 and the characteristic points finally displayed on the virtual image plane of the AR-HUD (1) eye2image . When the virtual image plane of the AR-HUD is adjusted to the lowest, that is, when the normalized travel value is 0, after turning off the virtual image plane of the AR-HUD, use Cam adas and Cam eye Take a picture of the calibration pattern (1) of No. 2, and calculate the two-dimensional image transformation relationship H between the characteristic points of the photographed physical calibration pattern (1) of No. 2 (2) adas2eye , then turn on the virtual image plane of the AR-HUD and use Cam eye Take a picture of the virtual image plane of the HUD of No. 2, and calculate the two-dimensional image transformation relationship H between the characteristic points obtained by photographing the virtual image plane of the HUD of No. 2 and the characteristic points finally displayed on the virtual image plane of the AR-HUD (2) eye2image , and finally obtain two sets of calibration parameters at two reference positions: the two-dimensional image transformation relationship H (1) adas2eye 、the two-dimensional image transformation relationship H (1) eye2image 、the two-dimensional image transformation relationship H (2) adas2eye and the two-dimensional image transformation relationship H (2) eye2image . Combine the calibration parameter groups and the corresponding normalized parameter groups to fit and determine the corresponding relationship of the normalized parameter calibration. The present application does not limit the mathematical method for establishing the corresponding relationship. For example, interpolation or fitting algorithms can be used

[0121] In the subsequent alignment step, taking the travel movement of the virtual image plane of the AR-HUD as an example, in the embodiment of the present invention, the in-vehicle computer extracts the real scene (such as lane lines) in the image of the driving recorder (Cam adasFor the pixel points corresponding thereto in [[ID=]], according to the travel of the virtual image plane of the AR-HUD at this time and the calibration correspondence of the normalization parameters, calculate the calibration parameters of the AR-HUD virtual image plane at the current travel, and then map the lane line pixel points in the driving recorder to the pixel points on the AR-HUD virtual image. At this time, in the driver's line of sight, the lane lines and the like displayed on the AR-HUD virtual image will cover the real lane lines in the driver's line of sight, and the virtual image screen of the AR-HUD can be adjusted up and down according to the habitual line of sight directions of different drivers to achieve virtual-real alignment.

[0122] Optionally, when the variable space is a variable plane travel, the alignment method includes: adjusting the device to at least 3 different calibration positions within the variable plane travel, repeating the virtual-real opening and closing calibration method, and determining the calibration correspondence of the normalization parameters, where at least 3 different calibration positions include: 3 different non-collinear extreme reference positions of the device.

[0123] In the embodiment of the present invention, taking the eyes of a mobile user as an example, the DMS is used to photograph the moving position of the human eye simulation auxiliary camera in the cab, and this moving position can be approximately in a two-dimensional plane (i.e., the preset plane area), and the normalized coordinates of at least three vertices of this two-dimensional plane can be set to (0,0), (0,1), (1,1).

[0124] In the embodiment of the present invention, the normalized coordinates of the variable plane are illustrated by four vertices. The human eye simulation auxiliary camera moves to the lower left extreme position within the variable plane travel in the cab, which is the normalized coordinate (0,0) (i.e., the first display reference position), and the DMS records the coordinates of the human eye in the DMS camera at this time, denoted as d (0) (i.e., the first coordinate of the line-of-sight projection body corresponding to the first display reference position). The human eye simulation auxiliary camera moves to the lower right, upper right, and upper left extreme positions within the feasible space in the cab, which are the normalized coordinates (0,1), (1,1), (1,0), and the DMS records the coordinates of the human eye in the DMS camera at these three positions, denoted as d (1) (i.e., the second coordinate of the line-of-sight projection body corresponding to the second display reference position), d (2) (i.e., the third coordinate of the line-of-sight projection body corresponding to the third display reference position), d (3) (i.e., the fourth coordinate of the line-of-sight projection body corresponding to the fourth display reference position). According to the correspondence between d (0) , d (1) , d (2) , d (3) and the normalized coordinates (0,0), (0,1), (1,1), (1,0), the fitting normalized matrix H normal can be obtained. Based on the coordinates of the human eye in the DMS camera recorded by the DMS and the fitting normalized matrix H normal, the normalized coordinates of the human eye within the preset plane area at this time can be obtained.

[0125] In the embodiment of the present invention, since the movement of the human eye is difficult to quantify, the extreme positions of the movement are used as the calibration basis, that is, the first display reference position, the second display reference position, the third display reference position, and the fourth display reference position are the four extreme positions. There are M non-collinear extreme positions (M>=3) where the human eye can move in the cab. Then, calibration work is carried out respectively to obtain M groups of calibration parameters. The M groups of calibration parameters respectively correspond to the values of the M two-dimensional normalized positions of the human eye photographed by the DMS in the cab. Combining the M groups of calibration parameter groups and the corresponding M two-dimensional normalized position values, the normalization parameter calibration correspondence is determined by fitting. The present application does not limit the mathematical method for establishing the correspondence. For example, interpolation or fitting algorithms can be used. Subsequently, using the value of the two-dimensional normalized position of the human eye photographed by the DMS in the cab as a single independent variable, the corresponding calibration parameters at any position of the human eye in the cab can be obtained.

[0126] In the subsequent alignment step, taking the movement of the human eye as an example, in the embodiment of the present invention, the vehicle-mounted computer extracts the pixel points corresponding to the real scene (such as lane lines) in the dashcam image (Cam adas ), according to the coordinates (two-dimensional coordinates) of the human eye in the DMS camera at this time, combined with the fitted normalization matrix H normal , determine the normalized coordinates of the human eye within the preset plane area. According to the normalized coordinates of the human eye within the preset plane area at this time and the normalization parameter calibration correspondence, calculate the calibration parameters of the human eye during the current journey, and then map the lane line pixel points in the dashcam to the pixel points on the AR-HUD virtual image. At this time, in the driver's line of sight, the lane lines and the like displayed on the AR-HUD virtual image will cover the real lane lines in the driver's line of sight, and the virtual image screen of the AR-HUD can be adjusted up and down according to the habitual line-of-sight directions of different drivers to achieve virtual-real alignment.

[0127] Optionally, when the variable space is a variable three-dimensional space journey, the alignment method includes: adjusting the device to at least 4 different calibration positions within the variable three-dimensional space journey, repeating the virtual-real opening and closing calibration method, and determining the normalization parameter calibration correspondence, where at least 4 different calibration positions include: 4 non-coplanar extreme reference positions of the device.

[0128] In the embodiment of the present invention, calibrating the normalization parameters with four non-coplanar extreme reference positions is similar to the variable space being a variable two-dimensional plane. The human eye simulation auxiliary camera moves to 4 non-coplanar extreme reference positions within the variable plane journey in the cab, repeating the virtual-real calibration method, and fitting the normalization matrix H according to the extreme reference positions and the corresponding normalized coordinates normal2, Based on the DMS recording the coordinates of the human eye in the DMS camera and fitting the normalization matrix H normal2 , the normalized coordinates of the human eye in the preset spatial region can be obtained at this time.

[0129] Similarly, in the embodiments of the present invention, since the movement of the human eye is difficult to quantify, the extreme positions of the movement are used as the calibration basis, that is, the first display reference position, the second display reference position, the third display reference position, and the fourth display reference position are the four extreme positions. There are P different non-coplanar extreme positions (P>=4) where the human eye can move in the cab. Then, the calibration work is carried out respectively to obtain P groups of calibration parameters. The P groups of calibration parameters respectively correspond to the values of the three-dimensional normalized positions of the human eye photographed by the DMS in the cab. Combining the P groups of calibration parameter groups and the corresponding values of the P three-dimensional normalized positions, the normalization parameter calibration correspondence is determined by fitting. The present application does not limit the mathematical method for establishing the correspondence. For example, interpolation or fitting algorithms are used. Subsequently, taking the value of the three-dimensional normalized position of the human eye photographed by the DMS in the cab as a single independent variable, the corresponding calibration parameters when the human eye is at any position in the cab can be obtained.

[0130] In the subsequent alignment step, taking the movement of the human eye as an example, in the embodiments of the present invention, the vehicle-mounted computer extracts the pixel points corresponding to the real scene (such as lane lines) in the image of the driving recorder (Cam adas ). According to the coordinates of the human eye in the DMS camera at this time, combined with the fitting normalization matrix H normal2 , the normalized coordinates (three-dimensional coordinates) of the human eye in the preset plane region are determined. According to the normalized coordinates of the human eye in the preset plane region at this time and the normalization parameter calibration correspondence, the calibration parameters of the human eye during the current journey are calculated, and then the lane line pixel points in the driving recorder are mapped to the pixel points on the AR-HUD virtual image. At this time, in the driver's line of sight, the lane lines and the like displayed on the AR-HUD virtual image will cover the real lane lines in the driver's line of sight, and the virtual image screen of the AR-HUD can be adjusted up and down according to the habitual line-of-sight directions of different drivers to achieve virtual-real alignment.

[0131] The embodiments of the present application can obtain the calibration parameters at any position of the movable device by using a mathematical algorithm to establish a function for generating calibration parameters with a single variable, so as to achieve real-time dynamic alignment. Optionally, the alignment method further includes: hanging a human eye simulation auxiliary camera on the robotic arm to adjust the position of the human eye simulation auxiliary camera and providing the corresponding three-dimensional normalized coordinates.

[0132] In the embodiments of the present invention, an auxiliary calibration camera Cam eye can be mounted on the robotic arm, and the robotic arm can be controlled to simulate the movement of the human eye up, down, left, right, forward and backward in the feasible region of the three-dimensional space in the cab. Due to the introduction of the robotic arm, the Cam simulating the human eyeeye The three-dimensional normalized coordinates of it in the feasible region can be given by a robotic arm or a DMS camera capable of detecting depth (i.e., the normalized coordinates within a preset planar region. For example, if one corner of the feasible region is defined as (0, 0, 0), then the corner farthest from this corner is (1, 1, 1)).

[0133] In the embodiments of the present invention, if a robotic arm is used to provide the three-dimensional normalized coordinates, the transformation relationship between the robotic arm coordinate system and the DMS camera coordinate system also needs to be calibrated. If a DMS camera capable of detecting depth is directly used to provide the three-dimensional normalized coordinates, there is no need for the transformation relationship between the two.

[0134] Although the embodiments of the present application only illustrate the above calibration methods, any combination of two, three, four, or five of these calibrations should also be within the protection scope of this embodiment.

[0135] Step S208: Project the entity identification points into the virtual image according to the two-dimensional image mapping relationship, where the virtual identifier in the virtual image is aligned with the entity reference object in the line-of-sight focusing area.

[0136] Figure 7 is an optional alignment effect diagram according to the embodiments of the present invention. Figure 7 The frame indicated by 101 in it is the virtual lane line displayed in the virtual image range of the AR-HUD, and the virtual lane line covers the real white lane line (102) in the Cam eye field of view. The vehicle-mounted computer identifies information such as lane lines, pedestrians, and vehicles ahead existing in the Cam adas image that can assist the driver in making driving decisions through a semantic recognition algorithm (such as a neural network, etc.), and extracts the pixels corresponding to them in the Cam adas image, and then maps them to the AR-HUD virtual image through the two-dimensional image transformation relationship. At this time, in the Cam eye field of view, the lane lines and the like displayed by the AR-HUD virtual image will cover the real lane lines in the Cam eye field of view, realizing the alignment of virtual and real.

[0137] In the embodiments of the present invention, it is possible to process the input human eye position and output the virtual image in real time, and the real-time performance depends on the lowest frame rate among the DMS camera, the Cam adas camera, or the AR-HUD display screen.

[0138] Optionally, the alignment method further includes: projecting the physical identification points into the virtual image in combination with the two-dimensional image mapping relationship and the attitude of the target vehicle. Specifically, due to the complex actual usage scenarios, the overall state of the target vehicle has attitude changes. For example, when the vehicle brakes or starts, the front of the vehicle correspondingly has changes such as lifting or lowering. If only relying on the calibration parameters under the normal and stable front of the vehicle without considering the attitude of the target vehicle, it will result in inconsistent scenes in the user's line of sight and the virtual image plane, which will affect the alignment effect. In the embodiments of the present application, the alignment is corrected by further combining the attitude of the target vehicle on the basis of the two-dimensional image mapping relationship, improving the accuracy and robustness of the virtual-real alignment, further expanding the application scenario range. In addition, the present application does not limit the method for obtaining the attitude of the target vehicle. For example, an external sensor can be used to obtain the attitude of the target vehicle in real time.

[0139] In the embodiments of the present application, through the virtual calibration pattern and the physical calibration pattern, the two-dimensional image pixel correspondence relationship between the image collected by the driving recorder in the user's field of view and the virtual image displayed by the display device can be established without three-dimensional reconstruction, and it can be achieved only by turning on and off the display device. While the operation is simple, the calculation efficiency and calibration speed are improved, and the requirement for hardware computing power is greatly reduced. In addition, the human eye simulation auxiliary camera is installed at the human eye position for simulating the human eye, which helps to industrialize the calibration process and enriches the usage scenarios.

[0140] Embodiment 2

[0141] A alignment device for a display device provided in this embodiment includes multiple implementation units, and each implementation unit corresponds to each implementation step in Embodiment 1 above.

[0142] Figure 8 is a schematic diagram of a alignment device for a display device according to an embodiment of the present invention, as Figure 8 shown, the alignment device may include: a first determination unit 81, a first acquisition unit 82, a first determination unit 83, and an alignment unit 84, where

[0143] The first determination unit 81 is used to determine the line-of-sight focus area of the user in the target vehicle, where at least one physical reference object is included in the line-of-sight focus area;

[0144] The first acquisition unit 82 is used to acquire the captured image of the first camera for collecting the original scene and extract the physical identification points in the captured image;

[0145] The first determination unit 83 is used to determine the two-dimensional image mapping relationship between the captured image and the virtual image in the user's current field of view through the pre-obtained calibration parameters, where the virtual image is the image to be projected by the display device;

[0146] An alignment unit 84 is configured to project the entity identification points into a virtual image according to a two-dimensional image mapping relationship, where a virtual identifier in the virtual image is aligned with an entity reference object in the line-of-sight focusing area.

[0147] The above alignment device for a display device can first determine the line-of-sight focusing area of a user in a target vehicle through a first determination unit 81. At least one entity reference object is included in the line-of-sight focusing area. The first acquisition unit 82 acquires a captured image of the original scene captured by a first camera and extracts the entity identification points in the captured image. The first determination unit 83 determines the two-dimensional image mapping relationship between the captured image and the virtual image in the current field of view of the user through pre-obtained calibration parameters, where the virtual image is an image to be projected by the display device. The alignment unit 84 projects the entity identification points into the virtual image according to the two-dimensional image mapping relationship, where the virtual identifier in the virtual image is aligned with the entity reference object in the line-of-sight focusing area. In this embodiment, after pre-calibrating to obtain calibration parameters, the real scene of the line-of-sight focusing area outside the vehicle is captured by a capturing device, and the entity identification points of the captured picture content are projected onto the virtual image, so that the virtual identifier in the virtual image in the user's field of view at the current moment is aligned with the entity reference object in the line-of-sight focusing area. The virtual image in the user's eyes is associated with the real scene covered by the virtual image in the user's eyes, achieving the "immersive" augmented reality experience effect of virtual-real alignment, improving the user's experience and usage interest, and thus solving the technical problem in the related art that the driver and passengers cannot obtain a good "immersive" augmented reality experience of virtual-real alignment while the line of sight moves, reducing the usage interest of the driver and passengers.

[0148] Optionally, the first determination unit includes: a first determination module configured to determine the focused line of sight of the user in front of the target vehicle based on the coordinates of the line-of-sight projection object of the user; a fixation point determination module configured to perform an intersection process on the focused line of sight and the plane where the ground in front of the target vehicle is located to obtain the fixation position point of the user; and a second determination module configured to determine the line-of-sight focusing area of the user in the target vehicle based on the fixation position point.

[0149] Optionally, the calibration parameters are determined by a virtual-real alignment calibration method in combination with the position states of the display device and the user's eyes and an eye simulation auxiliary camera.

[0150] Optionally, the alignment device further includes: a third determination unit configured to determine calibration parameters by means of a virtual-real opening and closing calibration method in combination with a human eye simulation auxiliary camera at the fixed position when the positions of the display device and the user's eyes are maintained at a fixed position; or, a fourth determination unit configured to, when the positions of the display device and / or the user's eyes change, repeat the virtual-real opening and closing calibration method by adjusting the device with a changed position to multiple different calibration positions and determine the normalization parameter calibration correspondence, and determine the calibration parameters according to the actual position of the device and the normalization parameter calibration correspondence.

[0151] Optionally, the actual positions of the display device and the user's eyes are obtained by a driver monitoring system.

[0152] Optionally, the calibration parameters include: a first two-dimensional image mapping relationship between a first image collected by a first camera and a second image collected by the human eye simulation auxiliary camera, and a second two-dimensional image mapping relationship between a third image collected by the human eye simulation auxiliary camera and a virtual calibration pattern.

[0153] Optionally, the third determination unit includes: a third determination module configured to determine the calibration parameters according to the images captured by the human eye simulation auxiliary camera and the first camera by controlling the turning on and off of the display device and in combination with a physical calibration pattern and a virtual calibration pattern.

[0154] Optionally, the third determination module includes: a first determination sub-module configured to, after turning off the display device, determine a third two-dimensional image mapping relationship between a first image collected by the first camera and a second image collected by the human eye simulation auxiliary camera in combination with the physical calibration pattern; a second determination sub-module configured to, after turning on the display device, determine a fourth two-dimensional image mapping relationship between a third image collected by the human eye simulation auxiliary camera and the virtual calibration pattern in combination with the virtual calibration pattern; and a third determination sub-module configured to represent the third two-dimensional image mapping relationship and the fourth two-dimensional image mapping relationship as calibration parameters.

[0155] Optionally, the alignment device further includes: a substitution unit configured to substitute the physical calibration pattern with an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.

[0156] Optionally, the third determination module further includes: a fourth determination sub-module, configured to, after turning off the display device, restore a virtual calibration pattern equivalent to that on the ground through an equivalent physical calibration pattern and project it onto a fourth image generated by the first camera, and an eye simulation auxiliary camera captures a fifth image of the equivalent physical calibration pattern, and determine a fifth two-dimensional image mapping relationship between the fourth image and the fifth image; a fifth determination sub-module, configured to, after turning on the display device, determine a sixth two-dimensional image mapping relationship between a sixth image captured by the eye simulation auxiliary camera and the virtual calibration pattern in combination with the virtual calibration pattern; a sixth determination sub-module, configured to characterize the fifth two-dimensional image mapping relationship and the sixth two-dimensional image mapping relationship as calibration parameters.

[0157] Optionally, the alignment device for the display device is further configured to: after turning off the display device, restore a virtual calibration pattern equivalent to that on the ground through an equivalent physical calibration pattern and project it onto a fourth image generated by the first camera, and an eye simulation auxiliary camera captures a fifth image of the equivalent physical calibration pattern, and determine a fifth two-dimensional image mapping relationship between the fourth image and the fifth image; after turning on the display device, determine a sixth two-dimensional image mapping relationship between a sixth image captured by the eye simulation auxiliary camera and the virtual calibration pattern in combination with the virtual calibration pattern; characterize the fifth two-dimensional image mapping relationship and the sixth two-dimensional image mapping relationship as calibration parameters.

[0158] Optionally, the alignment device for the display device is further configured to: determine a ground clearance value and a preset scaling ratio of the equivalent physical calibration pattern according to a first spatial pose of the first camera, the eye simulation auxiliary camera, and the equivalent physical calibration pattern obtained by calibration; according to the ground clearance value and the preset scaling ratio, restore a virtual calibration pattern equivalent to that on the ground through the equivalent physical calibration pattern in combination with the principle of light propagation and calculate a second spatial pose of the virtual calibration pattern; project the virtual calibration pattern onto the imaging plane of the first camera to generate a fourth image in combination with the second spatial pose.

[0159] Optionally, the normalization parameter calibration correspondence is a mapping relationship between any normalization parameter and a corresponding calibration parameter, where the normalization parameter is the proportion of the position movement amount of the display device and / or the eye simulation auxiliary camera in the variable stroke.

[0160] Optionally, the devices whose positions change include: the display device and / or the eye simulation auxiliary camera.

[0161] Optionally, when the position of the display device and / or the user's eyes changes, the alignment device for the display device is further configured to: adjust the device with the changed position to at least two different calibration positions in the variable space, repeat the virtual-real open-closed calibration method, and determine the normalization parameter calibration correspondence; determine the normalization position parameter on the variable space based on the actual position of the device, and determine the calibration parameter according to the normalization position parameter and the normalization parameter calibration correspondence.

[0162] Optionally, when the variable space is a variable linear travel, the alignment device for the display device is further configured to: adjust the device to at least two different calibration positions within the variable linear travel, repeat the virtual-real open-closed calibration method, and determine the normalization parameter calibration correspondence, where the at least two different calibration positions include: the highest display reference position and the lowest display reference position of the device in the variable linear travel.

[0163] Optionally, when the variable space is a variable planar travel, the alignment device for the display device is further configured to: adjust the device to at least three different calibration positions within the variable planar travel, repeat the virtual-real open-closed calibration method, and determine the normalization parameter calibration correspondence, where the at least three different calibration positions include: three non-collinear limit reference positions of the device.

[0164] Optionally, when the variable space is a variable three-dimensional space travel, the alignment device for the display device is further configured to: adjust the device to at least four different calibration positions within the variable three-dimensional space travel, repeat the virtual-real open-closed calibration method, and determine the normalization parameter calibration correspondence, where the at least four different calibration positions include: four non-coplanar limit reference positions of the device.

[0165] Optionally, the alignment device for the display device is further configured to: adjust the device with the changed position to at least two different calibration positions in the variable space, and repeat the virtual-real open-closed calibration method at different calibration positions to obtain a set of calibration parameters, where each calibration parameter in the set of calibration parameters corresponds one-to-one to each calibration position; determine a set of normalization parameters based on all the calibration positions and the variable space; and fit and determine the normalization parameter calibration correspondence by combining the set of calibration parameters and the set of normalization parameters.

[0166] Optionally, the alignment device for the display device is further configured to: adjust the position of the human eye simulation auxiliary camera by hanging the human eye simulation auxiliary camera with a robotic arm, and provide corresponding three-dimensional normalization coordinates.

[0167] According to another aspect of the embodiments of the present invention, a vehicle-mounted display system is further provided, including: a driver monitoring system for tracking the gaze focus area of a user in a target vehicle, where at least one physical reference object is included in the gaze focus area; a driving recorder for taking a picture of the gaze focus area to obtain a captured image; a vehicle-mounted controller connected to the driver monitoring system and the driving recorder respectively, and executing the alignment method for a display device according to any one of the above; a head-up display for projecting a virtual image to a preset position directly in front of the user, where a virtual identifier in the virtual image is aligned with the physical reference object in the gaze focus area.

[0168] According to another aspect of the embodiments of the present invention, a vehicle-mounted control device is further provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the alignment method for a display device according to any one of the above by executing the executable instructions.

[0169] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the alignment method for a display device according to any one of the above.

[0170] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0171] In the above embodiments of the present invention, the descriptions of the various embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0172] In several embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be electrical or other forms.

[0173] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, in each embodiment of the present invention, each functional unit can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0175] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0176] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An alignment method for a display device, characterized in that, Including: Determine the line-of-sight focusing area of the user in the target vehicle, where at least one physical reference object is included in the line-of-sight focusing area; Obtain a captured image of the original scene collected by the first camera, and extract the physical identification points in the captured image; Determine the two-dimensional image mapping relationship between the captured image and the virtual image in the current field of view of the user through pre-obtained calibration parameters, where the virtual image is the image to be projected by the display device; Project the physical identification points into the virtual image according to the two-dimensional image mapping relationship, where the virtual reference object in the virtual image is aligned with the physical reference object in the line-of-sight focusing area; Determine the calibration parameters by combining the human eye simulation auxiliary camera through the virtual-real alignment calibration method according to the position states of the display device and the user's eyes; When the positions of the display device and the user's eyes are kept at fixed positions, determine the calibration parameters by combining the human eye simulation auxiliary camera through the virtual-real opening and closing calibration method at the fixed positions; or when the positions of the display device and / or the user's eyes change, repeat the virtual-real opening and closing calibration method by adjusting the device with the changed position to multiple different calibration positions and determine the normalized parameter calibration correspondence, and determine the calibration parameters according to the actual position of the device and the normalized parameter calibration correspondence; When the positions of the display device and / or the user's eyes change, repeat the virtual-real opening and closing calibration method by adjusting the device with the changed position to multiple different calibration positions and determine the normalized parameter calibration correspondence, and determine the calibration parameters according to the actual position of the device and the normalized parameter calibration correspondence, including: adjusting the device with the changed position to at least 2 different calibration positions in the variable space and repeating the virtual-real opening and closing calibration method to determine the normalized parameter calibration correspondence; determining the normalized position parameter on the variable space based on the actual position of the device, and determining the calibration parameters according to the normalized position parameter and the normalized parameter calibration correspondence.

2. The alignment method according to claim 1, characterized in that The steps of determining the line-of-sight focusing area of the user in the target vehicle include: Based on the coordinates of the line-of-sight projection object of the user, determine the focusing line of sight of the user in front of the target vehicle; Perform an intersection process on the focusing line of sight and the plane where the ground in front of the target vehicle is located to obtain the user's fixation position point; Based on the fixation position point, determine the line-of-sight focusing area of the user in the target vehicle.

3. The alignment method according to claim 1, wherein The actual positions of the display device and the user's eyes are obtained through a driver monitoring system.

4. The alignment method according to claim 1, characterized in that The calibration parameters include: the first two-dimensional image mapping relationship between the first image collected by the first camera and the second image collected by the human eye simulation auxiliary camera, and the second two-dimensional image mapping relationship between the third image collected by the human eye simulation auxiliary camera and the virtual calibration pattern.

5. The alignment method according to claim 1, wherein Combining the human eye simulation assisted camera, the calibration parameters are determined by a virtual-real alignment calibration method, including: by controlling the turning on and off of the display device, combining a physical calibration pattern and a virtual calibration pattern, and determining the calibration parameters according to the images captured by the human eye simulation assisted camera and the first camera.

6. The alignment method according to claim 5, wherein By controlling the turning on and off of the display device, combining a physical calibration pattern and a virtual calibration pattern, and determining the calibration parameters according to the images captured by the human eye simulation assisted camera and the first camera, including: After turning off the display device, combining the physical calibration pattern, and determining the third two-dimensional image mapping relationship between the first image collected by the first camera and the second image collected by the human eye simulation assisted camera; After turning on the display device, combining the virtual calibration pattern, and determining the fourth two-dimensional image mapping relationship between the third image collected by the human eye simulation assisted camera and the virtual calibration pattern; Characterizing the third two-dimensional image mapping relationship and the fourth two-dimensional image mapping relationship as the calibration parameters.

7. The alignment method according to claim 5, wherein The physical calibration pattern is placed in front of the target vehicle, and the virtual calibration pattern is displayed on the virtual image plane projected by the display device, where the physical calibration pattern includes at least one physical feature point, and the virtual calibration pattern includes at least one virtual feature point.

8. The alignment method according to claim 5, wherein The alignment method further includes: the physical calibration pattern is replaced by an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation assisted camera.

9. The alignment method according to claim 8, wherein By controlling the turning on and off of the display device, combining the equivalent physical calibration pattern and the virtual calibration pattern, and determining the calibration parameters according to the images captured by the human eye simulation assisted camera and the first camera, including: After turning off the display device, by the equivalent physical calibration pattern, restoring the virtual calibration pattern equivalent to the ground and projecting it onto the fourth image generated by the first camera, and the fifth image collected by the human eye simulation assisted camera regarding the equivalent physical calibration pattern, determining the fifth two-dimensional image mapping relationship between the fourth image and the fifth image; After turning on the display device, combining the virtual calibration pattern, and determining the sixth two-dimensional image mapping relationship between the sixth image collected by the human eye simulation assisted camera and the virtual calibration pattern; Characterizing the fifth two-dimensional image mapping relationship and the sixth two-dimensional image mapping relationship as the calibration parameters.

10. The alignment method according to claim 9, characterized in that, Restoring the virtual calibration pattern equivalent to the ground and projecting it onto the fourth image generated by the first camera by the equivalent physical calibration pattern, including: According to the first spatial poses of the first camera, the human eye simulation assisted camera, and the equivalent physical calibration pattern obtained by calibration, determining the ground clearance value and the preset scaling ratio of the equivalent physical calibration pattern; According to the ground clearance value and the preset scaling ratio, combining the principle of light propagation to restore the virtual calibration pattern equivalent to the ground of the equivalent physical calibration pattern and calculating the second spatial pose of the virtual calibration pattern; Combining the second spatial pose to project the virtual calibration pattern onto the imaging plane of the first camera to generate the fourth image.

11. The alignment method according to claim 1, characterized in that, The calibration correspondence relationship of the normalization parameters is the mapping relationship between any normalization parameter and the corresponding calibration parameter. Among them, the normalization parameter is the ratio of the position movement amount of the display device and / or the human eye simulation auxiliary camera to the variable stroke.

12. The alignment method according to claim 1, characterized in that The devices whose positions are changed include: the display device and / or the human eye simulation auxiliary camera.

13. The alignment method according to claim 1, wherein When the variable space is a variable linear stroke, the alignment method includes: Adjusting the device to at least two different calibration positions within the variable linear stroke, repeating the virtual-real opening and closing calibration method, and determining the calibration correspondence relationship of the normalization parameters. Among them, the at least two different calibration positions include: the highest display reference position and the lowest display reference position of the device within the variable linear stroke.

14. The alignment method according to claim 1, wherein When the variable space is a variable planar stroke, the alignment method includes: Adjusting the device to at least three different calibration positions within the variable planar stroke, repeating the virtual-real opening and closing calibration method, and determining the calibration correspondence relationship of the normalization parameters. Among them, the at least three different calibration positions include: three non-collinear limit reference positions of the device.

15. The alignment method according to claim 1, characterized in that, When the variable space is a variable three-dimensional space stroke, the alignment method includes: Adjusting the device to at least four different calibration positions within the variable three-dimensional space stroke, repeating the virtual-real opening and closing calibration method, and determining the calibration correspondence relationship of the normalization parameters. Among them, the at least four different calibration positions include: four non-coplanar limit reference positions of the device.

16. The alignment method according to claim 1, characterized in that, Adjusting the device whose position has changed to at least two different calibration positions in the variable space, repeating the virtual-real opening and closing calibration method, and determining the calibration correspondence relationship of the normalization parameters includes: Adjusting the device whose position has changed to at least two different calibration positions in the variable space, and repeating the virtual-real opening and closing calibration method at the different calibration positions to obtain a set of calibration parameters. Among them, each calibration parameter in the set of calibration parameters corresponds one-to-one with each calibration position; Determining a set of normalization parameters according to all the calibration positions and the variable space; Fitting and determining the calibration correspondence relationship of the normalization parameters by combining the set of calibration parameters and the set of normalization parameters.

17. The alignment method according to claim 1, characterized in that, The alignment method further includes: adjusting the position of the human eye simulation auxiliary camera by mounting the human eye simulation auxiliary camera with a robotic arm, and providing corresponding three-dimensional normalized coordinates.

18. The alignment method according to claim 1, wherein The alignment method further includes: Projecting the physical identification point into the virtual image by combining the two-dimensional image mapping relationship and the attitude of the target vehicle.

19. An alignment device for a display device, characterized in that, Including: A first determination unit for determining the line-of-sight focusing area of the user in the target vehicle, where at least one physical reference object is included in the line-of-sight focusing area; A first acquisition unit for acquiring a captured image of the original scene by a first camera and extracting the physical identification points in the captured image; A first determination unit, configured to determine a two-dimensional image mapping relationship between the captured image and a virtual image in the current field of view of the user by using pre-obtained calibration parameters, where the virtual image is an image to be projected by a display device, and the calibration parameters are determined by a virtual-reality alignment calibration method by combining a human-eye simulation auxiliary camera according to the position states of the display device and the user's eyes; An alignment unit, configured to project the physical identification points into the virtual image according to the two-dimensional image mapping relationship, where a virtual identifier in the virtual image is aligned with a physical reference object in the line-of-sight focusing area; The alignment device further includes: a third determination unit, configured to, when the positions of the display device and the user's eyes are kept at fixed positions, determine the calibration parameters by using a virtual-reality opening / closing calibration method by combining the human-eye simulation auxiliary camera at the fixed positions; or a fourth determination unit, configured to, when the positions of the display device and / or the user's eyes change, repeat the virtual-reality opening / closing calibration method by adjusting the device whose position has changed to multiple different calibration positions and determine a normalized parameter calibration correspondence, and determine the calibration parameters according to the actual position of the device and the normalized parameter calibration correspondence; When the positions of the display device and / or the user's eyes change, repeating the virtual-reality opening / closing calibration method by adjusting the device whose position has changed to multiple different calibration positions and determining a normalized parameter calibration correspondence, and determining the calibration parameters according to the actual position of the device and the normalized parameter calibration correspondence includes: adjusting the device whose position has changed to at least two different calibration positions in a variable space and repeating the virtual-reality opening / closing calibration method to determine the normalized parameter calibration correspondence; determining a normalized position parameter on the variable space based on the actual position of the device, and determining the calibration parameters according to the normalized position parameter and the normalized parameter calibration correspondence.

20. The alignment device according to claim 19, wherein The first determination unit includes: A first determination module, configured to determine a focused line of sight of the user in front of the target vehicle based on the coordinates of the line-of-sight projection object of the user; A fixation point determination module, configured to perform an intersection process on the focused line of sight and a plane where the ground in front of the target vehicle is located to obtain a fixation position point of the user; A second determination module, configured to determine a line-of-sight focusing area of the user in the target vehicle based on the fixation position point; 21. The alignment device according to claim 20, characterized in that, The actual positions of the display device and the user's eyes are obtained by a driver monitoring system.

22. The alignment device according to claim 20, characterized in that, The calibration parameters include: a first two-dimensional image mapping relationship between a first image captured by a first camera and a second image captured by the human-eye simulation auxiliary camera, and a second two-dimensional image mapping relationship between a third image captured by the human-eye simulation auxiliary camera and a virtual calibration pattern.

23. The alignment device according to claim 20, wherein The third determination unit includes: a third determination module, configured to determine the calibration parameters by controlling the turning on and off of the display device, combining a physical calibration pattern and a virtual calibration pattern, and according to images captured by the human-eye simulation auxiliary camera and the first camera.

24. The alignment device according to claim 23, characterized in that, The third determination module includes: A first determination sub-module, configured to determine a third two-dimensional image mapping relationship between a first image collected by the first camera and a second image collected by the human eye simulation auxiliary camera in combination with the physical calibration pattern after the display device is turned off; A second determination sub-module, configured to determine a fourth two-dimensional image mapping relationship between a third image collected by the human eye simulation auxiliary camera and the virtual calibration pattern in combination with the virtual calibration pattern after the display device is turned on; A third determination sub-module, configured to represent the third two-dimensional image mapping relationship and the fourth two-dimensional image mapping relationship as the calibration parameters.

25. The alignment device according to claim 24, wherein The alignment device further includes: a substitution unit, configured to substitute the physical calibration pattern with an equivalent physical calibration pattern that is higher than and parallel to the physical calibration pattern in the field of view of the human eye simulation auxiliary camera.

26. The alignment device according to claim 25, characterized in that The third determination module further includes: A fourth determination sub-module, configured to, after the display device is turned off, restore a virtual calibration pattern equivalent to the ground through the equivalent physical calibration pattern and project it onto a fourth image generated by the first camera, and a fifth image of the equivalent physical calibration pattern collected by the human eye simulation auxiliary camera, and determine a fifth two-dimensional image mapping relationship between the fourth image and the fifth image; A fifth determination sub-module, configured to determine a sixth two-dimensional image mapping relationship between a sixth image collected by the human eye simulation auxiliary camera and the virtual calibration pattern in combination with the virtual calibration pattern after the display device is turned on; A sixth determination sub-module, configured to represent the fifth two-dimensional image mapping relationship and the sixth two-dimensional image mapping relationship as the calibration parameters.

27. A vehicle-mounted display system, characterized in that, Comprising: A driver monitoring system, configured to track a line-of-sight focusing area of a user in a target vehicle, where at least one physical reference object is included in the line-of-sight focusing area; A driving recorder, configured to capture the line-of-sight focusing area to obtain a captured image; A vehicle-mounted controller, connected to the driver monitoring system and the driving recorder respectively, and executing the alignment method for a display device according to any one of claims 1 to 18; A head-up display, configured to project a virtual image to a preset position directly in front of the user, where a virtual identifier in the virtual image is aligned with a physical reference object in the line-of-sight focusing area.

28. A vehicle-mounted control device, characterized in that, Comprising: A processor; And A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the alignment method for a display device according to any one of claims 1 to 18 by executing the executable instructions.

29. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the alignment method for a display device according to any one of claims 1 to 18.

Images