A data processing method and apparatus thereof

Abstract

Embodiments of the present application disclose a data processing method for a human-computer interaction system. The method comprises the following steps: a first device receives first position information sent by a second device, the first position information comprising position information of a first feature in a preset coordinate system, the first feature representing feature information of a user, the first device obtains a first pre-distortion model according to the first position information, and the first device corrects a projection image according to the first pre-distortion model, the projection image being an image projected by the first device. In the embodiments of the present application, the first device obtains the first pre-distortion model according to the first position information sent by the second device and comprising the feature information of the user, so that the first device can correct the projection virtual image according to the first pre-distortion model obtained according to the feature information of the user in real time, and the experience of the user is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a data processing method and device. Background Technology

[0002] Augmented Reality-Head-Up Display (AR-HUD) uses an optical projection system to project driver assistance information (digits, images, animations, etc.) onto the windshield of a car to form a virtual image. The driver can observe the corresponding driver assistance information through the display area on the windshield.

[0003] Because windshields have varying curvatures, the virtual image projected onto them by an optical projection system is often distorted. To eliminate this distortion, during the preparation phase before the official use of AR-HUD, an eye simulation device is placed near the driver's head. This device simulates the driver's eye position and captures a calibration image projected onto the AR-HUD display system in the driving area. By analyzing the position of a reference point on the calibration image, the distortion of the image is calculated, and image correction is performed based on this distortion.

[0004] Since the human eye simulation device is essentially fixed, the reference points in the captured calibration images are also essentially fixed. However, during actual use of AR-HUD, the driver's position frequently changes, such as with driver changes or seat adjustments. When the driver's position changes, the position of the driver's eyes differs from the position of the human eye simulation device during the preparation phase. Therefore, the distorted projected image seen by the driver may still be distorted, resulting in a poor projection image quality. Summary of the Invention

[0005] This application provides a data processing method and apparatus that can be applied to human-computer interaction systems, such as in-vehicle human-computer interaction systems. The method provided in this application is used to correct distorted projection images in real time, thereby improving the user experience.

[0006] The first aspect of this application provides a data processing method.

[0007] During the use of the human-computer interaction system, the first device receives the first location information sent by the second device. The first location information includes the position information of the first feature in the preset coordinate system. The first feature represents the feature information of the user collected by the second device.

[0008] The first device obtains a first pre-distortion model based on the first location information sent by the second device. The first device then corrects the projected image based on the first pre-distortion model; this projected image is the image projected by the first device.

[0009] In this embodiment, the first device obtains a first pre-distortion model based on the first location information including user feature information sent by the second device, so that the first device can correct the projected virtual image in real time based on the first pre-distortion model obtained from the user feature information, thereby improving the user experience.

[0010] Optionally, in one possible implementation, after receiving the first location information, the first device obtains second location information based on the first location information. The second location information is a location among multiple preset location information values ​​whose distance from the first location information in a preset coordinate system is less than a preset threshold. This preset location information is pre-set by the first device. After obtaining the second location information, the first device then obtains the first pre-distortion model corresponding to the second location information.

[0011] In this embodiment, the first device obtains the corresponding first pre-distortion model based on the preset location information, which saves the resources consumed by online calculation of the first pre-distortion model and improves the execution efficiency of the human-computer interaction system during the use phase.

[0012] Optionally, in one possible implementation, before receiving the first location information sent by the second device, the first device receives at least two first image information sent by the third device, which represent information about the images projected by the first device collected by the third device at different positions in a preset coordinate system.

[0013] The first device acquires standard image information, which represents a projected image without distortion. The first device compares at least two pieces of first image information with the standard image information to obtain at least two preset distortion variables, which represent the distortion of the first image information relative to the standard image information.

[0014] The first device calculates at least two preset distortion variables to obtain at least two first pre-distortion models, which correspond one-to-one with the first image information.

[0015] In this embodiment, information from at least two projected images collected by a third device at different locations and a standard image are used to calculate the corresponding preset distortion variables. Then, at least two first pre-distortion models are obtained through the corresponding preset distortion variables. In later use, the projected images viewed by the user at different locations can be calibrated, thereby improving the user experience.

[0016] Optionally, in one possible implementation, the first device receives gaze information sent by the second device, the gaze information representing information about a user's gaze reference point, which is located in the image projected by the first device. The first device determines a first field of view based on the gaze information, the first field of view representing the field of view that the user can observe.

[0017] The first device determines a first distortion variable based on gaze information and first position information. The first distortion variable represents the distortion of the human eye calibration image relative to the standard image. The human eye calibration image represents the image presented by the projection image of the first device in the user's human eye, and the standard image is a projection image without distortion.

[0018] The first device obtains the first pre-distortion model based on the determined first field of view and the first distortion variable.

[0019] In this embodiment, the projected image is calibrated in real time based on the user's gaze information collected in real time, so that the user can view the complete projected image from different positions, thus improving the user experience.

[0020] Alternatively, in one possible implementation, the feature information includes the user's eye information.

[0021] In this embodiment of the application, the feasibility of the technical solution is improved when the feature information includes human eye information.

[0022] Optionally, in one possible implementation, during the specific process of the first device correcting the projected image according to the first pre-distortion model, the first device performs image processing through one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA) to correct the projected image according to the first pre-distortion model.

[0023] In this embodiment of the application, when the first device performs image processing through one or more of CPU, GPU and FPGA, and then corrects the projected image, the feasibility of the solution is improved.

[0024] Optionally, in one possible implementation, during the process of the first device correcting the projected image according to the first pre-distortion model, the first device modulates light using one or more of liquid crystal on silicon (LCOS), digital light processing (DLP), and liquid crystal display (LCD) to correct the projected image according to the first pre-distortion model.

[0025] In this embodiment, the first device modulates light using one or more of LCOS, DLP, and LCD to correct the projected image, thereby improving the feasibility of the solution.

[0026] The second aspect of this application provides a data processing method.

[0027] During the use of the human-computer interaction system, the second device acquires first location information, which includes the position information of a first feature in a preset coordinate system. The first feature represents the user's feature information acquired by the second device. The first location information is used by the first device to correct the projected image, which is the image projected by the first device. The second device sends the first location information to the first device.

[0028] In this embodiment, the second device sends first location information, including user feature information, to the first device so that the first device can correct the image projected by the first device in real time based on the first location information, thereby improving the user experience.

[0029] Optionally, in one possible implementation, the second device acquires second image information, which includes the user's feature information, and the second device performs calculations based on the second image information to obtain first location information.

[0030] In this embodiment, the second device obtains the first location information by collecting image information including the user's feature information and performing calculations, thereby improving the feasibility of the solution.

[0031] Optionally, in one possible implementation, during the calculation process of the second device based on the first image information, the second device performs calculations using a feature recognition algorithm to obtain feature location information of the feature information in the second image information. The second device then performs further calculations using the feature location information to obtain the first location information.

[0032] In this embodiment, the second device calculates feature location information based on a feature recognition algorithm, and then obtains first location information based on the feature location information, thereby improving the feasibility of the solution.

[0033] Optionally, in one possible implementation, before the second device calculates the first position information using the feature position information, the second device also acquires depth information, which represents the straight-line distance from the feature information to the second device. In another implementation where the second device calculates the first position information using the feature position information, the second device calculates the first position information using both the feature position information and the depth information.

[0034] In this embodiment, the second device calculates the first location information by collecting the depth information and feature location information, thereby improving the accuracy of calculating the first location information.

[0035] Alternatively, in one possible implementation, the feature information includes the user's eye information.

[0036] In this embodiment of the application, the feasibility of the technical solution is improved when the feature information includes human eye information.

[0037] Optionally, in one possible implementation, after acquiring the first position information, the second device also acquires the user's gaze information, which represents the information of the user's gaze reference point, which is located in the image projected by the first device. The gaze information is used to determine the first distortion variable, which is used to determine the first pre-distortion model, and the first pre-distortion model is used to correct the projected image projected by the first device.

[0038] After obtaining the first location information and gaze information, the second device sends the first location information and gaze information to the first device.

[0039] In this embodiment, the projected image is calibrated in real time based on the user's gaze information collected in real time, so that the user can view the complete projected image from different positions, thus improving the user experience.

[0040] A third aspect of the embodiments of this application provides a display device.

[0041] Display devices include:

[0042] The receiving unit is used to receive first location information sent by the second device. The first location information includes the location information of a first feature in a preset coordinate system. The first feature represents the user's feature information.

[0043] The processing unit is used to obtain the first pre-distortion model based on the first position information;

[0044] The correction unit is used to correct the projected image according to the first pre-distortion model, wherein the projected image is the image projected by the first device.

[0045] Alternatively, in one possible implementation, the display device further includes:

[0046] The acquisition unit is used to acquire second position information based on first position information. The second position information is a position information among a plurality of preset position information whose distance from the first position information in a preset coordinate system is less than a preset threshold. The preset position information is preset by the first device.

[0047] The acquisition unit is also used to acquire the first pre-distortion model corresponding to the second position information.

[0048] Optionally, in one possible implementation, the receiving unit is further configured to receive at least two first image information sent by the third device, wherein the at least two first image information represent information of images projected by the first device acquired by the third device at different positions in a preset coordinate system;

[0049] The acquisition unit is also used to acquire standard image information, which represents a projection image without distortion;

[0050] The processing unit is also configured to compare at least two pieces of first image information with standard image information respectively to obtain at least two preset distortion variables, wherein the preset distortion variables represent the distortion variables of the first image information relative to the standard image information;

[0051] The processing unit is also used to calculate at least two first pre-distortion models based on at least two preset distortion variables, and the at least two first pre-distortion models correspond one-to-one with the first image information.

[0052] Optionally, in one possible implementation, the receiving unit is further configured to receive gaze information sent by the second device, the gaze information representing information about the user's gaze reference point, the reference point being calibrated in the image projected by the first device;

[0053] Display devices also include:

[0054] The determining unit is used to determine the first field of view range based on the gaze information, wherein the first field of view range represents the field of view range observed by the user;

[0055] The determining unit is also used to determine a first distortion variable based on the gaze information and the first position information. The first distortion variable represents the distortion of the human eye calibration image relative to the standard image. The human eye calibration image represents the image presented by the projection image of the first device in the user's human eye. The standard image is a projection image without distortion.

[0056] The processing unit is also used to obtain a first pre-distortion model based on the first field of view and the first distortion variable.

[0057] Optionally, in one possible implementation, the user's feature information includes the user's eye information.

[0058] Alternatively, in one possible implementation, the correction unit is specifically used to perform image processing based on a first pre-distortion model, using one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a field-programmable gate array (FPGA) to correct the projected image.

[0059] Alternatively, in one possible implementation, the correction unit is specifically used to correct the projected image by performing light modulation using one or more of liquid crystal on silicon (LCOS), digital light processing (DLP), and liquid crystal display (LCD) according to a first pre-distortion model.

[0060] The fourth aspect of this application provides a feature acquisition device.

[0061] Feature acquisition devices include:

[0062] The acquisition unit is used to acquire first location information, which includes the location information of a first feature in a preset coordinate system. The first feature represents the user's feature information. The first location information is used by the first device to correct the projected image. The projected image is an image projected by the first device.

[0063] The transmitting unit is used to transmit first location information to the first device.

[0064] Alternatively, in one possible implementation, the feature acquisition device further includes:

[0065] The acquisition unit is used to acquire second image information, which includes the user's feature information.

[0066] The processing unit is used to calculate and obtain the first position information based on the second image information.

[0067] Optionally, in one possible implementation, the processing unit is specifically used to calculate the feature location information in the second image information by means of a feature recognition algorithm;

[0068] The processing unit is specifically used to calculate the first position information using the feature position information.

[0069] Optionally, in one possible implementation, the acquisition unit is also used to acquire depth information, which represents the straight-line distance from the feature information to the second device;

[0070] The processing unit is also used to calculate the first location information using the feature location information, including:

[0071] The processing unit is also used to calculate the first position information using feature position information and depth information.

[0072] Alternatively, in one possible implementation, the feature information includes the user's eye information.

[0073] Optionally, in one possible implementation, the acquisition unit is further configured to acquire the user's gaze information, which represents information about the user's gaze reference point, the reference point being calibrated in the image projected by the first device, the gaze information being used to determine a first distortion variable, the first distortion variable being used to determine a first pre-distortion model, and the first pre-distortion model being used to correct the projected image.

[0074] The transmitting unit is also used to transmit first location information and gaze information to the first device.

[0075] The fifth aspect of this application provides a human-computer interaction system.

[0076] Human-computer interaction systems include:

[0077] A display device for performing the method as described in the first aspect of the embodiments of this application.

[0078] A feature acquisition device is used to perform the method as described in the second aspect of the embodiments of this application.

[0079] A sixth aspect of this application provides a display device.

[0080] The display device includes:

[0081] Processor, memory, input / output devices;

[0082] The processor is connected to memory and input / output devices;

[0083] The processor executes the method described in the first aspect of this application.

[0084] The seventh aspect of this application provides a feature acquisition device.

[0085] The feature acquisition device includes:

[0086] Processor, memory, input / output devices;

[0087] The processor is connected to memory and input / output devices;

[0088] The processor executes the method described in the first aspect of this application.

[0089] An eighth aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the embodiments of the first and / or second aspects of this application.

[0090] A ninth aspect of this application provides a computer program product that, when executed on a computer, causes the computer to perform the method described in the embodiments of the first and / or second aspects of this application.

[0091] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0092] In this embodiment, the first device obtains a first pre-distortion model based on the user's feature information and position information in a preset coordinate system. This allows the first device to adjust the pre-distortion model in real time, thereby using the first pre-distortion model to correct the image projected by the first device and improve the quality of the projected image seen by the user. Attached Figure Description

[0093] Figure 1 A schematic diagram of the human-computer interaction system provided in this application;

[0094] Figure 2 Another schematic diagram of the human-computer interaction system provided in this application;

[0095] Figure 3 A flowchart illustrating the data processing method provided in this application;

[0096] Figure 4 Another flowchart illustrating the data processing method provided in this application;

[0097] Figure 5 A schematic diagram of a scenario for the data processing method provided in this application;

[0098] Figure 6 Another scenario illustration of the data processing method provided in this application;

[0099] Figure 7 Another scenario illustration of the data processing method provided in this application;

[0100] Figure 8 A schematic diagram of the structure of the display device provided in this application;

[0101] Figure 9 Another structural schematic diagram of the display device provided in this application;

[0102] Figure 10 A schematic diagram of the feature acquisition device provided in this application;

[0103] Figure 11 Another structural schematic diagram of the feature acquisition device provided in this application;

[0104] Figure 12 Another structural schematic diagram of the display device provided in this application;

[0105] Figure 13 Another structural schematic diagram of the feature acquisition device provided in this application. Detailed Implementation

[0106] This application provides a data processing method and apparatus for obtaining a first pre-distortion model in a driving system based on the user's feature information and position information in a preset coordinate system. This allows a first device to adjust the pre-distortion model in real time based on the user's feature information, thereby correcting the image projected by the first device through the first pre-distortion model, improving the quality of the projected image seen by the user, and thus enhancing the user's experience.

[0107] Please see Figure 1 This is a schematic diagram of the human-computer interaction system provided in this application.

[0108] This application provides a human-computer interaction system, which includes a display device, a feature acquisition device, and a windshield of a vehicle. The feature acquisition device and the display device can be connected via wired or wireless means, without limitation here. If the feature acquisition device and the display device are connected via wired means, they can be connected via a data cable, such as a COM interface data cable, a USB interface data cable, a Type-C interface data cable, or a Micro-USB interface data cable. It is understood that other methods can also be used for wired connection, such as fiber optic connection, without limitation here. If the feature acquisition device and the display device are connected wirelessly, they can be connected via Wi-Fi, Bluetooth, infrared, or other wireless methods. It is understood that other methods can also be used for wireless connection, such as third-generation (3G), fourth-generation (4G), and fifth-generation (5G) access technologies, without limitation here.

[0109] Specifically, the display device can be a head-up display (HUD), an augmented reality head-up display (AR-HUD), or a display device with projection imaging capabilities; no specific limitation is made here.

[0110] Specifically, the feature acquisition device can be a camera, a standalone camera, or a camera with processing capabilities, such as an eye-tracking device; no specific limitation is made here.

[0111] Optionally, the display device may also include a computing processing unit for processing information sent by other devices, such as image information. This computing processing unit may be integrated into the display device or may be a separate processing device from the display device; no specific limitation is made here.

[0112] In this human-computer interaction system, a display device projects the desired image onto the windshield of a car. Specifically, the display device may also include an optical system for projecting the image onto the windshield. A feature acquisition device acquires the user's feature information and transmits it to a computing unit. The computing unit performs relevant calculations and feeds the results back to the display device. Specifically, this feature information may be human eye information. The display device then adjusts the projection system to suit the user's viewing position, ensuring that the user can view the complete projected virtual image from different locations.

[0113] In this application embodiment, depending on different implementation methods, more application scenarios may be included, such as... Figure 2 The diagram shown is another schematic of the human-computer interaction system provided in this application.

[0114] This application also provides a human-computer interaction system, which includes a display device, a camera, and a car's windshield. The camera and display device can be connected via wired or wireless means, which is not limited here. The connection method between the camera and display device is... Figure 1 The connection method between the feature acquisition device and the display device in the human-computer interaction system shown is similar, and will not be described in detail here.

[0115] Specifically, the display device can be a head-up display (HUD), an augmented reality head-up display (AR-HUD), or a display device with projection imaging capabilities; no specific limitation is made here.

[0116] Specifically, the shooting device can be a camera, a standalone webcam, or a video camera with processing capabilities, such as a human eye simulation device; there are no specific limitations here.

[0117] Optionally, the display device may also include a computing processing unit for processing information sent by other devices, such as image information. This computing processing unit may be integrated into the display device or may be a separate processing device from the display device; no specific limitation is made here.

[0118] The imaging device is used to capture the projected image from a simulated human eye perspective within a specific field of view, which is the space inside the vehicle where the projected virtual image can be partially or fully observed.

[0119] like Figure 2 The scenario shown is a possible implementation of a human-computer interaction system, specifically a preparation phase before the system is put into use. In this scenario, a camera captures the projected virtual image from various angles within a specific field of view. The captured images are then transmitted to a computing unit, which performs relevant calculations and feeds the results back to the display device. The display device then sets different pre-distortion models based on information from different camera positions. During the implementation phase of the human-computer interaction system, the corresponding pre-distortion models are obtained and the projected virtual image is adjusted according to the user's viewing position, allowing the user to view the complete projected virtual image from different locations.

[0120] To facilitate understanding of the embodiments of this application, the terms used in the embodiments of this application are explained below:

[0121] Eye Box Range: In AR-HUD display technology, when the driver's eyes are within the eye box range, they can see the complete projected virtual image of the AR-HUD. When the driver's eyes are outside the designed eye box range, the driver will only see a portion of the projected virtual image or will not see the projected virtual image at all.

[0122] The following is combined Figure 1 and Figure 2 The human-computer interaction system shown herein describes the data processing method in the embodiments of this application.

[0123] In this embodiment, after the eye-tracking device acquires human eye feature information and sends it to the AR-HUD display system, the AR-HUD display system can correct the projected virtual image using a pre-set pre-distortion model, or it can acquire human eye gaze information using the eye-tracking device, obtain a pre-distortion model using the human eye gaze information and human eye feature information, and then correct the projected virtual image using the pre-distortion model. The two different implementation methods are described below.

[0124] 1. Correct the projected virtual image using a pre-set pre-distortion model.

[0125] Please see Figure 3 This is a flowchart illustrating a data processing method according to an embodiment of this application.

[0126] In this embodiment, the AR-HUD display system represents the first device, the human eye tracking device represents the second device, and the human eye simulation device represents the third device, which will be used as an example for explanation.

[0127] In step 301, the human eye simulation device sends at least two first image information to the AR-HUD display system.

[0128] Before a human-computer interaction system is put into use, it will be pre-configured or trained. During the pre-configuration or training phase, the human eye simulation device will collect information of the image projected by the AR-HUD at different positions in a preset coordinate system, i.e., collect the first image information. After collecting at least two pieces of the first image information, the human eye simulation device will send these at least two pieces of the first image information to the AR-HUD display system.

[0129] Specifically, the AR-HUD display system first determines the available field of view of the AR-HUD display system, and divides the available field of view into several small areas, and records the position information of the center point of several small areas in the preset coordinate system. The position information of the center point of several small areas in the preset coordinate system represents the preset position information, which is preset by the AR-HUD display system.

[0130] In one possible implementation, such as Figure 5 As shown, when the preset coordinate system is the camera coordinate system with the human eye tracking device as the origin, the AR-HUD display system records the position coordinates of the center points of several small areas in the camera coordinate system.

[0131] In one possible implementation, such as Figure 6 As shown, when the preset coordinate system is the world coordinate system with the AR-HUD display system as the origin, the AR-HUD display system records the position coordinates of the center points of several small areas in the world coordinate system.

[0132] After the AR-HUD system records the position information of the center points of several small areas in a preset coordinate system, the human eye simulation device is installed or placed at the spatial points corresponding to each position information to capture the projected virtual image. It should be noted that there are many ways to capture this image, such as by taking a picture or by photography; no specific method is limited here. For example, the human eye simulation device can be placed at the spatial point (12,31,22) in the camera coordinate system to capture the projected virtual image of the AR-HUD display system.

[0133] In one possible implementation, before acquiring the projected virtual image through a human eye simulation device, the projected virtual image can be calibrated through an AR-HUD display system, for example, by calibrating it using a chessboard format or by calibrating it using a dot matrix pattern; the specific method is not limited here.

[0134] Calibrating the projected virtual image allows for the calculation of corresponding distortion variables in later stages using the calibrated points, which improves the accuracy of distortion calculation compared to calculating on an uncalibrated image.

[0135] In step 302, the AR-HUD display system acquires standard image information.

[0136] After receiving at least two first image messages from the human eye simulation device, the AR-HUD display system acquires standard image messages from its local storage, which represent undistorted projected images.

[0137] Optionally, in one possible implementation, the acquired standard image is a calibrated standard image. The specific calibration method can be a checkerboard format or a bitmap format. The specific method is not limited here. Preferably, the calibration method of the standard image can be the same as the calibration method of the received at least two first image information.

[0138] In step 303, the AR-HUD display system compares at least two first image pieces with a standard image to obtain at least two preset distortion variables.

[0139] After acquiring a standard image, the AR-HUD display system compares at least two received first image information with the standard image to obtain at least two preset distortion variables, which represent the distortion of the first image information relative to the standard image information.

[0140] Specifically, in one possible implementation, after the first image information and the standard image are calibrated, the AR-HUD display system calculates the transformation formula between the calibration points of the standard image and the calibration points in the first image information. For example, if the standard image is calibrated with a 100*100 dot matrix and the first image information has an 80*80 dot matrix, then the system calculates the transformation formula from the 80*80 dot matrix to the 100*100 dot matrix to obtain the preset distortion variable.

[0141] In practical applications, since the distortion of the first image information relative to the standard image information is quite complex, a corresponding calculation method can be designed. The specific calculation method is not limited here.

[0142] In step 304, the AR-HUD display system calculates at least two first pre-distortion models based on at least two preset distortion variables.

[0143] After obtaining at least two preset distortion variables, the AR-HUD display system performs calculations based on the at least two preset distortion variables to obtain at least two first pre-distortion models, and the at least two pre-distortion models correspond one-to-one with the first image information.

[0144] Specifically, in one possible implementation, the AR-HUD display system can calculate a transformation mathematical model corresponding to the standard image using a standard image and pre-set distortion variables; this transformation mathematical model is the first pre-distortion model. In practical applications, the AR-HUD display system can adjust the standard image according to this transformation mathematical model and project the adjusted image, so that when the user views the projected image based on their position information in the first image information corresponding to the transformation mathematical model, they can see the complete standard image.

[0145] Specifically, in one possible implementation, the AR-HUD display system can also calculate modified projection parameters using a pre-set distortion variable and the AR-HUD display system's projection parameters; these modified projection parameters constitute the first pre-distortion model. In practical applications, the AR-HUD display system can project a standard image based on the modified projection parameters. Because the projection parameters are modified, the standard image changes accordingly. Since these projection parameters are obtained from the pre-set distortion variable, when a user views the projected image from their position within the first image information corresponding to these projection parameters, they will see the complete standard image.

[0146] Specifically, after obtaining multiple first predistortion models, a correspondence can be established between each first predistortion model and the corresponding position information in the first image information, and this correspondence can be stored locally in the AR-HUD display system.

[0147] In step 305, the eye-tracking device acquires the second image information.

[0148] During the deployment phase of the human-computer interaction system, when a user enters the vehicle, the eye-tracking device collects second image information, which includes the user's feature information.

[0149] Specifically, in one possible implementation, the user's feature information includes eye information. When the user enters the vehicle, the eye-tracking device takes a photo or video to collect a second image of the user, which includes the user's eye information. When the eye-tracking device collects data via video recording, the user's image information is determined by extracting frames from the video recording after acquisition.

[0150] It is understandable that this feature information may include more information, such as facial information, nose information, mouth information, etc., but no specific limit is made here.

[0151] In step 306, the eye-tracking device calculates the feature location information in the second image information using a feature recognition algorithm.

[0152] After the human eye tracking device acquires the second image information, the human eye simulation device calculates the feature location information of the feature information in the second image information through a feature recognition algorithm. This feature location information represents the position information of the feature information in the second image information.

[0153] Specifically, in one possible implementation, the eye-tracking device uses an eye recognition algorithm to identify the user's eye information and its position in the second image information, thereby obtaining the position of the eye in the image coordinate system, such as... Figure 7 As shown, this image coordinate system represents a two-dimensional coordinate system with the image center as the origin. For example, the location of the user's eye information in the second image information can be identified using the Huffman circle detection method. Alternatively, the location of the user's eye information in the second image information can be identified using a convolutional neural network; the specific method is not limited here.

[0154] In step 307, the eye-tracking device collects depth information.

[0155] The eye-tracking device is also used to collect depth information, which represents the straight-line distance from the user's feature information to the eye-tracking device.

[0156] Specifically, in one possible implementation, the eye-tracking device obtains the straight-line distance from the user's eye information to the eye-tracking device through a ranging function. For example, the eye-tracking device obtains the straight-line distance from the user's eye information to the eye-tracking device through infrared ranging. It is understood that this depth information can also be obtained through other methods, such as ultrasonic ranging, but this is not limited here.

[0157] In step 308, the human eye tracking device calculates the first position information using feature position information and depth information.

[0158] After acquiring depth information, the eye-tracking device calculates using feature location information and depth information to obtain first location information, which represents the location of the user's feature information in a preset coordinate system.

[0159] Specifically, in one possible implementation, when the preset coordinate system is the camera coordinate system, the eye-tracking device calculates the first position information using feature position information, depth information, and the intrinsic parameters of the eye-tracking device. For example, it can be calculated using the following formula:

[0160] z c =ds

[0161] x c =Z(uC) u ) / f u

[0162] y c =Z(vC) v ) / f v

[0163] Among them, z c This represents the Z-axis value of the user's feature information in the camera coordinate system. c This represents the value of the user's feature information on the X-axis in the camera coordinate system, and the value of the Y-axis. c The value of the Y-axis in the camera coordinate system represents the user's feature information's position, d represents depth information, s represents the scaling factor in the intrinsic parameters of the eye-tracking device, and f represents the depth information. u f represents the focal length in the horizontal direction within the intrinsic parameters of an eye-tracking device. v The vertical focal length represents the intrinsic parameters of the eye-tracking device; u represents the value corresponding to the X-axis in the image coordinate system of the feature position information; v represents the value corresponding to the Y-axis in the image coordinate system of the feature position information; C u and C v This represents the X-axis and Y-axis values ​​corresponding to the origin coordinates in the image coordinate system.

[0164] It should be noted that when the preset coordinate system is the camera coordinate system, the first position information is equal to the position information of the user's feature information in the camera coordinate system.

[0165] It is understandable that in practical applications, the user's feature information in the camera coordinate system can also be obtained through other formulas, but this is not limited here.

[0166] When the preset coordinate system is the world coordinate system with the AR-HUD display system as the origin, the first position information represents the position information of the user's feature information in the world coordinate system. Then, the human eye tracking device calculates the first position information based on the position information of the user's feature information in the camera coordinate system.

[0167] Specifically, in one possible implementation, the eye-tracking device can calculate the first position information in the following way:

[0168]

[0169] R = R Z *R Y *R X ,T=(t x ,t y ,t z ) T

[0170]

[0171] Where ω, δ, and θ are rotation parameters (ω, δ, θ), t x t y and t z The translation parameters (t) of the three axes x , t y , t z ), x w The x-axis value and y-axis value represent the user's feature information in the world coordinate system. w The z-axis value of the user's feature information in the world coordinate system is the position information of the user's feature information. w The z-axis value represents the user's feature information in the world coordinate system. c This represents the Z-axis value of the user's feature information in the camera coordinate system. c This represents the value of the user's feature information on the X-axis in the camera coordinate system, and the value of the Y-axis. c This represents the Y-axis value corresponding to the user's feature information in the camera coordinate system.

[0172] It is understandable that the user's feature information in the world coordinate system can also be calculated using other formulas, but specific formulas are not limited here.

[0173] In step 309, the eye-tracking device sends the first location information to the AR-HUD display system.

[0174] After obtaining the first location information, the eye-tracking device sends the first location information to the AR-HUD display system.

[0175] In step 310, the AR-HUD display system obtains the second location information based on the first location information.

[0176] After receiving the AR-HUD display system sent by the human eye tracking device, the AR-HUD display system obtains the second position information based on the first position information. The second position information represents the position information among multiple preset position information whose distance from the first position information in the preset coordinate system is less than a preset threshold.

[0177] Specifically, in one possible implementation, the AR-HUD display system calculates the preset position information that has the smallest distance from the first position information in a preset coordinate system based on the first position information and each of a plurality of preset position information. For example, this can be calculated using the following formula:

[0178]

[0179] Where j represents the index number corresponding to the value with the smallest distance between the preset location information and the first location information, and x i This indicates the value of the X-axis in the preset position information, and the value of the Y-axis. i This represents the value of the Y-axis in the preset position information, z. i This represents the Z-axis value in the preset position information, x w The x-axis value and y-axis value represent the user's feature information in the world coordinate system. w The z-axis value of the user's feature information in the world coordinate system is the position information of the user's feature information. w This represents the Z-axis value of the user's feature information in the world coordinate system.

[0180] It is understandable that the distance between the preset location information and the first location information can also be calculated using other formulas. For example, when the first location information is the location information of the user's feature information in the camera coordinate system, then the value of the location information in the corresponding camera coordinate system (x) can be used. c y c , z c Replace (x) in the above formula w y w , z w The specific calculation formula is not specified here.

[0181] The distance between each preset location information and the first location information is obtained by the above method. Then, the preset location information whose distance from the first location information is less than a preset range is selected as the second location information. Preferably, the preset location information with the smallest distance from the first location information can be selected as the second location information.

[0182] In step 311, the AR-HUD display system acquires the first pre-distortion model corresponding to the second position information.

[0183] After obtaining the second location information, the AR-HUD display system searches for the first pre-distortion model corresponding to the second location information in the local storage.

[0184] In step 312, the AR-HUD display system corrects the projected image according to the first pre-distortion model.

[0185] After obtaining the first predistortion model, the AR-HUD display system corrects the image projected by the first device based on the first predistortion model.

[0186] Specifically, in one possible implementation, when the first pre-distortion model represents the transformation mathematical model of the standard image, the AR-HUD display system adjusts the standard image according to the transformation mathematical model and projects the adjusted image, so that when the user's eye views the projected image at the preset position information corresponding to the transformation mathematical model, it can see the complete standard image.

[0187] Specifically, in one possible implementation, the AR-HUD display system can process a standard image using one or more of the following: CPU, GPU, and FPGA, based on a transformation mathematical model, to obtain an adjusted image. This allows the user's eye to see the complete standard image when viewing the adjusted projected image at a preset position corresponding to the transformation mathematical model. It is understood that other methods can also be used to process the standard image to achieve the same adjustment, but this is not limited here.

[0188] Specifically, in one possible implementation, when the first pre-distortion model represents the modified projection parameters, the AR-HUD display system projects a standard image according to the modified projection parameters. Since the projection parameters are modified, the standard image changes according to the changes in the projection parameters. Since the projection parameters are obtained based on the preset distortion parameters, the user's eye can see the complete standard image when viewing the projected image at the preset position information corresponding to the projection parameters.

[0189] Specifically, in one possible implementation, the AR-HUD display system can modulate light using one or more of the following technologies: liquid crystal on silicon (LCOS), digital light processing (DLP), and liquid crystal display (LCD), based on modified projection parameters. This allows the user's eye to see the complete standard image when viewing the light-modulated projected image at a preset position corresponding to the projection parameters. It is understood that other methods can also be used to modulate light to adjust the projected image; specific methods are not limited here.

[0190] In this embodiment, steps 301 to 304 are steps in the preparation stage before the human-computer interaction system is put into use. Therefore, in actual application, that is, in the use stage of the human-computer interaction system, only steps 305 to 312 can be executed. The specifics are not limited here.

[0191] In this embodiment, the AR-HUD display system uses the user's feature information collected by the eye-tracking device to determine the first pre-distortion model, and then corrects the projected image according to the first pre-distortion model, so that the user can view the complete projected image from different positions, thus improving the user's visual experience.

[0192] Second, the projected virtual image is corrected by acquiring real-time eye gaze information through an eye-tracking device.

[0193] Please see Figure 4 This is another flowchart illustrating the data processing method in an embodiment of this application.

[0194] In this embodiment, the first device is represented by an AR-HUD display system, and the second device is represented by an eye-tracking device.

[0195] In step 401, the eye-tracking device acquires the second image information.

[0196] In step 402, the human eye tracking device calculates the feature location information in the second image information using a feature recognition algorithm.

[0197] In step 403, the eye-tracking device collects depth information.

[0198] In step 404, the human eye tracking device calculates the first position information using feature position information and depth information.

[0199] The method steps performed in steps 401 to 404 of this embodiment are the same as those described above. Figure 3 Steps 305 to 308 in the illustrated embodiment are similar and will not be described in detail here.

[0200] In step 405, the eye-tracking device acquires the user's gaze information.

[0201] The eye-tracking device is also used to acquire the user's gaze information, which represents information about the user's gaze reference point, which is located in the image projected by the first device.

[0202] Specifically, in one possible implementation, when a user enters the vehicle, the user chooses whether to activate the calibration mode, which is used to calibrate the current projected virtual image. If the user activates the calibration mode, the AR-HUD system projects image information with reference point calibration, such as an image calibrated using a dot matrix calibration method or a chessboard calibration method. The reference point represents a point in the dot matrix image or a point in the chessboard; the specific method is not limited here. It is understood that this calibration mode can also be implemented automatically, for example, automatically activating the calibration mode when the current user enters the vehicle; the specific timing or method of activating the calibration mode is not limited here.

[0203] After projecting image information with reference point calibration, the AR-HUD display system prompts the user to look at points in the image information by sending instruction messages. An eye-tracking device collects the user's eye information during this gaze, thus obtaining gaze information. For example, the AR-HUD display system issues a voice prompt to the user to enter calibration mode and projects the calibrated image information onto the windshield. The system voice also instructs the user to look at each calibrated reference point in the image information. If the user's gaze at a reference point exceeds a preset time period, for example, more than 3 seconds, the AR-HUD display system determines that the user has gazed at that reference point and obtains the corresponding eye information. It should be understood that the preset time period of 3 seconds is merely an example; in actual applications, different values ​​can be set according to different scenarios, and this is not limited here. It should be noted that this instruction message can be system voice or information on the projected image used to instruct the user to view the reference points; this is not specifically limited here.

[0204] Specifically, in one possible implementation, when the user gazes at the reference point according to the prompt information, the eye-tracking device emits infrared light to form a bright spot at the pupil of the user's eye. The bright spot is formed at different positions in the pupil depending on the angle between the eye-tracking device and the pupil. The direction of the user's gaze can be calculated by the position of the bright spot relative to the center point of the pupil. The eye-tracking device then determines the coordinates of the reference point actually observed by the user in the projected virtual image based on the position of the user's eye in the preset coordinate system and the direction of the user's gaze.

[0205] It is understandable that in practical applications, eye-tracking devices can also collect the coordinates of the reference points observed by the human eye in other ways, which are not limited here.

[0206] During the process of the eye-tracking device collecting data on the user's gaze at each reference point, the device may not be able to acquire the coordinates of the reference point observed by the user at that point because some reference points are outside the user's field of vision at their current location. After the user has gazed at each observable reference point, the coordinates collected by the eye-tracking device can form an eye calibration image, which is the calibrated image information that the user can observe at their current location, i.e., gaze information.

[0207] In step 406, the eye-tracking device sends first position information and gaze information to the AR-HUD display system.

[0208] After obtaining the first location information and gaze information, the eye-tracking device sends the first location information and gaze information to the AR-HUD display system.

[0209] In step 407, the AR-HUD display system determines the first field of view based on the gaze information.

[0210] After receiving gaze information, the AR-HUD display system determines a first field of view based on the gaze information. This first field of view represents the range of the field of view that the user can observe from the current position.

[0211] Specifically, the AR-HUD display system determines the first field of view based on the human eye calibration image information in the gaze information.

[0212] In step 408, the AR-HUD display system determines the first distortion variable based on the gaze information and the first position information.

[0213] After determining the first field of view, the AR-HUD display system determines the first distortion variable based on the first position information and gaze information. The first distortion variable represents the distortion of the human eye calibration image relative to the standard image, which is a projection image without distortion.

[0214] Specifically, in one possible implementation, the AR-HUD display system obtains the coordinate information of the eye calibration image relative to the first position information based on the user's eye information in the first position information and the coordinates of each reference point in the eye calibration image information in the gaze information. Then, through coordinate transformation, it obtains the position information of the eye calibration image in the preset coordinate system. Finally, it calculates the position information of the eye calibration image in the preset coordinate system and the position information of the standard image in the preset coordinate system to obtain the first distortion variable.

[0215] It is understandable that in practical applications, the first distortion variable can also be determined in other ways, such as by using the position information of a certain reference point in the image calibrated by the human eye in the preset coordinate system and the position information of the corresponding reference point in the standard image calibrated by the same calibration method in the preset coordinate system. The specific method is not limited here.

[0216] In step 409, the AR-HUD display system obtains a first pre-distortion model based on the first field of view and the first distortion variable.

[0217] After obtaining the first distortion variable, the AR-HUD display system obtains the first pre-distortion model based on the first field of view and the first distortion variable.

[0218] Specifically, in one possible implementation, the AR-HUD display system determines the size of the projected virtual image based on the field of view that the user's eyes can see at the current position, and then calculates the transformation mathematical model corresponding to the standard image based on the first distortion variable and the standard image. Finally, it determines the first pre-distortion model based on the transformation mathematical model corresponding to the standard image and the size of the projected virtual image.

[0219] Specifically, in one possible implementation, the AR-HUD display system determines the size of the projected virtual image based on the field of view that the user's eyes can see at the current location, then calculates the modified projection parameters based on the first distortion variable and the projection parameters of the AR-HUD display system, and finally determines the first pre-distortion model based on the modified projection parameters and the size of the projected virtual image.

[0220] It is understandable that in practical applications, the first predistortion model can be determined in other ways, but this is not limited here.

[0221] In step 410, the AR-HUD display system corrects the projected image according to the first pre-distortion model.

[0222] Step 410 in this embodiment is the same as described above. Figure 3 Step 312 in the illustrated embodiment is similar and will not be described in detail here.

[0223] In this embodiment, the AR-HUD display system determines the first pre-distortion model by collecting human eye gaze information, and then corrects the projected image according to the first pre-distortion model, so that users can calibrate the projected image in real time and improve the user experience.

[0224] The information processing method in the embodiments of this application has been described above. The device in the embodiments of this application is described below. Please refer to [link / reference]. Figure 8This is a schematic diagram of the structure of an embodiment of the display device provided in this application.

[0225] A display device, comprising:

[0226] The receiving unit 801 is used to receive first location information sent by the second device. The first location information includes the location information of the first feature in a preset coordinate system. The first feature represents the user's feature information.

[0227] Processing unit 802 is used to obtain a first pre-distortion model based on the first position information;

[0228] The correction unit 803 is used to correct the projected image according to the first pre-distortion model, wherein the projected image is the image projected by the first device.

[0229] In this embodiment, the operations performed by each unit of the display device are the same as those described above. Figure 2 and Figure 3 The AR-HUD display system described in the illustrated embodiment is similar and will not be repeated here.

[0230] Please see Figure 9 This is a schematic diagram of another embodiment of the display device provided in this application.

[0231] A display device, comprising:

[0232] The receiving unit 901 is used to receive first location information sent by the second device. The first location information includes the location information of the first feature in a preset coordinate system. The first feature represents the user's feature information.

[0233] Processing unit 902 is used to obtain a first pre-distortion model based on the first position information;

[0234] The correction unit 903 is used to correct the projected image according to the first pre-distortion model, wherein the projected image is the image projected by the first device.

[0235] Optionally, the display device also includes:

[0236] Acquisition unit 904 is used to acquire second position information based on first position information. The second position information is a position information among a plurality of preset position information whose distance from the first position information in a preset coordinate system is less than a preset threshold. The preset position information is preset by the first device.

[0237] The acquisition unit 904 is also used to acquire the first pre-distortion model corresponding to the second position information.

[0238] Optionally, the receiving unit 901 is further configured to receive at least two first image information sent by the third device, wherein the at least two first image information represent information of images projected by the first device acquired by the third device at different positions in a preset coordinate system;

[0239] The acquisition unit 904 is also used to acquire standard image information, which represents a projection image without distortion;

[0240] The processing unit 902 is further configured to compare at least two pieces of first image information with standard image information respectively to obtain at least two preset distortion variables, wherein the preset distortion variables represent the distortion variables of the first image information relative to the standard image information;

[0241] The processing unit 902 is also configured to calculate at least two first pre-distortion models based on at least two preset distortion variables, wherein the at least two first pre-distortion models correspond one-to-one with the first image information.

[0242] Optionally, the receiving unit 901 is further configured to receive gaze information sent by the second device, the gaze information representing information about the user's gaze reference point, the reference point being marked in the image projected by the first device;

[0243] Display devices also include:

[0244] The determining unit 905 is used to determine the first field of view range based on the gaze information, wherein the first field of view range represents the field of view range observed by the user;

[0245] The determining unit 905 is further configured to determine a first distortion variable based on the gaze information and the first position information. The first distortion variable represents the distortion variable of the human eye calibration image relative to the standard image. The human eye calibration image represents the image presented by the projection image of the first device in the user's human eye. The standard image is a projection image without distortion.

[0246] Processing unit 902 is also used to obtain a first pre-distortion model based on the first field of view and the first distortion variable.

[0247] Optionally, the user's characteristic information includes the user's eye information.

[0248] Optionally, the correction unit 903 is specifically used to perform image processing based on the first pre-distortion model, using one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a field-programmable gate array (FPGA) to correct the projected image.

[0249] Optionally, the correction unit 903 is specifically used to correct the projected image by performing light modulation through one or more of liquid crystal on silicon (LCOS), digital light processing technology (DLP), and liquid crystal display (LCD) according to the first pre-distortion model.

[0250] In this embodiment, the operations performed by each unit of the display device are the same as those described above. Figure 2 and Figure 3 The AR-HUD display system described in the illustrated embodiment is similar and will not be repeated here.

[0251] Please see Figure 10 This is a schematic diagram of an embodiment of the feature acquisition device provided in this application.

[0252] A feature acquisition device, comprising:

[0253] The acquisition unit 1001 is used to acquire first position information, which includes the position information of a first feature in a preset coordinate system. The first feature represents the user's feature information. The first position information is used by the first device to correct the projected image. The projected image is the image projected by the first device.

[0254] The sending unit 1002 is used to send first location information to the first device.

[0255] In this embodiment, the operations performed by each unit of the feature acquisition device are the same as those described above. Figure 2 and Figure 3 The human eye tracking device described in the illustrated embodiment is similar and will not be repeated here.

[0256] Please see Figure 11 This is a schematic diagram of another embodiment of the feature acquisition device provided in this application.

[0257] The acquisition unit 1101 is used to acquire first position information, the first position information including the position information of a first feature in a preset coordinate system, the first feature representing the user's feature information, the first position information being used by the first device to correct the projected image, the projected image being the image projected by the first device;

[0258] The sending unit 1102 is used to send first location information to the first device.

[0259] Optionally, the feature acquisition device also includes:

[0260] Acquisition unit 1103 is used to acquire second image information, which includes user feature information;

[0261] The processing unit 1104 is used to calculate and obtain the first position information based on the second image information.

[0262] Optionally, the processing unit 1104 is specifically used to calculate the feature location information in the second image information by means of a feature recognition algorithm;

[0263] The processing unit 1104 is specifically used to calculate the first position information using the feature position information.

[0264] Optionally, the acquisition unit 1103 is also used to acquire depth information, which represents the straight-line distance from the feature information to the second device;

[0265] Processing unit 1104 is further configured to calculate the first position information using the feature position information, including:

[0266] The processing unit 1104 is also used to calculate the first position information by using the feature position information and the depth information.

[0267] Optionally, the feature information includes the user's eye information.

[0268] Optionally, the acquisition unit 1101 is further configured to acquire the user's gaze information, which represents the information of the user's gaze reference point. The reference point is calibrated in the image projected by the first device. The gaze information is used to determine the first distortion variable, the first distortion variable is used to determine the first pre-distortion model, and the first pre-distortion model is used to correct the projected image.

[0269] The sending unit 1102 is also used to send first location information and gaze information to the first device.

[0270] In this embodiment, the operations performed by each unit of the feature acquisition device are the same as those described above. Figure 2 and Figure 3 The human eye tracking device described in the illustrated embodiment is similar and will not be repeated here.

[0271] It should be noted that, in practical applications, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. For example, the acquisition unit of the feature acquisition device can be a camera, and the determination unit can be a processor. Alternatively, the acquisition unit and the determination unit in the display device can both be located on a single processor, with the processor implementing the functions described by the acquisition unit and the determination unit.

[0272] Please see Figure 12 This is a schematic diagram of another embodiment of the display device provided in this application.

[0273] The display device includes a processor 1201, a memory 1202, a bus 1205, and an interface 1204. The processor 1201 is connected to the memory 1202 and the interface 1204. The bus 1205 connects the processor 1201, the memory 1202, and the interface 1204. The interface 1204 is used to receive or send data. The processor 1201 is a single-core or multi-core central processing unit, a specific integrated circuit, or one or more integrated circuits configured to implement embodiments of the present invention. The memory 1202 can be random access memory (RAM) or non-volatile memory, such as at least one hard disk drive. The memory 1202 is used to store computer-executable instructions. Specifically, the computer-executable instructions may include a program 1203.

[0274] In this embodiment, the processor 1201 can execute the aforementioned... Figure 2 and Figure 3 The specific operations performed by the AR-HUD display system in the illustrated embodiment will not be described in detail here.

[0275] Please see Figure 13 This is a schematic diagram of another embodiment of the feature acquisition device provided in this application.

[0276] The feature acquisition device includes a processor 1301, a memory 1302, a bus 1305, and an interface 1304. The processor 1301 is connected to the memory 1302 and the interface 1304. The bus 1305 connects the processor 1301, the memory 1302, and the interface 1304. The interface 1304 is used to receive or send data. The processor 1301 is a single-core or multi-core central processing unit, a specific integrated circuit, or one or more integrated circuits configured to implement embodiments of the present invention. The memory 1302 can be random access memory (RAM) or non-volatile memory, such as at least one hard disk drive. The memory 1302 is used to store computer-executable instructions. Specifically, the computer-executable instructions may include a program 1303.

[0277] In this embodiment, the processor 1301 can execute the aforementioned... Figure 2 and Figure 3 The specific operations performed by the eye-tracking device in the illustrated embodiment will not be described in detail here.

[0278] It should be understood that the processor mentioned in the above embodiments of this application, or the processor provided in the above embodiments of this application, may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0279] It should also be understood that the number of processors in the above embodiments of this application can be one or more, and can be adjusted according to the actual application scenario. This is merely an illustrative example and is not intended to limit the number of processors. Similarly, the number of memories in the embodiments of this application can be one or more, and can be adjusted according to the actual application scenario. This is merely an illustrative example and is not intended to limit the number of memories.

[0280] It should be noted that when the display device or feature acquisition device includes a processor (or processing unit) and a storage unit, the processor in this application may be integrated with the storage unit, or the processor and the storage unit may be connected through an interface. This can be adjusted according to the actual application scenario and is not limited.

[0281] This application also provides a computer program or a computer program product including a computer program, which, when executed on a computer, will enable the computer to implement the method flow of any of the above method embodiments with an AR-HUD display system or an eye-tracking device.

[0282] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the method flow related to the AR-HUD display system or human eye tracking device in any of the above method embodiments.

[0283] In the above Figures 2-3 In each embodiment, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0284] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0285] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0286] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0287] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0288] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0289] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms "a," "the," and "the" used in the embodiments of this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that in the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship; for example, A / B can represent A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural.

[0290] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: When the calibration mode is activated, the first device receives first location information sent by the second device. The first location information includes the location information of a first feature in a preset coordinate system. The first feature represents the user's feature information. The first location information is calculated by the second device based on the feature location information and depth information. The feature location information is the feature location information of the user's feature information in the second image information. The depth information represents the straight-line distance from the feature information to the second device. The calibration mode is used to indicate that the projected image is calibrated. The first device obtains a first pre-distortion model based on the first position information. The first pre-distortion model represents either the transformation mathematical model of the standard image or the modified projection parameters. The first device corrects the projected image according to the first pre-distortion model, and the projected image is the image projected by the first device. The first device obtains the first pre-distortion model based on the first location information, including: The first device receives gaze information sent by the second device, the gaze information representing information about the user's gaze reference point, the reference point being marked in the image projected by the first device; The first device determines a first field of view based on the gaze information, wherein the first field of view represents the field of view observed by the user; The first device determines a first distortion variable based on the gaze information and the first position information. The first distortion variable represents the distortion of the human eye calibration image relative to the standard image. The human eye calibration image represents the image presented by the projection image of the first device in the user's human eye. The standard image is a projection image without distortion. The first device obtains a first pre-distortion model based on the first field of view and the first distortion variable.

2. The method according to claim 1, characterized in that, The first device obtains a first pre-distortion model based on the first location information, and further includes: The first device obtains second location information based on the first location information. The second location information is a location information among a plurality of preset location information whose distance from the first location information in the preset coordinate system is less than a preset threshold. The preset location information is preset by the first device. The first device acquires the first pre-distortion model corresponding to the second location information.

3. The method according to claim 2, characterized in that, Before the first device receives the first location information sent by the second device, the method further includes: The first device receives at least two first image information sent by the third device, wherein the at least two first image information represent information of images projected by the first device acquired by the third device at different positions in the preset coordinate system; The first device acquires standard image information, which represents a projection image without distortion; The first device compares the at least two first image information with the standard image information respectively to obtain at least two preset distortion variables, wherein the preset distortion variables represent the distortion variables of the first image information relative to the standard image information; The first device calculates at least two first pre-distortion models based on the at least two preset distortion variables, and the at least two first pre-distortion models correspond one-to-one with the first image information.

4. The method according to any one of claims 1 to 3, characterized in that, The user's characteristic information includes the user's eye information.

5. The method according to any one of claims 1 to 3, characterized in that... When the first pre-distortion model represents the transformation mathematical model of the standard image, the first device corrects the projected image according to the first pre-distortion model, including: The first device performs image processing based on the first pre-distortion model using one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a field-programmable gate array (FPGA) to correct the projected image.

6. The method according to any one of claims 1 to 3, characterized in that, When the first predistortion model represents the modified projection parameters, the first device corrects the projected image according to the first predistortion model, including: The first device corrects the projected image by performing light modulation using one or more of the following: liquid crystal on silicon (LCOS), digital light processing (DLP), and liquid crystal display (LCD) based on the first pre-distortion model.

7. A data processing method, characterized in that, include: When calibration mode is activated, the second device acquires first position information, which includes the position information of a first feature in a preset coordinate system. The first feature represents the user's feature information. The first position information is used by the first device to obtain a first pre-distortion model and to correct the projected image based on the first pre-distortion model. The first pre-distortion model represents any one of a transformation mathematical model of a standard image and modified projection parameters. The projected image is an image projected by the first device. Acquiring the first position information by the second device includes: the second device acquiring second image information, which includes the user's feature information; the second device calculating the feature position information of the feature information in the second image information using a feature recognition algorithm; the second device acquiring depth information, which represents the straight-line distance from the feature information to the second device; and the second device calculating the first position information using the feature position information and the depth information. The second device sends the first location information to the first device; After the second device acquires the first location information, the method further includes: The second device acquires the user's gaze information, which represents the information of the user's gaze reference point. The reference point is calibrated in the image projected by the first device. The gaze information is used to determine a first distortion variable, which is used to determine a first pre-distortion model. The first pre-distortion model is used to correct the projected image. The second device sends the first location information to the first device, including: The second device sends the first location information and the gaze information to the first device.

8. The method according to claim 7, characterized in that, The feature information includes the user's eye information.

9. A display device, characterized in that, include: The receiving unit is configured to receive first position information sent by the second device when the calibration mode is activated. The first position information includes the position information of a first feature in a preset coordinate system. The first feature represents the user's feature information. The first position information is calculated by the second device based on the feature position information and depth information. The feature position information is the feature position information of the user's feature information in the second image information. The depth information represents the straight-line distance from the feature information to the second device. The calibration mode is used to indicate that the projected image is calibrated. The processing unit is configured to obtain a first pre-distortion model based on the first position information, wherein the first pre-distortion model represents either a transformation mathematical model of a standard image or modified projection parameters. The correction unit is used to correct the projected image according to the first pre-distortion model, wherein the projected image is an image projected by the first device; The receiving unit is further configured to receive gaze information sent by the second device, the gaze information representing information about the user's gaze reference point, the reference point being calibrated in the image projected by the first device; The display device further includes: A determining unit is configured to determine a first field of view range based on the gaze information, wherein the first field of view range represents the field of view range observed by the user; The determining unit is further configured to determine a first distortion variable based on the gaze information and the first position information. The first distortion variable represents the distortion variable of the human eye calibration image relative to the standard image. The human eye calibration image represents the image presented by the projection image of the first device in the user's human eye. The standard image is a projection image without distortion. The processing unit is further configured to obtain a first pre-distortion model based on the first field of view and the first distortion variable.

10. The display device according to claim 9, characterized in that, The display device further includes: The acquisition unit is configured to acquire second location information based on the first location information, wherein the second location information is a location information among a plurality of preset location information whose distance from the first location information in the preset coordinate system is less than a preset threshold, and the preset location information is preset by the first device. The acquisition unit is also used to acquire the first pre-distortion model corresponding to the second location information.

11. The display device according to claim 10, characterized in that, The receiving unit is also configured to receive at least two first image information sent by the third device, wherein the at least two first image information represent information of the images projected by the first device collected by the third device at different positions in the preset coordinate system; The acquisition unit is also used to acquire standard image information, which represents a projection image without distortion; The processing unit is further configured to compare the at least two first image information with the standard image information respectively to obtain at least two preset distortion variables, wherein the preset distortion variables represent the distortion variables of the first image information relative to the standard image information; The processing unit is further configured to calculate at least two first pre-distortion models based on the at least two preset distortion variables, wherein the at least two first pre-distortion models correspond one-to-one with the first image information.

12. The display device according to any one of claims 9 to 11, characterized in that, The user's characteristic information includes the user's eye information.

13. The display device according to any one of claims 9 to 11, characterized in that... When the first predistortion model represents the transformation mathematical model of the standard image, the correction unit is specifically used to perform image processing based on the first predistortion model using one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a field-programmable gate array (FPGA) to correct the projected image.

14. The display device according to any one of claims 9 to 11, characterized in that, When the first pre-distortion model represents the modified projection parameters, the correction unit is specifically used to correct the projected image by performing light modulation through one or more of silicon-based liquid crystal LCOS, digital light processing technology DLP, and liquid crystal display LCD according to the first pre-distortion model.

15. A feature acquisition device, characterized in that, include: The acquisition unit is used to acquire first position information when the calibration mode is activated. The first position information includes the position information of a first feature in a preset coordinate system. The first feature represents the user's feature information. The first position information is used by the first device to obtain a first pre-distortion model and to correct the projected image according to the first pre-distortion model. The first pre-distortion model represents any one of the transformation mathematical model of the standard image and the modified projection parameters. The projected image is the image projected by the first device. A sending unit is configured to send the first location information to the first device; The acquisition unit is further configured to acquire the user's gaze information, the gaze information representing the information of the user's gaze reference point, the reference point being calibrated in the image projected by the first device, the gaze information being used to determine a first distortion variable, the first distortion variable being used to determine a first pre-distortion model, and the first pre-distortion model being used to correct the projected image; The sending unit is also configured to send the first location information and the gaze information to the first device; The acquisition unit is used to acquire second image information, which includes the user's feature information; The processing unit is specifically used to calculate, through a feature recognition algorithm, the feature location information of the feature information in the second image information; The acquisition unit is also used to acquire depth information, which represents the straight-line distance from the feature information to the second device; The processing unit is further configured to calculate the first location information using the feature location information and the depth information.

16. The feature acquisition device according to claim 15, characterized in that, The feature information includes the user's eye information.

17. A display device, characterized in that, The display device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program stored in the memory to cause the display device to perform the method as described in any one of claims 1-6.

18. A feature acquisition device, characterized in that, The feature acquisition device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program stored in the memory to cause the feature acquisition device to perform the method as described in any one of claims 7-8.

19. A display device, characterized in that, include: Processor and interface circuitry; The interface circuit is used to receive code instructions and transmit them to the processor; The processor is configured to run the code instructions to perform the method as described in any one of claims 1-6.

20. A feature acquisition device, characterized in that, include: Processor and interface circuitry; The interface circuit is used to receive code instructions and transmit them to the processor; The processor is configured to run the code instructions to perform the method as described in any one of claims 7-8.

21. A human-computer interaction system, characterized in that, include: A display device for performing the method as described in any one of claims 1-6; A feature acquisition device for performing the method as described in any one of claims 7-8.

22. A readable storage medium for storing instructions that, when executed, cause the method of any one of claims 1-8 to be implemented.

Images