Dynamic distortion correction methods, devices, equipment and media
By acquiring the observer's current eye position information and mapping relationship, dynamic distortion correction is performed, which solves the image distortion problem caused by changes in the observer's position, and improves the alignment accuracy between the virtual image and the real environment and the user experience.
Patent Information
- Application Number
- CN202611163885.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-03
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-08-03
AI Technical Summary
In the fusion display of virtual images and real environments, dynamic distortion caused by changes in the observer's position affects the alignment accuracy between the image and the real environment and the user's visual experience.
By acquiring the observer's current eye position information, the target distortion value of each point in the image is determined using the mapping relationship, and dynamic distortion correction is performed, including correction of the target dynamic distortion value and static distortion value.
It effectively reduces image distortion caused by changes in the observer's position, improves display quality and accurate spatial positioning in virtual-real fusion applications, and enhances user experience.
Smart Images

Figure CN122656942B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of three-dimensional imaging display technology, and in particular to dynamic distortion correction methods, devices, equipment and media. Background Technology
[0002] In the field of technology involving the fusion of virtual images with real environments, ensuring the accuracy of geometric shape and visual consistency of projected images under different observation conditions is a core technical challenge. This is especially true in applications requiring precise overlay of images onto physical space or a reference plane, such as augmented reality navigation, precision industrial projection, or in-vehicle head-up displays. The optical characteristics of the imaging equipment and changes in the observer's position can significantly impact the final visual effect.
[0003] Typically, imaging devices are optically optimized and calibrated for a preset reference eye position during design. Within this reference eye position, the imaging device can present an image that meets expectations. However, in practical applications, observers often cannot remain at this reference eye position indefinitely; eye position deviation is common. When the observer deviates from the reference eye position, the image, originally calibrated at the reference eye position, undergoes new geometric distortions due to the change in eye position. This results in blurred edges and shifted positions of key feature points in the observed image, severely affecting the alignment accuracy between the image and the real environment, as well as the user's visual experience.
[0004] Therefore, there is an urgent need for a dynamic distortion correction method that can reduce dynamic distortion caused by changes in the observer's position and improve the display effect. Summary of the Invention
[0005] This application provides a dynamic distortion correction method, apparatus, device, and medium to solve the problem of dynamic distortion caused by changes in the observer's position in the prior art.
[0006] In a first aspect, embodiments of this application provide a dynamic distortion correction method, the method comprising: Obtain the position information of the observer's current eye position; based on the position information of the current eye position and the mapping relationship between the saved position information of the eye position and the distortion value corresponding to each point in the image, determine the target distortion value corresponding to each point in the image acquired at the current eye position; The image is distorted based on the target distortion value corresponding to each point. The target distortion value includes the target dynamic distortion value and the target static distortion value; In this process, a preset image is projected onto a set plane for imaging, and the target dynamic distortion value corresponding to each point is determined based on the positional offset between points with the same number in the images acquired by each eye position and the reference eye position. Based on the positional offset between the image acquired from the reference eye position and the points with the same number in the ideal image, determine the target static distortion value corresponding to each point; The ideal image is the image of the preset image on the set plane under ideal conditions.
[0007] In one possible implementation, the process of determining the mapping relationship between the stored eye position information and the distortion value corresponding to each point in the image includes: An imaging device projects a preset image marked with each point onto the set plane to determine a first image acquired at the reference eye position and a second image acquired at multiple test eye positions; each of the multiple test eye positions is any eye position within the eye box space other than the reference eye position. For each test eye position, each point in the target second image acquired at that test eye position is projected onto the imaging plane of the first image to obtain an overlapping image of the first image and the target second image on the imaging plane; based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, and the pre-saved static distortion value corresponding to each numbered point, the mapping relationship between the position information of the test eye position and the distortion value corresponding to each point in the image is determined.
[0008] In one possible implementation, obtaining the overlapping image of the first image and the second image on the imaging plane includes: Based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image, and the depth information of the first image, the position information of each point in the target second image on the imaging plane when projected onto the imaging plane of the first image is determined; wherein, the depth information is a fixed value; The overlapping image is determined based on the position information of the points whose numbers correspond to each other in the first image and the target second image on the imaging plane.
[0009] In one possible implementation, obtaining the overlapping image of the first image and the target second image on the imaging plane includes: Based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image, and the depth information of the first image, multiple iterations are performed, wherein each iteration includes the following process: For each numbered point, based on the relative pose information and the depth information, determine the position information of the point with that number in the second target image on the imaging plane when each point in the second target image is projected onto the imaging plane of the first image; based on the position information of the two points with that number in the first image and the second target image on the imaging plane, determine the distance between the two points with that number in the first image and the second target image on the imaging plane; If the current iteration does not meet the preset iteration completion conditions, the depth information is adjusted based on the first distance between the corresponding points in the first image and the target second image and the second distance between the corresponding points in the first image and the target second image determined in the previous iteration, so as to determine the imaging plane where the adjusted first image is located, and then proceed to the next iteration. If the current iteration meets the preset iteration completion conditions, the iteration ends, and the target iteration with the smallest distance between the corresponding points in the first image and the target second image is determined among all iterations; the overlapping image is determined based on the position information of the corresponding points in the first image and the target second image on the adjusted imaging plane determined by the target iteration.
[0010] In one possible implementation, adjusting the depth information includes: Determine the increase or decrease information between the first distance and the second distance; The depth information is adjusted based on the increase / decrease information and the adjustment trend of the depth information in the previous iteration.
[0011] In one possible implementation, after obtaining the overlapping image of the first image and the target second image on the imaging plane, the method further includes: Determine a first region in the overlapping image that is formed by each point belonging to the first image, and a second region that is formed by each point belonging to each second image; The common area of the first region and each of the second regions is determined, and a target area within the common area that meets the image specifications of the preset image is determined according to the image specifications of the preset image, and the target area is used as the final projection display area.
[0012] In one possible implementation, determining the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image includes: Based on the different position information of the points in the target second image and the first image that correspond to each other in the overlapping image, and the static distortion value corresponding to each numbered point that is saved in advance, the distortion value corresponding to each point in the target second image is determined. The mapping relationship is determined based on the position information of each test eye position and the distortion value of each point in the corresponding target second image.
[0013] In one possible implementation, determining the mapping relationship based on the position information of each test eye position and the distortion value of each point in the corresponding target second image includes: Obtain a pre-configured fitting function, which is used to indicate the relationship between the positional information of each test eye position and the distortion value of each point in the corresponding second image. The fitting function includes the initial value corresponding to each parameter. Perform multiple fitting operations until the preset termination condition is met. Each fitting operation includes the following steps: Based on the fitting function, the position information of the test eye position, and the position information of each point in the target second image, the predicted distortion value corresponding to each point is determined; based on the predicted distortion value and the distortion value corresponding to each point, the residual distortion value corresponding to each point is determined. The total residual distortion value is determined based on the residual distortion value of each numbered point in the second image corresponding to each test eye position, and the preset weight corresponding to each test eye position. The values of the parameters of the fitting function are adjusted based on the total residual distortion value.
[0014] In one possible implementation, determining the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image includes: Based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, a first sub-mapping relationship is determined between the position information of the test eye position and the dynamic distortion value corresponding to each point in the image. Based on the pre-saved static distortion value corresponding to each numbered point, a second sub-mapping relationship is determined between the position information of the test eye and the static distortion value corresponding to each point in the image; The mapping relationship is determined based on the first sub-mapping relationship and the second sub-mapping relationship.
[0015] In one possible implementation, the process of determining the reference eye position includes: Based on the current eye position information, determine the blocking height of the current eye position in the preset eye box space; Obtain the reference eye position corresponding to the saved gear height.
[0016] Secondly, embodiments of this application also provide a dynamic distortion correction device, the device comprising: The acquisition module is used to acquire the position information of the observer's current eye position; The processing module is used to determine the target distortion value corresponding to each point in the image acquired at the current eye position based on the current eye position information and the mapping relationship between the saved eye position information and the distortion value corresponding to each point in the image. The distortion correction module is used to correct the distortion of the image based on the target distortion value corresponding to each point. The target distortion value includes the target dynamic distortion value and the target static distortion value; In this process, a preset image is projected onto a set plane for imaging, and the target dynamic distortion value corresponding to each point is determined based on the positional offset between points with the same number in the images acquired by each eye position and the reference eye position. Based on the positional offset between the image acquired from the reference eye position and the points with the same number in the ideal image, determine the target static distortion value corresponding to each point; The ideal image is the image of the preset image on the set plane under ideal conditions.
[0017] In one possible implementation, the processing module is further configured to project a preset image marked with each point onto the set plane using an imaging device, and determine a first image acquired at the reference eye position and a second image acquired at multiple test eye positions; each of the multiple test eye positions is any eye position within the eye box space other than the reference eye position; for each test eye position, project each point in the target second image acquired at the test eye position onto the imaging plane where the first image is located, to obtain an overlapping image of the first image and the target second image on the imaging plane; and determine the mapping relationship between the position information of the test eye position and the distortion value corresponding to each point in the image based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, and the pre-saved static distortion value corresponding to each numbered point.
[0018] In one possible implementation, the processing module is specifically configured to determine, based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image and the depth information of the first image, the position information of each point in the target second image on the imaging plane when projected onto the imaging plane of the first image; wherein the depth information is a fixed value; and to determine the overlapping image based on the position information of the corresponding points in the first image and the target second image on the imaging plane.
[0019] In one possible implementation, the processing module is specifically configured to perform multiple iterations based on the stored relative pose information between the first image acquisition device acquiring the first image and the second image acquisition device acquiring the target second image, and the depth information of the first image. Each iteration includes the following process: for each numbered point, based on the relative pose information and the depth information, determining the position information of the point with that number in the target second image on the imaging plane when projected onto the imaging plane of the first image; based on the position information of the two points with that number in the first image and the target second image on the imaging plane, determining the two points corresponding to that number in the first image and the target second image on the imaging plane. The distance between points; if the current iteration does not meet the preset iteration completion condition, the depth information is adjusted according to the first distance between the corresponding points in the first image and the target second image and the second distance between the corresponding points in the first image and the target second image determined in the previous iteration, so as to determine the imaging plane where the adjusted first image is located, and the next iteration is performed; if the current iteration meets the preset iteration completion condition, the iteration ends, and the target iteration with the smallest distance between the corresponding points in the first image and the target second image is determined; the overlapping image is determined according to the position information of the corresponding points in the first image and the target second image on the adjusted imaging plane determined by the target iteration.
[0020] In one possible implementation, the processing module is specifically used to determine the increase or decrease information between the first distance and the second distance; and to adjust the depth information according to the increase or decrease information and the adjustment trend of the depth information in the previous iteration.
[0021] In one possible implementation, the processing module is further configured to determine a first region formed by each point belonging to the first image in the overlapping image, and a second region formed by each point belonging to each second image; determine a common region of the first region and each second region, and determine a target region within the common region that meets the image specifications according to the image specifications of the preset image, and use the target region as the final projection display region.
[0022] In one possible implementation, the processing module is specifically configured to determine the distortion value corresponding to each point in the target second image based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, and the static distortion value corresponding to each numbered point that is saved in advance; and to determine the mapping relationship based on the position information of each test eye position and the distortion value of each point in the corresponding target second image.
[0023] In one possible implementation, the processing module is specifically used to obtain a pre-configured fitting function, which indicates the relationship between the positional information of each test eye position and the distortion value of each point in the corresponding second image. The fitting function includes initial values for each parameter. Multiple fitting operations are performed until a preset termination condition is met, wherein each fitting operation includes the following steps: determining the predicted distortion value corresponding to each point based on the fitting function, the positional information of the test eye position, and the positional information of each point in the target second image; determining the residual distortion value corresponding to each point based on the predicted distortion value and the distortion value; determining the total residual distortion value based on the residual distortion value of each numbered point in the second image corresponding to each test eye position and the preset weight corresponding to each test eye position; and adjusting the values corresponding to the parameters of the fitting function based on the total residual distortion value.
[0024] In one possible implementation, the processing module is specifically configured to: determine a first sub-mapping relationship between the position information of the test eye position and the dynamic distortion value corresponding to each point in the image, based on the different position information of the corresponding points numbered in the target second image and the first image on the overlapping image; determine a second sub-mapping relationship between the position information of the test eye position and the static distortion value corresponding to each point in the image, based on the pre-saved static distortion value corresponding to each numbered point; and determine the mapping relationship based on the first sub-mapping relationship and the second sub-mapping relationship.
[0025] In one possible implementation, the processing module is specifically used to determine the height of the current eye position in a preset eye box space based on the position information of the current eye position; and to obtain the reference eye position corresponding to the saved height of the current eye position.
[0026] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor, the processor being configured to execute a computer program stored in a memory to implement the steps of the dynamic distortion correction method as described in any of the above.
[0027] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the dynamic distortion correction method as described in any of the above.
[0028] In this embodiment, the electronic device determines the target distortion value corresponding to each point in the image acquired at the current eye position based on the observer's current eye position information and the saved mapping relationship between the eye position information and the distortion value corresponding to each point in the image. This target distortion value includes a target static distortion value and a target dynamic distortion value. Then, distortion correction is performed on the image based on the target dynamic distortion value and the target static distortion value corresponding to each point. By performing dual correction of dynamic and static distortion for each point, this embodiment ensures that regardless of how the observer's eye moves within the eye box space, the observed image is one with reduced distortion. This not only suppresses image distortion caused by parallax but also lays a precise spatial positioning foundation for subsequent augmented reality navigation and other virtual-real fusion applications, greatly improving the display effect. Attached Figure Description
[0029] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1 This is a schematic diagram of a dynamic distortion correction process provided in an embodiment of this application; Figure 2 A schematic diagram of a preset image provided in an embodiment of this application; Figure 3 A schematic diagram showing the distribution of multiple test eye positions provided in an embodiment of this application; Figure 4 A schematic diagram of a second image acquired at a test eye position, provided as an embodiment of this application; Figure 5This is a schematic diagram of the second image being back-projected onto the imaging plane of the first image, as provided in an embodiment of this application. Figure 6 A flowchart illustrating the determination of the mapping relationship between the eye position information and the dynamic distortion value corresponding to each point in the image, as provided in the embodiments of this application. Figure 7 This is a schematic diagram of a dynamic distortion correction device provided in an embodiment of this application; Figure 8 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0032] In the field of technology involving the fusion of virtual images with real environments, ensuring the accuracy of geometric shape and visual consistency of projected images under different observation conditions is a core technical challenge. This is especially true in applications requiring precise overlay of images onto physical space or a reference plane, such as augmented reality navigation, precision industrial projection, or in-vehicle head-up displays. The optical characteristics of the imaging equipment and changes in the observer's position can significantly impact the final visual effect.
[0033] Typically, imaging devices are optically optimized and calibrated for a preset reference eye position during design. Within this reference eye position, the imaging device can present an image that meets expectations. However, in practical applications, observers often cannot remain at this reference eye position indefinitely; eye position deviation is common. When the observer deviates from the reference eye position, the image, originally calibrated at the reference eye position, undergoes new geometric distortions due to the change in eye position. This results in blurred edges and shifted positions of key feature points in the observed image, severely affecting the alignment accuracy between the image and the real environment, as well as the user's visual experience.
[0034] The following is one possible application scenario: A head-up display (HUD) is a novel way of displaying information in a vehicle by reflecting light onto the windshield. Specifically, the display device of the imaging unit emits display light, which is then projected onto the windshield through corresponding optical lenses to produce a virtual image.
[0035] However, in practical applications, factors such as changes in the observation angle caused by the driver's eye movements (forward, backward, left, right, up, and down), the relative attitude shift between the projection module and the windshield caused by bumps, pitch, and turns during vehicle operation, the slight deformation caused by thermal expansion and contraction of optical lenses and structural components due to changes in ambient temperature, and the displacement of components due to road vibration and long-term use can all cause the originally accurately calibrated image to become distorted, misaligned, have edge deformation, or be offset. Moreover, the size and shape of the distortion continue to change and cannot be compensated for once with a fixed set of calibration parameters.
[0036] Therefore, there is an urgent need for a dynamic distortion correction method that can reduce dynamic distortion caused by changes in the observer's position and improve the display effect.
[0037] Based on this, in order to avoid dynamic distortion caused by changes in the observer's position and improve the display effect, this application provides a dynamic distortion correction method. This dynamic distortion correction method can be executed in an electronic device, which can be any one of a vehicle, a vehicle processor, a vehicle controller, a HUD processor, a controller, an imaging device, an imaging device processor, an imaging device controller, or other components with computing functions. The dynamic distortion correction method is used to determine the distortion value between the image acquired at the current eye position and the image acquired at a reference eye position based on the observer's current eye position information, so as to perform distortion correction based on the distortion value and improve the accuracy of the image acquired at the current eye position.
[0038] Specifically, the dynamic distortion correction method provided in this application includes: acquiring the position information of the observer's current eye position; determining the target distortion value corresponding to each point in the image acquired at the current eye position based on the current eye position information and the mapping relationship between the saved eye position information and the distortion value corresponding to each point in the image; and performing distortion correction on the image based on the target distortion value corresponding to each point. The target distortion value includes a target dynamic distortion value and a target static distortion value. Specifically, a preset image is projected onto a set plane for imaging, and the target dynamic distortion value corresponding to each point is determined based on the positional offset between points with the same number in the images acquired at each eye position and the reference eye position. The target static distortion value corresponding to each point is determined based on the positional offset between points with the same number in the image acquired at the reference eye position and the ideal image. The ideal image is the image of the preset image on the set plane under ideal conditions.
[0039] The dynamic distortion correction process provided in this application is applicable to the technical field of virtual image and real environment fusion display, especially in application scenarios that require precise superposition of images on physical space or reference plane, such as augmented reality navigation, precision industrial projection or vehicle head-up display, etc., and is not limited thereto.
[0040] The dynamic distortion correction method provided in the embodiments of this application will be described in the following examples.
[0041] In one possible implementation, Figure 1 This application provides a schematic diagram of a dynamic distortion correction process, which includes: S101: Obtain the position information of the observer's current eye position; based on the position information of the current eye position and the mapping relationship between the saved position information of the eye position and the distortion value corresponding to each point in the image, determine the target distortion value corresponding to each point in the image acquired at the current eye position.
[0042] In this embodiment of the application, after determining that the current dynamic distortion correction conditions are met, the electronic device can obtain the position information of the observer's current eye position, thereby determining the target dynamic distortion value based on the position, and performing dynamic distortion correction based on the target dynamic distortion value.
[0043] The electronic device determines that the current conditions for dynamic distortion correction are met either by receiving an instruction or by determining that dynamic distortion correction can be performed based on the current state of the imaging device.
[0044] The following describes several possible ways in which electronic devices determine whether the current dynamic distortion correction conditions are met: Method 1: If the electronic device detects that the imaging device has been started, or detects that the current state of the imaging device is powered on, then the electronic device determines that the current dynamic distortion correction conditions are met.
[0045] Method 2: If the electronic device receives a command to enable the 3D imaging function of the imaging device, or recognizes that the current state of the imaging device is the 3D imaging function enabled, then the electronic device determines that the current dynamic distortion correction conditions are met.
[0046] Method 3: If the electronic device receives a dynamic distortion correction command, then the electronic device determines that the current dynamic distortion correction conditions are met.
[0047] In this embodiment, the electronic device can acquire the observer's current eye position information through an image acquisition device. This image acquisition device can be a Driver Monitoring System (DMS), camera, radar sensor, infrared camera, etc., and is not limited thereto. The observer's current eye position information can be the relative position of the current eye position to a reference eye position, or the relative position of the current eye position in the eyebox space, or the relative position of the current eye position in the world coordinate system. The current eye position information can be three-dimensional coordinate information.
[0048] For example, in this embodiment of the application, the electronic device can acquire an image of the observer's face using an image acquisition device, and perform key point recognition on the face image to determine the position of the observer's eyes in the face image. Then, based on the installation position and parameters of the image acquisition device, the spatial position information corresponding to the eye position is determined, and this spatial position information is the position information of the observer's current eye position.
[0049] In addition, electronic devices can also obtain the position information of the observer's current eye position in other ways, such as determining the point cloud data corresponding to the observer's eye, and determining the position information corresponding to the point cloud data as the position information of the observer's current eye position.
[0050] In this embodiment, the electronic device stores a mapping relationship between eye position information and the distortion value corresponding to each point in the image acquired at that eye position. After determining the observer's current eye position information, the electronic device can determine the target distortion value corresponding to each point in the image acquired at that current eye position based on the mapping relationship.
[0051] Here, each of the aforementioned points can be all the pixels in the image, or it can be a predetermined number of pixels in the image. If each point is a predetermined number of pixels in the image, then the value of the predetermined number is not limited, and the distribution of the predetermined number of pixels in the image is also not limited. Preferably, the predetermined number of pixels is evenly distributed in the image, or the majority of the predetermined number of pixels is distributed in the edge region of the image, and the minority is distributed in the center region of the image.
[0052] In this embodiment of the application, the mapping relationship stored in the electronic device can be a mapping table that records the distortion value of each point in the image corresponding to each eye position, or it can be a mathematical model established through physical measurement, which indicates the relationship between the position information of the eye position, the position information (u,v) of the point, and the distortion value (du,dv).
[0053] Furthermore, in this embodiment of the application, the target distortion value corresponding to each point can be a two-dimensional vector. The values corresponding to the two dimensions of the vector are the distortion component value in the x-direction and the distortion component value in the y-direction of the image coordinate system, respectively. That is, if the position information of the point is (u,v), then the target distortion value corresponding to the point is (du,dv).
[0054] In this embodiment, the target distortion value corresponding to each point in the image acquired at the current eye position can be a target dynamic distortion value, or it can be both a target dynamic distortion value and a target static distortion value. Specifically, the target dynamic distortion value is the positional offset between points with the same number in the images acquired at the current eye position and the reference eye position when a preset image is projected onto a set plane for imaging; the static distortion value is the positional offset between points with the same number in the image acquired at the reference eye position and the ideal image.
[0055] Based on the above embodiments, the following is a detailed description of the process for determining the mapping relationship between the saved eye position information and the distortion value corresponding to each point in the image: To improve the accuracy of distortion correction, based on the above embodiments, in this embodiment, the process of determining the mapping relationship between the saved eye position information and the distortion value corresponding to each point in the image includes: An imaging device projects a preset image marked with each point onto the set plane to determine a first image acquired at the reference eye position and a second image acquired at multiple test eye positions; each of the multiple test eye positions is any eye position within the eye box space other than the reference eye position. For each test eye position, each point in the target second image acquired at that test eye position is projected onto the imaging plane of the first image to obtain an overlapping image of the first image and the target second image on the imaging plane; based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, and the pre-saved static distortion value corresponding to each numbered point, the mapping relationship between the position information of the test eye position and the distortion value corresponding to each point in the image is determined.
[0056] In this embodiment, the electronic device can control the imaging device to project a preset image containing specific markers onto a set plane. The preset image can be a high-contrast dot matrix or a checkerboard pattern, and each dot in the preset image is marked. If the preset image is a dot matrix, the dots in the preset image are used as each marked dot; if the preset image is a checkerboard pattern, the intersections of the squares in the preset image are used as each marked dot.
[0057] Example, Figure 2 A schematic diagram of a preset image provided in an embodiment of this application, as shown below. Figure 2As shown, the preset image is a black and white bitmap, where each dot represents a blank circular area in the image. The dot density in the preset image can be adjusted as needed. The more dots each dot contains in the preset image, the more accurate the mapping relationship obtained based on the preset image. However, the more dots there are, the greater the computational burden on the electronic device. Furthermore, the image size of the preset image is related to the resolution of the display device.
[0058] It should be noted that any one of the points in the embodiments of this application can be a specific pixel or a closed region including multiple pixels. The shape of the closed region is not limited, such as a circle or a square.
[0059] After the electronic device controls the imaging device to project a preset image containing specific markers onto a set plane, the electronic device can use image acquisition devices installed in the eye box space to simulate the human eye's viewing angle, acquiring a first image at a reference eye position and second images at multiple test eye positions. These multiple test eye positions cover several key locations within the eye box space. The image acquisition process can be either a single image acquisition device moving sequentially to the reference eye position and multiple test eye positions under the drive of a precision displacement stage, or an array of multiple fixed image acquisition devices simultaneously acquiring the first and second images to improve image acquisition efficiency.
[0060] Multiple test eye positions can be set at different locations within the eye box space to acquire the second image. Each of these test eye positions can be any location within the eye box space other than the reference eye position. The reference eye position can be the exact center of the eye box space or any other location; there are no restrictions. Furthermore, the number of test eye positions and their distribution within the eye box space are not limited and can be determined based on the actual environment, such as the size of the visible field of view and the degree of dynamic distortion.
[0061] Figure 3 This is a schematic diagram showing the distribution of multiple test eye positions provided in an embodiment of this application. Figure 3 As shown, any location within the eye box space can be used as the test eye position, and a camera can be deployed at the location of the test eye position to acquire a second image. Figure 3 The eye box space shown contains four test eye positions and one reference eye position. The four cameras at the edges are the test eye positions, and the camera in the center is the reference eye position.
[0062] It should be noted that due to dynamic distortion, the distribution of points in the second image acquired at each test eye position may differ from the distribution in the preset image. For example, if the preset image is as described above... Figure 2 The dot matrix pattern shown is... Figure 4This is a schematic diagram of a second image acquired at a test eye position, as provided in an embodiment of this application. Figure 4 As shown, the distance between any two adjacent points in the preset image is consistent, but the distance between any two adjacent points in the second image changes, meaning that the second image exhibits dynamic distortion compared to the preset image.
[0063] Based on this, in this embodiment, the key to determining the mapping relationship lies in finding the correspondence of the same point at different eye positions in the image coordinate system. Specifically, the electronic device controls the image acquisition device to acquire a second image at the location of each test eye position, and projects each point in each second image back onto the imaging plane of the first image through geometric transformation, so as to obtain an overlapping image of the first image and each second image on the imaging plane. Then, the electronic device determines the mapping relationship between the position information of each eye position and the different positions of the corresponding points in the first and second images on the overlapping image, based on the position information of each eye position and the different positions of the corresponding points in the first and second images on the overlapping image.
[0064] Example, Figure 5 This is a schematic diagram of the second image being back-projected onto the imaging plane of the first image, as provided in the embodiments of this application. Figure 5 As shown, by back projection, each point in the second image acquired at eye position 1 can be projected onto the imaging plane of the first image acquired at the reference eye position, so as to obtain an overlapping image of the first image and the second image on the imaging plane.
[0065] In this embodiment, if two points in the overlapping image correspond to the same point in the preset image, then the two points are considered to be corresponding points. To accurately determine the corresponding points in the overlapping image, before projection using an imaging device, each point in the preset image is numbered, ensuring that each point in the projected first image and each second image has a corresponding number, and consequently, each point in the overlapping image also has a number. Based on this, every two points with the same number in the overlapping image can be identified as corresponding points, i.e., points whose numbers correspond to each other.
[0066] In one possible implementation, for each second image, the electronic device can determine the projection position of each point in the second image onto the imaging plane of the first image, and mark each projection position as a numbered point, wherein the number of the point at each projection position is consistent with the number of its corresponding point in the second image. Then, based on the position information of each eye position and the different position information of the corresponding points in the first and second images on the overlapping image, the electronic device determines the mapping relationship between the eye position information and the distortion value corresponding to each point in the image.
[0067] Since the number of second images is related to the number of test eye positions, the electronic device uses the same technical means to determine the overlapping image corresponding to the first image and each second image. Therefore, this application embodiment will take the process of determining the overlapping image corresponding to the first image and the target second image as an example, wherein the target second image is the second image acquired by any test eye position.
[0068] Specifically, the following describes several possible methods for determining overlapping images: Method 1: Obtaining the overlapping image of the first image and the second image on the imaging plane includes: Based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image, and the depth information of the first image, the position information of each point in the target second image on the imaging plane when projected onto the imaging plane of the first image is determined; wherein, the depth information is a fixed value; The overlapping image is determined based on the position information of the points whose numbers correspond to each other in the first image and the target second image on the imaging plane.
[0069] In this first method, the electronic device acquires the relative pose information between the first image acquisition device that acquires the first image and the second image acquisition device that acquires the target second image. This relative pose information can be a relative six degrees of freedom (6DoF) pose, or it can be a three-dimensional translation, etc. Furthermore, the electronic device can also acquire the depth information of the saved first image.
[0070] Electronic devices can construct perspective projection equations by combining the relative pose information between the first image acquisition device of the first image and the second image acquisition device of the target second image, the camera intrinsic parameter matrices corresponding to the first image acquisition device and the second image acquisition device respectively, and the depth information. This allows them to project each point in the target second image onto the imaging plane of the first image and determine the position information of the projected point corresponding to each point in the target second image in the overlapping image.
[0071] It should be noted that the depth information used by the electronic device in the above embodiments is the Virtual Image Distance (VID). This depth information is a fixed value, and the specific value of this depth information can be specified by optical design parameters or by industry experts.
[0072] Method 2: Obtaining the overlapping image of the first image and the target second image on the imaging plane includes: Based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image, and the depth information of the first image, multiple iterations are performed, wherein each iteration includes the following process: For each numbered point, based on the relative pose information and the depth information, determine the position information of the point with that number in the second target image on the imaging plane when each point in the second target image is projected onto the imaging plane of the first image; based on the position information of the two points with that number in the first image and the second target image on the imaging plane, determine the distance between the two points with that number in the first image and the second target image on the imaging plane; If the current iteration does not meet the preset iteration completion conditions, the depth information is adjusted based on the first distance between the corresponding points in the first image and the target second image and the second distance between the corresponding points in the first image and the target second image determined in the previous iteration, so as to determine the imaging plane where the adjusted first image is located, and then proceed to the next iteration. If the current iteration meets the preset iteration completion conditions, the iteration ends, and the target iteration with the smallest distance between the corresponding points in the first image and the target second image is determined among all iterations; the overlapping image is determined based on the position information of the corresponding points in the first image and the target second image on the adjusted imaging plane determined by the target iteration.
[0073] In practical optical and electronic devices, due to the presence of higher-order aberrations such as astigmatism and field curvature, the effective imaging depth at different field of view may not be a strictly plane, or the true virtual image distance may not be accurately known during the calibration phase. In such cases, simply relying on preset, fixed depth information may lead to deviations in the positional information of corresponding points in overlapping images.
[0074] Based on this, the embodiments of this application continuously adjust the value of the depth information to adjust the imaging plane where the first image is located, so that the distance error between the corresponding points in the overlapping image obtained by back projection calculation converges to the minimum value.
[0075] In other words, in this embodiment, the depth information used by the electronic device to determine overlapping images can also be obtained through iterative optimization. That is, multiple overlapping images are determined using multiple depth information, and the optimal projection result is selected by comparison. Specifically, the electronic device can set an iteration completion condition and an adjustment step size corresponding to the depth information. Before the first iteration, the electronic device selects an initial value as the depth information corresponding to the first image. This initial value can be a value obtained based on optical design or it can be randomly generated.
[0076] Since the processing procedure of the electronic device is consistent in each iteration, this embodiment will use one iteration as an example to illustrate the process by which the electronic device determines the overlapping image: During this iteration, for each numbered point, the electronic device determines the position information of the point with that number in the second target image on the imaging plane when each point in the second target image is projected onto the imaging plane of the first image, based on the relative pose information and depth information; and determines the distance between the two points with that number in the first image and the second target image on the imaging plane based on the position information of the two points with that number in the first image and the second target image.
[0077] Among them, the electronic device can use the Euclidean distance algorithm to determine the distance between two points corresponding to any number on the imaging plane.
[0078] Afterwards, the electronic device will determine whether the iteration completion condition is met. If it is met, the iteration process ends; otherwise, it continues to the next iteration. The iteration completion condition can be reaching a preset maximum number of iterations, or the change in distance between two points corresponding to the same number in at least two consecutive rounds being less than a preset threshold, indicating that the algorithm has converged, or the minimum distance between two points corresponding to each other has been found.
[0079] Among them, the minimum distance between the two points corresponding to the number has been shown to be: regardless of whether the value of the depth information is increased or decreased, the distance between the two points corresponding to the number is greater than the distance between the two points in the previous iteration.
[0080] For example, an electronic device can determine whether the number of iterations in the current iteration is not less than a preset number. If the number of iterations is less than the preset number, it determines not to end the iteration process and enters the next iteration; if the number of iterations is not less than the preset number, it determines to end the iteration process.
[0081] In this embodiment, after the iteration ends, the electronic device backtracks the calculation results of all iteration rounds and selects the iteration round with the smallest distance between two corresponding points as the target iteration. The electronic device determines the depth information used in this target iteration as the optimal virtual image distance corresponding to the reference eye position in the current scene, and the position information of each numbered point in the target second image calculated by this target iteration on the imaging plane is determined as the position information of each numbered point in the overlapping image.
[0082] Furthermore, in order to improve the accuracy of the projection point determination, in this embodiment of the application, if it is determined not to end the iteration process, the electronic device can adjust the depth information based on the first distance between the numbered corresponding points in the current iteration and the second distance between the numbered corresponding points determined in the previous iteration before entering the next iteration.
[0083] In this iteration, the first distance between the corresponding points with numbers can be the average distance between the corresponding points with numbers on the imaging plane in this iteration, and the second distance between the corresponding points with numbers determined in the previous iteration can be the average distance between the corresponding points with numbers on the imaging plane in the previous iteration.
[0084] For example, in an embodiment of this application, the electronic device can determine the first average distance between points corresponding to each number on the imaging plane in the current iteration based on the distance between points corresponding to each number in the current iteration, and use the first average as the first distance; and determine the second average distance between points corresponding to each number on the imaging plane in the previous iteration based on the distance between points corresponding to each number in the previous iteration, and use the second average as the second distance.
[0085] In one possible implementation, adjusting the depth information includes: Determine the increase or decrease information between the first distance and the second distance; The depth information is adjusted based on the increase / decrease information and the adjustment trend of the depth information in the previous iteration.
[0086] In this embodiment of the application, the electronic device can adjust the depth information based on the increase or decrease information between the first distance between the numbered corresponding points in the current iteration and the second distance between the numbered corresponding points determined in the previous iteration, as well as the adjustment trend of the depth information in the previous iteration.
[0087] The following describes several possible ways to adjust the depth information based on the addition or subtraction of information between the first and second distances: Method 1: Adjust the value corresponding to the depth information based on a fixed adjustment step size.
[0088] Specifically, if the increase / decrease information between the first and second distances is an increase (i.e., the first distance is greater than the second distance), then the depth information used in this iteration is not better than the depth information used in the previous iteration, indicating that the adjustment trend of the depth information in the previous iteration was incorrect. This iteration needs to use the opposite adjustment trend to adjust the depth information, with the adjustment value being the preset adjustment step size. If the increase / decrease information between the first and second distances is a decrease (i.e., the first distance is less than the second distance), then the depth information used in this iteration is better than the depth information used in the previous iteration, indicating that the adjustment trend of the depth information in the previous iteration was correct. This iteration can continue to use this adjustment trend to adjust the depth information, with the adjustment value being the preset adjustment step size.
[0089] In this method, the adjustment step size is a fixed value.
[0090] For example, if the increment / decrease information is decreasing, and the adjustment trend of the previous round's adjustment of the depth information value is increasing, then the depth information value will continue to increase; if the increment / decrease information is decreasing, and the adjustment trend of the previous round's adjustment of the depth information value is decreasing, then the depth information value will continue to decrease. If the increment / decrease information is increasing, and the adjustment trend of the previous round's adjustment of the depth information value is increasing, then the depth information value will decrease; if the increment / decrease information is increasing, and the adjustment trend of the previous round's adjustment of the depth information value is decreasing, then the depth information value will increase.
[0091] Method 2: Adjust the value corresponding to the depth information based on the change in adjustment step size.
[0092] In the embodiments of this application, the adjustment step size when adjusting the depth information in each iteration can be a preset fixed value, and the adjustment step size can also be flexibly set according to the actual situation.
[0093] For example, the electronic device can determine the amount of change of the first distance relative to the second distance. If the amount of change is large, the adjustment step size can be set to be larger; if the amount of change is small, the adjustment step size can be set to be smaller.
[0094] In one possible implementation, the electronic device may pre-store a reference adjustment step size corresponding to a reference change amount, and an adjustment value corresponding to a set change amount step size. After the electronic device determines the change amount between the first distance of the current iteration and the second distance of the previous iteration, the electronic device can determine the difference between the change amount and the reference change amount, and determine a first multiple relationship between the difference and the set change amount step size. Based on the first multiple relationship, the adjustment value, and the reference adjustment step size, the adjustment step size corresponding to the current iteration is determined.
[0095] For example, if the change between the first distance in the current iteration and the second distance in the previous iteration is consistent with the baseline change, the electronic device determines the baseline adjustment step size corresponding to the stored baseline change as the adjustment step size for the current iteration. If the change is inconsistent with the baseline change, the electronic device determines the first product of the first multiple relationship and the adjustment value, and determines the adjustment step size for the current iteration based on the magnitude relationship between the change and the baseline change, the first product, and the baseline adjustment step size.
[0096] Specifically, if the change between the first distance in this iteration and the second distance in the previous iteration is greater than the baseline change, then the first sum of the first product and the baseline adjustment step size is determined, and the first sum is determined as the adjustment step size corresponding to this iteration; if the change is less than the baseline change, then the first difference between the first product and the baseline adjustment step size is determined, and the first difference is determined as the adjustment step size corresponding to this iteration.
[0097] In another possible implementation, the electronic device may only store the adjustment value corresponding to the set change step size. After determining the change between the first distance of the current iteration and the second distance of the previous iteration, the electronic device can determine the second multiple relationship between the set change step sizes, and then determine the adjustment step size corresponding to the current iteration based on the second multiple relationship and the adjustment value.
[0098] For example, the electronic device determines the second product of the second multiple relationship and the adjustment value, and determines the second product as the adjustment step size corresponding to the current iteration.
[0099] In this embodiment, the electronic device adaptively finds the imaging plane with the highest overlap by iterating a preset number of times, compensating for the manufacturing error of the imaging device, thereby achieving more accurate point projection than fixed depth information, and providing data assurance for high-precision dynamic distortion correction.
[0100] Based on the above embodiments, in this embodiment, after determining the overlapping image of the first image and the target second image on the imaging plane, the electronic device can begin to calculate the mapping relationship. Several possible methods for determining the mapping relationship are described below: Method 1: Determining the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image includes: Based on the different position information of the points in the target second image and the first image that correspond to each other in the overlapping image, and the static distortion value corresponding to each numbered point that is saved in advance, the distortion value corresponding to each point in the target second image is determined. The mapping relationship is determined based on the position information of each test eye position and the distortion value of each point in the corresponding target second image.
[0101] In this method, after determining the different position information of the corresponding points in the target second image and the first image on the overlapping image, the distance between the corresponding points on the overlapping image can be determined based on the different position information of the corresponding points on the overlapping image, and the distance can be used as the dynamic distortion value corresponding to each numbered point.
[0102] Afterwards, the static distortion value corresponding to each numbered point can be obtained. The static distortion value is the positional offset between the points with the same number in the first image and the ideal image. The ideal image is the imaging of the preset image on a set plane under ideal conditions. The ideal state is a state without any distortion.
[0103] In this embodiment, after determining the dynamic and static distortion values corresponding to each numbered point, the electronic device can superimpose the dynamic and static distortion values corresponding to each numbered point to obtain the distortion value corresponding to that point. The electronic device determines the mapping relationship based on the position information of each test eye position and the distortion value of each point in the corresponding target second image.
[0104] In one possible implementation, based on the above embodiments, in this application embodiment, determining the mapping relationship according to the position information of each test eye position and the distortion value of each point in the corresponding target second image includes: Obtain a pre-configured fitting function, which is used to indicate the relationship between the positional information of each test eye position and the distortion value of each point in the corresponding second image. The fitting function includes the initial value corresponding to each parameter. Perform multiple fitting operations until the preset termination condition is met. Each fitting operation includes the following steps: Based on the fitting function, the position information of the test eye position, and the position information of each point in the target second image, the predicted distortion value corresponding to each point is determined; based on the predicted distortion value and the distortion value corresponding to each point, the residual distortion value corresponding to each point is determined. The total residual distortion value is determined based on the residual distortion value of each numbered point in the second image corresponding to each test eye position, and the preset weight corresponding to each test eye position. The values of the parameters of the fitting function are adjusted based on the total residual distortion value.
[0105] In this embodiment, the electronic device is pre-configured with a fitting function, which indicates the relationship between the positional information of each test eye and the distortion value of each point in the corresponding second image. This fitting function can be the least squares method or other optimization algorithms. Furthermore, the fitting function includes initial values for each parameter, and the electronic device can perform multiple fittings based on this function to obtain the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image.
[0106] Since the fitting process is consistent each time, this embodiment of the application will use a single fitting process as an example for illustration. This fitting process includes the following: The electronic device calculates the predicted distortion value for each point in the target second image based on the position information of the test eye and the position information of each point using a fitting function; then, based on the predicted distortion value and the actual distortion value, it determines the residual distortion value for each point. This distortion value is determined based on the method described above, and is the actual distortion value.
[0107] The electronic device determines the total residual distortion value based on the residual distortion value of each numbered point in the second image corresponding to each test eye position, and adjusts the values corresponding to the parameters of the fitting function based on the total residual distortion value.
[0108] Specifically, the fitting function incorporates the positional information of the test eye corresponding to the second target image. and the location information of each point As an input variable, the distortion value of the point. As an output variable, for example, the fitting function can be and Here, F() and G() represent the mapping relationship, with no limit on the input parameters and implementation methods, such as the simplest linear polynomial scheme: The coefficients a, b, c, d, e, and f are initial values. The electronic device can substitute the positional information of the test eye and the positional information of each point in the second target image into the fitting function to determine the predicted distortion value corresponding to each point.
[0109] In the actual fitting process, to improve the practicality of the correction, the electronic device can also assign different weights to different eye positions in different regions. For example, the central region of the eye box is the area where the observer's gaze lingers the longest, so the calibration data of this region is given a higher weight to ensure that the image of the central visual field is extremely accurate; while the edge region of the eye box is given a lower weight to allow for small residuals. The electronic device determines the total residual distortion value based on the residual distortion value of each numbered point in the second image corresponding to each test eye position, as well as the preset weight corresponding to each test eye position.
[0110] Method 2: Determining the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image includes: Based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, a first sub-mapping relationship is determined between the position information of the test eye position and the dynamic distortion value corresponding to each point in the image. Based on the pre-saved static distortion value corresponding to each numbered point, a second sub-mapping relationship is determined between the position information of the test eye and the static distortion value corresponding to each point in the image; The mapping relationship is determined based on the first sub-mapping relationship and the second sub-mapping relationship.
[0111] In this second method, the electronic device can first determine the first sub-mapping relationship between the position information of the test eye position and the dynamic distortion value corresponding to each point in the overlapping image, based on the different position information of the corresponding points in the target second image and the first image. The process of the electronic device determining the first sub-mapping relationship is the same as the process of the electronic device determining the mapping relationship based on the distortion value of each point in the first method, and will not be described again here.
[0112] The electronic device can also determine a second sub-mapping relationship between the position information of the test eye and the static distortion value corresponding to each point in the image, based on the pre-saved static distortion value corresponding to each numbered point. In this second mapping relationship, the static distortion values corresponding to the same numbered points in different images acquired from different eye positions are the same or approximately the same. Then, the electronic device combines the first and second sub-mapping relationships into a final mapping relationship.
[0113] In practical applications, if the mapping relationship includes a first sub-mapping relationship and a second sub-mapping relationship, when the electronic device performs distortion correction on the image acquired at the current eye position, it can first determine the target dynamic distortion value corresponding to each point in the image based on the first sub-mapping relationship, then determine the target static distortion value corresponding to each point in the image based on the second sub-mapping relationship, and determine the distortion value corresponding to each point based on the target dynamic distortion value and the target static distortion value, and then correct the image based on the distortion value corresponding to each point.
[0114] Since the static distortion value corresponding to each point is the positional offset between the point with the same number in the image acquired from the reference eye position and the ideal image, the static distortion values corresponding to the same number in different images acquired from different eye positions are the same or approximately the same. In this embodiment, when determining the mapping relationship, the dynamic distortion value and the static distortion value are separated, that is, the first sub-mapping relationship corresponding to the dynamic distortion value and the second sub-mapping relationship corresponding to the static distortion value are determined separately. This can avoid the influence of the static distortion value on the fitting process of the dynamic distortion value, thereby improving the fitting accuracy.
[0115] Based on the above embodiments, in this embodiment, after obtaining the overlapping image of the first image and the target second image on the imaging plane, the method further includes: Determine a first region in the overlapping image that is formed by each point belonging to the first image, and a second region that is formed by each point belonging to each second image; The common area of the first region and each of the second regions is determined, and a target area within the common area that meets the image specifications of the preset image is determined according to the image specifications of the preset image, and the target area is used as the final projection display area.
[0116] In practical applications, the visible range corresponding to different eye positions may be inconsistent, and inconsistent visible range may lead to image cropping and black border problems. Therefore, in order to ensure that the final output corrected image has complete visibility under all possible eye positions, the embodiments of this application determine the mapping relationship between the position information of the eye position and the distortion value corresponding to each point in the image based on the display area that can be seen by all eye positions.
[0117] Furthermore, the edges of the projected image seen by the eyes of observers at different positions will change; some image areas visible from the central eye position may be cropped at the edge eye positions. Therefore, and to ensure the integrity of the displayed content, this application embodiment determines the mapping relationship between the eye position information and the distortion value corresponding to each point in the image based on the display area that can be seen by all eye positions when determining the mapping relationship.
[0118] Specifically, after completing image acquisition and back-projection calculations for all test eye positions, the electronic device can first determine the distribution of points belonging to the first image in the overlapping images, and extract the outer contour formed by these points to form a first region. Similarly, after determining the distribution of each point belonging to each second image, the electronic device will also form a second region corresponding to each second image by extracting the outer contour formed by the points corresponding to each second image. The first and second regions are typically represented as polygons in the image coordinate system.
[0119] Next, the electronic device performs geometric intersection calculations between the first region and each of the second regions, resulting in a series of intersection areas. These intersection areas represent the portion of the image commonly visible to the reference eye position and all test eye positions. The electronic device continues to iteratively calculate the intersection of these intersection areas, eventually obtaining a common area covered by all eye positions. However, the shape of this common area may be an irregular polygon, while the physical pixel arrangement of the display screen in the imaging device is rectangular. Therefore, it is necessary to find the largest inscribed rectangle within this common area. This largest inscribed rectangle is the area containing the final target display area.
[0120] Furthermore, in this embodiment, after the electronic device determines the largest inscribed rectangle, it can adjust the largest inscribed rectangle according to the image specifications of a preset image to obtain a target area that meets the image rules. This target area is the final target display area. The image specifications can be the aspect ratio.
[0121] In addition, since the target area is usually smaller than the original projection area, after the electronic device determines the target area, it needs to normalize the image coordinates to map each point in the overlapping image to the target area.
[0122] The electronic device can use affine transformation or perspective transformation matrices to map each point in the overlapping image to the target region. For example, the electronic device calculates the new coordinates of each point within the target region based on the position information of each point in the overlapping image, as well as the size and positional relationship of the target region.
[0123] Furthermore, in this embodiment of the application, in order to expand the display range of the projection, after determining the first region corresponding to the first image in the overlapping images and the second region corresponding to each second image, the electronic device can also determine the union region of the first region and each second region, and take the union region as the region where the final target display area is located.
[0124] Furthermore, since the shape of the union region may be an irregular polygon, while the physical pixel arrangement of the display screen in the imaging device is rectangular, it is necessary to find the largest inscribed rectangle within this union region. This largest inscribed rectangle is the area where the final target display area is located. Subsequently, the electronic device can also map each point in the overlapping image onto this target display area.
[0125] The following flowchart illustrates the process of determining the mapping relationship between the eye position information and the dynamic distortion value corresponding to each point in the image, as provided in the embodiments of this application: Figure 6 This is a flowchart illustrating the determination of the mapping relationship between the eye position information and the dynamic distortion value corresponding to each point in the image, as provided in the embodiments of this application. Figure 6 As shown, the process includes: S601: Project a preset image, marked with each point, onto a set plane using an imaging device.
[0126] S602: Determine the first image acquired at the reference eye position and the second image acquired at each test eye position.
[0127] S603: Determine the coordinates of each point in the first image and each second image according to the static calibration method.
[0128] In this embodiment of the application, in order to better determine each point in the first image and each second image, the electronic device may adopt a static calibration method to determine the coordinates corresponding to each point in the first image and each second image.
[0129] Specifically, the electronic device can input the first image and each second image into a pre-trained point extraction model to extract the marker positions in the first image and each second image, thereby obtaining the coordinates corresponding to each point in the first image and each second image. The marker position can be a bright spot in the image, or it can be a circled area, etc.
[0130] Furthermore, the electronic device assigns a number to each point, which represents the order of that point in the image, so as to facilitate the association between points from different camera positions. For example, the numbering can be determined based on the spatial relationship between points, and the method is not limited.
[0131] S604: The relative pose information between the first image acquisition device that acquires the first image and the second image acquisition device that acquires the second image, unifying each point in each second image onto the imaging plane of the first image.
[0132] In this embodiment, each second image acquisition device has 6DoF relative pose information with the first image acquisition device. The electronic device can perform coordinate system transformation based on this 6DoF relative pose information, camera intrinsic parameters, and pixel array to unify each point in all second images onto the imaging plane of the first image. The specific algorithm is not limited.
[0133] S605: Determine the target area to be displayed based on the point distribution in the first image on the imaging plane and the overlapping image corresponding to each second image.
[0134] S606: Establish a mapping relationship between N eye positions and M point offsets.
[0135] The mapping relationship can be either the offset of each point in the image corresponding to each eye position, or it can be the summarization of the eye position location information. Location information of the points and dynamic distortion value The calculation formula between them.
[0136] Based on the above embodiments, in the embodiments of this application, Figure 1 The schematic diagram of a dynamic distortion correction process also includes S102: Distortion correction is performed on the image based on the target distortion value corresponding to each point; wherein, the target distortion value includes a target dynamic distortion value and a target static distortion value; wherein, a preset image is projected onto a set plane for imaging, and the target dynamic distortion value corresponding to each point is determined based on the positional offset between points with the same number in the images acquired at each eye position and the reference eye position; the target static distortion value corresponding to each point is determined based on the positional offset between points with the same number in the image acquired at the reference eye position and the ideal image; the ideal image is the imaging of the preset image on the set plane under ideal conditions.
[0137] In this embodiment of the application, after the electronic device determines the target distortion value corresponding to each point, the electronic device will enter the final distortion correction stage.
[0138] The following describes several possible distortion correction methods: Method 1: If the target distortion value is the target dynamic distortion value, then distortion correction is performed based on the target dynamic distortion value corresponding to each point.
[0139] In this embodiment, the target distortion value corresponding to each point determined based on the mapping relationship can be the target dynamic distortion value. Specifically, the target dynamic distortion value corresponding to any point refers to the vector difference between the image position of a point after back-projection and the image position of that point at the reference eye position when observed from the current eye position. In other words, it is the positional offset of that point in the images acquired from the current eye position and the reference eye position when the preset image is projected onto a set plane.
[0140] Based on this, in the embodiments of this application, the electronic device can perform distortion correction according to the target dynamic distortion value corresponding to each point, so that the distortion-corrected image collected at any eye position in the eye box space is the same as or approximately the same as the image collected at the reference eye position.
[0141] Specifically, after the electronic device determines the target dynamic distortion value corresponding to each point in the current eye position, it generates a first correction matrix based on the dynamic distortion value corresponding to each point. The electronic device then controls the imaging device to perform coordinate transformation and texture sampling on each point of the original preset image according to the first correction matrix, correcting the pixels that were originally misaligned due to distortion to the correct physical position perceived by the human eye.
[0142] Method 2: If the target distortion value is the target dynamic distortion value and the target static distortion value, then distortion correction is performed based on the target dynamic distortion value and the target static distortion value corresponding to each point.
[0143] In addition, in practical applications, each point may also have static distortion in addition to dynamic distortion. This static distortion represents the inherent imaging deviation of the optical and electronic equipment of the imaging device.
[0144] In this embodiment, the target distortion value corresponding to each point determined based on the mapping relationship can be a target dynamic distortion value and a target static distortion value. Specifically, the target static distortion value corresponding to any point is the positional offset between the image acquired at the reference eye position and the ideal image corresponding to that point. Furthermore, the target static distortion value corresponding to each point can be a fixed value.
[0145] Based on this, in the embodiments of this application, the target distortion value corresponding to each point in the current eye position determined by the electronic device actually includes the target dynamic distortion value and the target static distortion value. The electronic device performs vector superposition or composite operation on the target dynamic distortion value and the target static distortion value corresponding to each point to obtain a second correction matrix that integrates the inherent error of the optical electronic device and the geometric error caused by the viewpoint movement.
[0146] Subsequently, the electronic device controls the imaging device to perform coordinate transformation and texture sampling on each point of the original preset image according to the second correction matrix, correcting the pixels that were originally misaligned due to distortion to their correct physical positions as perceived by the human eye. After this dual correction, the image on the windshield maintains extremely high geometric fidelity regardless of how the observer's eyeball moves within the eye box space. This not only reduces the degree of image distortion caused by parallax but also lays a precise spatial positioning foundation for subsequent augmented reality navigation and other virtual-real fusion applications, greatly improving the display effect.
[0147] To make dynamic distortion correction more closely resemble real-world scenarios, based on the above embodiments, in this application embodiment, the process of determining the reference eye position includes: Based on the current eye position information, determine the blocking height of the current eye position in the preset eye box space; Obtain the reference eye position corresponding to the saved gear height.
[0148] In traditional calibration schemes, the reference eye position is typically mechanically locked at the geometric center of the eyebox space. However, in practical applications, observers of different heights have different sitting postures, or the same observer may adjust their seat height, or different observers may have different heights, resulting in significant changes in the horizontal level of their natural line of sight. If the geometric center of the eyebox space is always used as the reference eye position, the corrected image may exhibit an overall parallax shift for tall or short observers. To address this issue, this application proposes a hierarchical reference eye position management strategy. That is, during the calibration phase, the electronic device does not establish only one mapping relationship, but instead sets different reference positions at different height levels within the eyebox space, and stores the mapping relationship between the positional information of the eye position determined based on each reference eye position and the distortion value corresponding to each point in the image.
[0149] Based on this, the electronic device, during operation, determines the height of the current eye position within a preset eye box space based on the acquired position information of the observer's current eye position, especially its vertical coordinates. Then, the electronic device can acquire a saved reference eye position pre-bound to that height. For example, when the electronic device detects that the observer's eye position is high, it automatically switches to a high reference eye position mode. In this mode, the reference eye position is no longer the center of the eye box, but a specific point in the upper half of the eye box. Based on this new reference eye position, the electronic device recalculates the dynamic distortion value of the current eye position relative to this high reference eye position. This dynamic reference switching mechanism has significant technical advantages. First, it moves the zero point of correction to near the observer's most frequently used viewpoint, making the image at this frequently used viewpoint almost require no dynamic distortion compensation, thus achieving ultimate clarity at that viewpoint. Second, for distortion caused by viewpoint movement, the calculation of the compensation amount is more linear due to the closer reference point, reducing calculation errors caused by nonlinear distortion. Through this intelligent reference switching based on height, the embodiments of this application enable the imaging device to have adaptive capabilities, greatly improving the experience for observers of different body types.
[0150] In this embodiment of the application, after determining the reference eye position corresponding to the current eye position height, the electronic device will also obtain the mapping relationship between the position information of the eye position determined based on the reference eye position height and the distortion value corresponding to each point in the image, so as to determine the target distortion value corresponding to each point in the image according to the mapping relationship corresponding to the reference eye position height.
[0151] Based on the above embodiments, this application also provides a dynamic distortion correction device. Figure 7 This application provides a schematic diagram of a dynamic distortion correction device, which includes: The acquisition module 701 is used to acquire the position information of the observer's current eye position; Processing module 702 is used to determine the target distortion value corresponding to each point in the image acquired at the current eye position based on the current eye position information and the mapping relationship between the saved eye position information and the distortion value corresponding to each point in the image. The distortion correction module 703 is used to perform distortion correction on the image based on the target distortion value corresponding to each point. The target distortion value includes the target dynamic distortion value and the target static distortion value; In this process, a preset image is projected onto a set plane for imaging, and the target dynamic distortion value corresponding to each point is determined based on the positional offset between points with the same number in the images acquired by each eye position and the reference eye position. Based on the positional offset between the image acquired from the reference eye position and the points with the same number in the ideal image, determine the target static distortion value corresponding to each point; The ideal image is the image of the preset image on the set plane under ideal conditions.
[0152] In one possible implementation, the processing module 702 is further configured to project a preset image marked with each point onto the set plane using an imaging device, and determine a first image acquired at the reference eye position and a second image acquired at multiple test eye positions; each of the multiple test eye positions is any eye position within the eye box space other than the reference eye position; for each test eye position, project each point in the target second image acquired at the test eye position onto the imaging plane where the first image is located, to obtain an overlapping image of the first image and the target second image on the imaging plane; and determine the mapping relationship between the position information of the test eye position and the distortion value corresponding to each point in the image based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, and the static distortion value corresponding to each numbered point that is saved in advance.
[0153] In one possible implementation, the processing module 702 is specifically configured to determine, based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image and the depth information of the first image, the position information of each point in the target second image on the imaging plane when projected onto the imaging plane of the first image; wherein the depth information is a fixed value; and to determine the overlapping image based on the position information of the points numbered corresponding to each other in the first image and the target second image on the imaging plane.
[0154] In one possible implementation, the processing module 702 is specifically configured to perform multiple iterations based on the stored relative pose information between the first image acquisition device acquiring the first image and the second image acquisition device acquiring the target second image, and the depth information of the first image. Each iteration includes the following process: for each numbered point, based on the relative pose information and the depth information, determining the position information of the point with that number in the target second image on the imaging plane when projected onto the imaging plane of the first image; based on the position information of the two points with that number in the first image and the target second image on the imaging plane, determining the two points corresponding to that number in the first image and the target second image on the imaging plane. The distance between points; if the current iteration does not meet the preset iteration completion condition, the depth information is adjusted according to the first distance between the corresponding points in the first image and the target second image and the second distance between the corresponding points in the first image and the target second image determined in the previous iteration, so as to determine the imaging plane where the adjusted first image is located, and the next iteration is performed; if the current iteration meets the preset iteration completion condition, the iteration ends, and the target iteration with the smallest distance between the corresponding points in the first image and the target second image is determined; the overlapping image is determined according to the position information of the corresponding points in the first image and the target second image on the adjusted imaging plane determined by the target iteration.
[0155] In one possible implementation, the processing module 702 is specifically used to determine the increase or decrease information between the first distance and the second distance; and to adjust the depth information according to the increase or decrease information and the adjustment trend of the depth information in the previous iteration.
[0156] In one possible implementation, the processing module 702 is further configured to determine a first region formed by each point belonging to the first image in the overlapping image, and a second region formed by each point belonging to each second image; determine a common region of the first region and each second region, and determine a target region within the common region that meets the image specifications according to the image specifications of the preset image, and use the target region as the final projection display region.
[0157] In one possible implementation, the processing module 702 is specifically configured to determine the distortion value corresponding to each point in the target second image based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, and the static distortion value corresponding to each numbered point that is saved in advance; and to determine the mapping relationship based on the position information of each test eye position and the distortion value of each point in the corresponding target second image.
[0158] In one possible implementation, the processing module 702 is specifically used to obtain a pre-configured fitting function, which indicates the relationship between the positional information of each test eye position and the distortion value of each point in the corresponding second image. The fitting function includes initial values for each parameter. Multiple fitting operations are performed until a preset termination condition is met, wherein each fitting operation includes the following steps: determining the predicted distortion value corresponding to each point based on the fitting function, the positional information of the test eye position, and the positional information of each point in the target second image; determining the residual distortion value corresponding to each point based on the predicted distortion value and the distortion value; determining the total residual distortion value based on the residual distortion value of each numbered point in the second image corresponding to each test eye position and the preset weight corresponding to each test eye position; and adjusting the values corresponding to the parameters of the fitting function based on the total residual distortion value.
[0159] In one possible implementation, the processing module 702 is specifically configured to: determine a first sub-mapping relationship between the position information of the test eye position and the dynamic distortion value corresponding to each point in the image, based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image; determine a second sub-mapping relationship between the position information of the test eye position and the static distortion value corresponding to each point in the image, based on the pre-saved static distortion value corresponding to each numbered point; and determine the mapping relationship based on the first sub-mapping relationship and the second sub-mapping relationship.
[0160] In one possible implementation, the processing module 702 is specifically used to determine the height of the current eye position in a preset eye box space based on the position information of the current eye position; and to obtain the reference eye position corresponding to the saved height of the current eye position.
[0161] Based on the above embodiments, this application also provides an electronic device. Figure 8 This application provides a schematic diagram of an electronic device structure, such as... Figure 8As shown, it includes: processor 801, communication interface 802, memory 803 and communication bus 804, wherein processor 801, communication interface 802 and memory 803 communicate with each other through communication bus 804. The memory 803 stores a computer program, which, when executed by the processor 801, causes the processor 801 to perform the following steps: Obtain the position information of the observer's current eye position; based on the position information of the current eye position and the mapping relationship between the saved position information of the eye position and the distortion value corresponding to each point in the image, determine the target distortion value corresponding to each point in the image acquired at the current eye position; The image is distorted based on the target distortion value corresponding to each point. The target distortion value includes the target dynamic distortion value and the target static distortion value; In this process, a preset image is projected onto a set plane for imaging, and the target dynamic distortion value corresponding to each point is determined based on the positional offset between points with the same number in the images acquired by each eye position and the reference eye position. Based on the positional offset between the image acquired from the reference eye position and the points with the same number in the ideal image, determine the target static distortion value corresponding to each point; The ideal image is the image of the preset image on the set plane under ideal conditions.
[0162] In one possible implementation, the process of determining the mapping relationship between the stored eye position information and the distortion value corresponding to each point in the image includes: An imaging device projects a preset image marked with each point onto the set plane to determine a first image acquired at the reference eye position and a second image acquired at multiple test eye positions; each of the multiple test eye positions is any eye position within the eye box space other than the reference eye position. For each test eye position, each point in the target second image acquired at that test eye position is projected onto the imaging plane of the first image to obtain an overlapping image of the first image and the target second image on the imaging plane; based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, and the pre-saved static distortion value corresponding to each numbered point, the mapping relationship between the position information of the test eye position and the distortion value corresponding to each point in the image is determined.
[0163] In one possible implementation, obtaining the overlapping image of the first image and the second image on the imaging plane includes: Based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image, and the depth information of the first image, the position information of each point in the target second image on the imaging plane when projected onto the imaging plane of the first image is determined; wherein, the depth information is a fixed value; The overlapping image is determined based on the position information of the points whose numbers correspond to each other in the first image and the target second image on the imaging plane.
[0164] In one possible implementation, obtaining the overlapping image of the first image and the target second image on the imaging plane includes: Based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image, and the depth information of the first image, multiple iterations are performed, wherein each iteration includes the following process: For each numbered point, based on the relative pose information and the depth information, determine the position information of the point with that number in the second target image on the imaging plane when each point in the second target image is projected onto the imaging plane of the first image; based on the position information of the two points with that number in the first image and the second target image on the imaging plane, determine the distance between the two points with that number in the first image and the second target image on the imaging plane; If the current iteration does not meet the preset iteration completion conditions, the depth information is adjusted based on the first distance between the corresponding points in the first image and the target second image and the second distance between the corresponding points in the first image and the target second image determined in the previous iteration, so as to determine the imaging plane where the adjusted first image is located, and then proceed to the next iteration. If the current iteration meets the preset iteration completion conditions, the iteration ends, and the target iteration with the smallest distance between the corresponding points in the first image and the target second image is determined among all iterations; the overlapping image is determined based on the position information of the corresponding points in the first image and the target second image on the adjusted imaging plane determined by the target iteration.
[0165] In one possible implementation, adjusting the depth information includes: Determine the increase or decrease information between the first distance and the second distance; The depth information is adjusted based on the increase / decrease information and the adjustment trend of the depth information in the previous iteration.
[0166] In one possible implementation, after obtaining the overlapping image of the first image and the target second image on the imaging plane, the method further includes: Determine a first region in the overlapping image that is formed by each point belonging to the first image, and a second region that is formed by each point belonging to each second image; The common area of the first region and each of the second regions is determined, and a target area within the common area that meets the image specifications of the preset image is determined according to the image specifications of the preset image, and the target area is used as the final projection display area.
[0167] In one possible implementation, determining the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image includes: Based on the different position information of the points in the target second image and the first image that correspond to each other in the overlapping image, and the static distortion value corresponding to each numbered point that is saved in advance, the distortion value corresponding to each point in the target second image is determined. The mapping relationship is determined based on the position information of each test eye position and the distortion value of each point in the corresponding target second image.
[0168] In one possible implementation, determining the mapping relationship based on the position information of each test eye position and the distortion value of each point in the corresponding target second image includes: Obtain a pre-configured fitting function, which is used to indicate the relationship between the positional information of each test eye position and the distortion value of each point in the corresponding second image. The fitting function includes the initial value corresponding to each parameter. Perform multiple fitting operations until the preset termination condition is met. Each fitting operation includes the following steps: Based on the fitting function, the position information of the test eye position, and the position information of each point in the target second image, the predicted distortion value corresponding to each point is determined; based on the predicted distortion value and the distortion value corresponding to each point, the residual distortion value corresponding to each point is determined. The total residual distortion value is determined based on the residual distortion value of each numbered point in the second image corresponding to each test eye position, and the preset weight corresponding to each test eye position. The values of the parameters of the fitting function are adjusted based on the total residual distortion value.
[0169] In one possible implementation, determining the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image includes: Based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, a first sub-mapping relationship is determined between the position information of the test eye position and the dynamic distortion value corresponding to each point in the image. Based on the pre-saved static distortion value corresponding to each numbered point, a second sub-mapping relationship is determined between the position information of the test eye and the static distortion value corresponding to each point in the image; The mapping relationship is determined based on the first sub-mapping relationship and the second sub-mapping relationship.
[0170] In one possible implementation, the process of determining the reference eye position includes: Based on the current eye position information, determine the blocking height of the current eye position in the preset eye box space; Obtain the reference eye position corresponding to the saved gear height.
[0171] Since the principle of the above-mentioned electronic device in solving the problem is similar to that of the dynamic distortion correction method, the implementation of the above-mentioned electronic device can be found in the embodiments of the method, and repeated parts will not be described again.
[0172] The communication bus mentioned in the aforementioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 802 is used for communication between the aforementioned electronic device and other devices. The memory can include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.
[0173] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0174] Based on the above embodiments, this invention also provides a computer-readable storage medium storing a computer program executable by a processor. When the program is run on the processor, it causes the processor to perform the following steps: Obtain the position information of the observer's current eye position; based on the position information of the current eye position and the mapping relationship between the saved position information of the eye position and the distortion value corresponding to each point in the image, determine the target distortion value corresponding to each point in the image acquired at the current eye position; The image is distorted based on the target distortion value corresponding to each point. The target distortion value includes the target dynamic distortion value and the target static distortion value; In this process, a preset image is projected onto a set plane for imaging, and the target dynamic distortion value corresponding to each point is determined based on the positional offset between points with the same number in the images acquired by each eye position and the reference eye position. Based on the positional offset between the image acquired from the reference eye position and the points with the same number in the ideal image, determine the target static distortion value corresponding to each point; The ideal image is the image of the preset image on the set plane under ideal conditions.
[0175] In one possible implementation, the process of determining the mapping relationship between the stored eye position information and the distortion value corresponding to each point in the image includes: An imaging device projects a preset image marked with each point onto the set plane to determine a first image acquired at the reference eye position and a second image acquired at multiple test eye positions; each of the multiple test eye positions is any eye position within the eye box space other than the reference eye position. For each test eye position, each point in the target second image acquired at that test eye position is projected onto the imaging plane of the first image to obtain an overlapping image of the first image and the target second image on the imaging plane; based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, and the pre-saved static distortion value corresponding to each numbered point, the mapping relationship between the position information of the test eye position and the distortion value corresponding to each point in the image is determined.
[0176] In one possible implementation, obtaining the overlapping image of the first image and the second image on the imaging plane includes: Based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image, and the depth information of the first image, the position information of each point in the target second image on the imaging plane when projected onto the imaging plane of the first image is determined; wherein, the depth information is a fixed value; The overlapping image is determined based on the position information of the points whose numbers correspond to each other in the first image and the target second image on the imaging plane.
[0177] In one possible implementation, obtaining the overlapping image of the first image and the target second image on the imaging plane includes: Based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image, and the depth information of the first image, multiple iterations are performed, wherein each iteration includes the following process: For each numbered point, based on the relative pose information and the depth information, determine the position information of the point with that number in the second target image on the imaging plane when each point in the second target image is projected onto the imaging plane of the first image; based on the position information of the two points with that number in the first image and the second target image on the imaging plane, determine the distance between the two points with that number in the first image and the second target image on the imaging plane; If the current iteration does not meet the preset iteration completion conditions, the depth information is adjusted based on the first distance between the corresponding points in the first image and the target second image and the second distance between the corresponding points in the first image and the target second image determined in the previous iteration, so as to determine the imaging plane where the adjusted first image is located, and then proceed to the next iteration. If the current iteration meets the preset iteration completion conditions, the iteration ends, and the target iteration with the smallest distance between the corresponding points in the first image and the target second image is determined among all iterations; the overlapping image is determined based on the position information of the corresponding points in the first image and the target second image on the adjusted imaging plane determined by the target iteration.
[0178] In one possible implementation, adjusting the depth information includes: Determine the increase or decrease information between the first distance and the second distance; The depth information is adjusted based on the increase / decrease information and the adjustment trend of the depth information in the previous iteration.
[0179] In one possible implementation, after obtaining the overlapping image of the first image and the target second image on the imaging plane, the method further includes: Determine a first region in the overlapping image that is formed by each point belonging to the first image, and a second region that is formed by each point belonging to each second image; The common area of the first region and each of the second regions is determined, and a target area within the common area that meets the image specifications of the preset image is determined according to the image specifications of the preset image, and the target area is used as the final projection display area.
[0180] In one possible implementation, determining the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image includes: Based on the different position information of the points in the target second image and the first image that correspond to each other in the overlapping image, and the static distortion value corresponding to each numbered point that is saved in advance, the distortion value corresponding to each point in the target second image is determined. The mapping relationship is determined based on the position information of each test eye position and the distortion value of each point in the corresponding target second image.
[0181] In one possible implementation, determining the mapping relationship based on the position information of each test eye position and the distortion value of each point in the corresponding target second image includes: Obtain a pre-configured fitting function, which is used to indicate the relationship between the positional information of each test eye position and the distortion value of each point in the corresponding second image. The fitting function includes the initial value corresponding to each parameter. Perform multiple fitting operations until the preset termination condition is met. Each fitting operation includes the following steps: Based on the fitting function, the position information of the test eye position, and the position information of each point in the target second image, the predicted distortion value corresponding to each point is determined; based on the predicted distortion value and the distortion value corresponding to each point, the residual distortion value corresponding to each point is determined. The total residual distortion value is determined based on the residual distortion value of each numbered point in the second image corresponding to each test eye position, and the preset weight corresponding to each test eye position. The values of the parameters of the fitting function are adjusted based on the total residual distortion value.
[0182] In one possible implementation, determining the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image includes: Based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, a first sub-mapping relationship is determined between the position information of the test eye position and the dynamic distortion value corresponding to each point in the image. Based on the pre-saved static distortion value corresponding to each numbered point, a second sub-mapping relationship is determined between the position information of the test eye and the static distortion value corresponding to each point in the image; The mapping relationship is determined based on the first sub-mapping relationship and the second sub-mapping relationship.
[0183] In one possible implementation, the process of determining the reference eye position includes: Based on the current eye position information, determine the blocking height of the current eye position in the preset eye box space; Obtain the reference eye position corresponding to the saved gear height.
[0184] Since the principle of the computer-readable storage medium in solving the problem is similar to that of the dynamic distortion correction method, the implementation of the computer-readable storage medium can be found in the embodiments of the method, and repeated details will not be repeated.
[0185] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0186] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0187] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0188] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0189] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A dynamic distortion correction method, characterized in that, The method includes: Obtain the position information of the observer's current eye position; based on the position information of the current eye position and the mapping relationship between the saved position information of the eye position and the distortion value corresponding to each point in the image, determine the target distortion value corresponding to each point in the image acquired at the current eye position; The image is distorted based on the target distortion value corresponding to each point. The target distortion value includes the target dynamic distortion value and the target static distortion value; In this process, a preset image is projected onto a set plane for imaging, and the target dynamic distortion value corresponding to each point is determined based on the positional offset between points with the same number in the images acquired by each eye position and the reference eye position. Based on the positional offset between the image acquired from the reference eye position and the points with the same number in the ideal image, determine the target static distortion value corresponding to each point; The ideal image is the image of the preset image on the set plane under ideal conditions; The process of determining the mapping relationship between the saved eye position information and the distortion value corresponding to each point in the image includes: An imaging device projects a preset image marked with each point onto the set plane to determine a first image acquired at the reference eye position and a second image acquired at multiple test eye positions; each of the multiple test eye positions is any eye position within the eye box space other than the reference eye position. For each test eye position, each point in the target second image acquired at that test eye position is projected onto the imaging plane of the first image to obtain an overlapping image of the first image and the target second image on the imaging plane; based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, and the pre-saved static distortion value corresponding to each numbered point, the mapping relationship between the position information of the test eye position and the distortion value corresponding to each point in the image is determined.
2. The method according to claim 1, characterized in that, Obtaining the overlapping image of the first image and the second image on the imaging plane includes: Based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image, and the depth information of the first image, the position information of each point in the target second image on the imaging plane when projected onto the imaging plane of the first image is determined; wherein, the depth information is a fixed value; The overlapping image is determined based on the position information of the points whose numbers correspond to each other in the first image and the target second image on the imaging plane.
3. The method according to claim 1, characterized in that, The process of obtaining the overlapping image of the first image and the target second image on the imaging plane includes: Based on the relative pose information between the first image acquisition device that acquired the first image and the second image acquisition device that acquired the target second image, and the depth information of the first image, multiple iterations are performed, wherein each iteration includes the following process: For each numbered point, based on the relative pose information and the depth information, determine the position information of the point with that number in the second target image on the imaging plane when each point in the second target image is projected onto the imaging plane of the first image; based on the position information of the two points with that number in the first image and the second target image on the imaging plane, determine the distance between the two points with that number in the first image and the second target image on the imaging plane; If the current iteration does not meet the preset iteration completion conditions, the depth information is adjusted based on the first distance between the corresponding points in the first image and the target second image and the second distance between the corresponding points in the first image and the target second image determined in the previous iteration, so as to determine the imaging plane where the adjusted first image is located, and then proceed to the next iteration. If the current iteration meets the preset iteration completion conditions, the iteration ends, and the target iteration with the smallest distance between the corresponding points in the first image and the target second image is determined among all iterations; the overlapping image is determined based on the position information of the corresponding points in the first image and the target second image on the adjusted imaging plane determined by the target iteration.
4. The method according to claim 3, characterized in that, The adjustment of the depth information includes: Determine the increase or decrease information between the first distance and the second distance; The depth information is adjusted based on the increase / decrease information and the adjustment trend of the depth information in the previous iteration.
5. The method according to claim 1, characterized in that, After obtaining the overlapping image of the first image and the target second image on the imaging plane, the method further includes: Determine a first region in the overlapping image that is formed by each point belonging to the first image, and a second region that is formed by each point belonging to each second image; The common area of the first region and each of the second regions is determined, and a target area within the common area that meets the image specifications of the preset image is determined according to the image specifications of the preset image, and the target area is used as the final projection display area.
6. The method according to claim 1, characterized in that, The process of determining the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image includes: Based on the different position information of the points in the target second image and the first image that correspond to each other in the overlapping image, and the static distortion value corresponding to each numbered point that is saved in advance, the distortion value corresponding to each point in the target second image is determined. The mapping relationship is determined based on the position information of each test eye position and the distortion value of each point in the corresponding target second image.
7. The method according to claim 6, characterized in that, The step of determining the mapping relationship based on the position information of each test eye position and the distortion value of each point in the corresponding target second image includes: Obtain a pre-configured fitting function, which is used to indicate the relationship between the positional information of each test eye position and the distortion value of each point in the corresponding second image. The fitting function includes the initial value corresponding to each parameter. Perform multiple fitting operations until the preset termination condition is met. Each fitting operation includes the following steps: Based on the fitting function, the position information of the test eye position, and the position information of each point in the target second image, the predicted distortion value corresponding to each point is determined; based on the predicted distortion value and the distortion value corresponding to each point, the residual distortion value corresponding to each point is determined. The total residual distortion value is determined based on the residual distortion value of each numbered point in the second image corresponding to each test eye position, and the preset weight corresponding to each test eye position. The values of the parameters of the fitting function are adjusted based on the total residual distortion value.
8. The method according to claim 1, characterized in that, The process of determining the mapping relationship between the positional information of the test eye and the distortion value corresponding to each point in the image includes: Based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, a first sub-mapping relationship is determined between the position information of the test eye position and the dynamic distortion value corresponding to each point in the image. Based on the pre-saved static distortion value corresponding to each numbered point, a second sub-mapping relationship is determined between the position information of the test eye and the static distortion value corresponding to each point in the image; The mapping relationship is determined based on the first sub-mapping relationship and the second sub-mapping relationship.
9. The method according to any one of claims 1-8, characterized in that, The process of determining the reference eye position includes: Based on the current eye position information, determine the blocking height of the current eye position in the preset eye box space; Obtain the reference eye position corresponding to the saved gear height.
10. A dynamic distortion correction device, characterized in that, The device includes: The acquisition module is used to acquire the position information of the observer's current eye position; The processing module is used to determine the target distortion value corresponding to each point in the image acquired at the current eye position based on the current eye position information and the mapping relationship between the saved eye position information and the distortion value corresponding to each point in the image. The distortion correction module is used to correct the distortion of the image based on the target distortion value corresponding to each point. The target distortion value includes the target dynamic distortion value and the target static distortion value; In this process, a preset image is projected onto a set plane for imaging, and the target dynamic distortion value corresponding to each point is determined based on the positional offset between points with the same number in the images acquired by each eye position and the reference eye position. Based on the positional offset between the image acquired from the reference eye position and the points with the same number in the ideal image, determine the target static distortion value corresponding to each point; The ideal image is the image of the preset image on the set plane under ideal conditions; The processing module is further configured to project a preset image marked with each point onto the set plane using an imaging device, and determine the first image acquired at the reference eye position and the second image acquired at multiple test eye positions; each of the multiple test eye positions is any eye position within the eye box space other than the reference eye position; for each test eye position, each point in the target second image acquired at the test eye position is projected onto the imaging plane where the first image is located to obtain an overlapping image of the first image and the target second image on the imaging plane; based on the different position information of the points numbered corresponding to each other in the target second image and the first image on the overlapping image, and the pre-saved static distortion value corresponding to each numbered point, the mapping relationship between the position information of the test eye position and the distortion value corresponding to each point in the image is determined.
11. An electronic device, characterized in that, The electronic device includes a processor that executes a computer program stored in a memory to implement the steps of the dynamic distortion correction method as described in any one of claims 1-9.
12. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the dynamic distortion correction method as described in any one of claims 1-9.
Citation Information
Patent Citations
AR-HUD image distortion correction method, device and system under dynamic eye position
CN111476104A
