A head-mounted system

Abstract

The application discloses a filtering stabilization algorithm based on a head-mounted display, and relates to the technical field of head-mounted displays; the algorithm is realized based on a head-mounted display system, and the system comprises the following modules: a data acquisition module, which acquires original image data from a high-resolution image sensor built in a head-mounted display device, and synchronously acquires data of a head posture sensor to acquire posture information of the head-mounted display device in a three-dimensional space; and a preprocessing module, which adopts an adaptive median filtering algorithm to perform denoising processing on the original image data.The application constructs accurate rotation and translation matrices, accurately transforms from a world coordinate system to a head-mounted display device coordinate system through reasonable matrix operation, guarantees correct mapping of an image in a spatial position, performs image distortion correction by using an efficient bilinear interpolation algorithm, ensures that pixel values in new coordinates can be quickly and accurately calculated when image deformation caused by head shaking is processed, effectively reduces image blur and distortion, and improves the definition and accuracy of the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of head-mounted display, and in particular to a filtering stabilization algorithm based on head-mounted display. BACKGROUND

[0002] With the rapid development of immersive technologies such as VR and AR, head-mounted display devices have been widely used in various fields such as gaming, education, medical treatment, and industry. When using head-mounted display devices, the natural movements of the head, such as rotation and tilting, can cause the images generated by the device to shake and blur, which not only reduces the user's visual experience, but also can cause dizziness, fatigue, and other problems, seriously affecting the use effect and user experience of head-mounted display devices. Therefore, how to effectively solve the image jitter problem caused by head shaking of head-mounted display devices has become a technical problem to be solved in this field.

[0003] Some existing head-mounted display devices use built-in inertial sensors such as gyroscopes and accelerometers to monitor the movement state of the head and attempt to stabilize the image. However, these methods have many limitations. On the one hand, some methods only use simple filtering algorithms to process sensor data, which is difficult to accurately reflect the complex movement of the head, resulting in poor image stabilization effect and still obvious jitter phenomenon. On the other hand, although some complex stabilization algorithms improve stability to some extent, they have high computational complexity and require high performance of hardware devices, which is difficult to meet the real-time requirements and may introduce additional delay, affecting the user's interactive experience. In addition, traditional image stabilization methods mostly do not fully consider the characteristics of user head movement in different scenarios and the characteristics of different types of image content, lack of pertinence and adaptability, and cannot provide high-quality visual experience. SUMMARY

[0004] The purpose of the present application is to solve the problems existing in the prior art and propose a filtering stabilization algorithm based on head-mounted display.

[0005] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0006] A filtering stabilization algorithm based on head-mounted display is realized based on a head-mounted display system, which includes:

[0007] A data acquisition module acquires raw image data from a high-resolution image sensor built into the head-mounted display device and synchronously acquires data from a head posture sensor to obtain posture information of the head-mounted display device in three-dimensional space;

[0008] The preprocessing module adopts an adaptive median filtering algorithm to perform denoising processing on the original image data to suppress random noise generated in the image acquisition process; the window size of the adaptive median filtering is dynamically adjusted according to the local statistical characteristics of the image; the Kalman filtering algorithm is used to filter the Euler angle data collected by the head posture sensor to remove high-frequency noise and errors of the posture data;

[0009] The image stabilization module constructs a rotation transformation matrix from the world coordinate system to the head-mounted device coordinate system according to the filtered head posture data; and performs distortion correction on the denoised image data by using the rotation transformation matrix to compensate for the image deformation caused by head shaking;

[0010] The post-processing module performs enhancement processing on the image after the stabilization processing; the histogram equalization algorithm is adopted to adjust the contrast of the image; and the enhanced image is fused with the original image to improve the detail information and stability of the image;

[0011] The control and optimization module dynamically adjusts the parameters of the filtering algorithm according to the image stabilization effect and system performance indicators;

[0012] The data storage and transmission module transmits the processed image data to the head-mounted device for display.

[0013] Preferably, the data acquisition module comprises:

[0014] The image sensor data acquisition submodule is responsible for acquiring original image data from the high-resolution image sensor built-in the head-mounted device; the resolution of the image sensor is MxN, and the original image data collected at time t is represented as a two-dimensional matrix , wherein represents the pixel value collected by the image sensor at position (i, j) at time t, i∈{1, 2, …, M}, j∈{1, 2, …, N};

[0015] The head posture sensor data acquisition submodule synchronously collects data of the head posture sensor to obtain the posture information of the head-mounted device in the three-dimensional space; the head posture is described by three Euler angles: pitch angle α(t), yaw angle β(t), and roll angle γ(t); the Euler angle data output by the posture sensor at time t is represented as a vector , wherein , , .

[0016] Preferably, the preprocessing module comprises:

[0017] The image denoising submodule employs an adaptive median filtering algorithm to denoise the original image data Iraw(t). For each pixel (i,j) in the image, the median of the pixel values ​​within its k×k neighborhood is used as the denoised pixel value. Let the denoised image data be... The denoising process can be expressed as:

[0018]

[0019] in, This represents the midpoint operation, and Z represents the set of integers.

[0020] Preferably, the preprocessing module further includes:

[0021] The attitude data filtering submodule uses the Kalman filter algorithm to filter the Euler angle data E(t) acquired by the head attitude sensor. The state equation and observation equation of the Kalman filter algorithm are defined as follows:

[0022]

[0023] in, Let A be the system state vector, including Euler angles and their angular velocities; let B be the state transition matrix, describing the relationship between the system states at different times; let C be the control input matrix; let U(t) be the control input vector; let W(t) be the process noise vector, assumed to follow a Gaussian distribution with mean 0 and covariance Q; let Z(t) be the observation vector, i.e., the Euler angle data actually measured by the sensors; let C be the observation matrix; and let V(t) be the observation noise vector, assumed to follow a Gaussian distribution with mean 0 and covariance R. The Kalman filter algorithm iteratively calculates through two steps: prediction and update, to obtain the optimal state estimate. Then, the filtered Euler angle data is extracted. .

[0024] Preferably, the image stabilization module includes:

[0025] The coordinate transformation submodule constructs the rotation transformation matrix R(t). First, it converts the Euler angles into rotation matrix form, assuming rotations about the x-axis, y-axis, and z-axis. , , The corresponding rotation matrices are respectively , , Then we have:

[0026]

[0027]

[0028]

[0029] The total rotation transformation matrix R(t) is: .

[0030] Preferably, the image stabilization module further comprises:

[0031] an image warping correction sub-module, which warps the denoised image data using the rotation transformation matrix R(t) to obtain a corrected image, and for each pixel point (i, j) in the image, the new coordinates (i', j') of the pixel point in the corrected image are calculated by the following formula:

[0032]

[0033] wherein R(t) is the rotation transformation matrix , and R(t)-1 is the inverse matrix of R(t); since the coordinate transformation can cause the pixel points to be mapped to non-integer positions, a bilinear interpolation algorithm is used to calculate the pixel values of the corrected image at integer coordinates; let the corrected image data be , then for any integer coordinates (m, n), the following formula is satisfied:

[0034]

[0035] wherein , are the coordinates of the four nearest pixel points to (m, n), , .

[0036] Preferably, the post-processing module comprises:

[0037] an image enhancement sub-module, which adjusts the contrast of the image using a histogram equalization algorithm to make the gray level distribution of the image more uniform; let the gray level histogram of the image be represented as , then the cumulative distribution function (CDF) is:

[0038] ;

[0039]

[0040] wherein L is the number of gray levels of the image; the pixel value of the image after histogram equalization is calculated by the following formula:

[0041]

[0042] wherein represents a floor operation.

[0043] Preferably, the post-processing module further comprises:​​

[0044] The image fusion sub-module adopts a weighted fusion algorithm to adaptively determine a fusion weight according to the stability degree and the detail features of the image. , and ; the final image data after fusion is represented as: .

[0045] Preferably, the control and optimization module comprises:

[0046] The parameter adjustment sub-module determines the optimal parameter value by minimizing the mean square error or the structural similarity index evaluation index, and the mean square error function is defined as:

[0047] ;

[0048] wherein, MSE is the ideal non-shaking image data; the parameter combination that minimizes the MSE is found through an iterative search algorithm.

[0049] Preferably, the head-mounted system further comprises a display compensation module in the anti-shake mode, and the display compensation module in the anti-shake mode comprises:

[0050] The motion vector calculation sub-module calculates the motion vector of the display picture according to the data change of the head posture sensor within the adjacent two time intervals after the anti-shake mode is triggered.

[0051] The compensation strategy determination sub-module determines the appropriate compensation strategy according to the size and direction of the motion vector; the compensation strategy comprises: setting different compensation weight coefficients according to the module length of the motion vector, and the longer the module length is, the larger the weight coefficient is.

[0052] The present application has the following advantages:

[0053] 1. The present application constructs accurate rotation and translation matrices, realizes accurate transformation from the world coordinate system to the head-mounted device coordinate system through reasonable matrix operation, ensures correct mapping of the image in the spatial position, and uses an efficient bilinear interpolation algorithm to correct the image distortion, so that the pixel value in the new coordinate can be quickly and accurately calculated when processing the image distortion caused by head shaking, the image blur and distortion are effectively reduced, and the clarity and accuracy of the image are improved.

[0054] 2. The present application integrates multiple high-precision sensors, can accurately collect the rotation and translation information of the head in real time, provides a reliable data basis for subsequent image stabilization processing, uses data filtering preprocessing technology to effectively remove sensor noise interference, ensures that the obtained head posture data is more accurate, and makes the system more sensitive to capture the subtle movement of the head.​​

[0055] 3. The unique anti-shake mode triggering mechanism of the present application can start the corresponding compensation strategy in time according to the severity of head movement, effectively cope with various complex head shaking conditions, and maintain the stable display of the image. The present application includes compensation weight adjustment and direction priority setting based on motion vector, can flexibly adjust the compensation amplitude and direction according to different motion conditions, so that the shaking of the display picture in different directions can be compensated, and the stability and visual effect of the image are greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 A processing flowchart of a filtering stabilization algorithm based on a head-mounted device is proposed in the present application. DETAILED DESCRIPTION

[0057] The technical solutions of the present application will be further described in detail below in combination with specific embodiments.

[0058] Embodiment 1:

[0059] A filtering stabilization algorithm based on a head-mounted device is realized based on a head-mounted system, and the system comprises:

[0060] A data acquisition module acquires original image data from a high-resolution image sensor built in the head-mounted device, and synchronously acquires data of a head posture sensor to obtain posture information of the head-mounted device in a three-dimensional space;

[0061] A preprocessing module adopts an adaptive median filtering algorithm to perform denoising processing on the original image data to suppress random noise generated in the image acquisition process; the window size of the adaptive median filtering is dynamically adjusted according to the local statistical characteristics of the image; and a Kalman filtering algorithm is used to filter the Euler angle data acquired by the head posture sensor to remove high-frequency noise and errors of the posture data, and improve the accuracy and stability of the posture estimation;

[0062] An image stabilization module constructs a rotation transformation matrix from a world coordinate system to a coordinate system of the head-mounted device according to the filtered head posture data; and uses the rotation transformation matrix to perform distortion correction on the denoised image data to compensate for the image deformation caused by head shaking;

[0063] A post-processing module performs enhancement processing on the image after the stabilization processing; adopts a histogram equalization algorithm to adjust the contrast of the image; and performs fusion processing on the enhanced image and the original image to improve the detail information and stability of the image;

[0064] A control and optimization module dynamically adjusts the parameters of the filtering algorithm according to the image stabilization effect and system performance indicators;

[0065] The data storage and transmission module transmits the processed image data to the head-mounted device for display; a high-speed data transmission interface is adopted to ensure the real-time and stability of data transmission.

[0066] The data acquisition module includes:

[0067] The image sensor data acquisition sub-module is responsible for acquiring raw image data from the high-resolution image sensor built-in the head-mounted device; the resolution of the image sensor is set as M x N, and the raw image data acquired at time t is represented as a two-dimensional matrix wherein represents the pixel value acquired by the image sensor at position (i, j) at time t, i ∈ {1, 2, …, M}, j ∈ {1, 2, …, N};

[0068] The head pose sensor data acquisition sub-module synchronously acquires data of the head pose sensor to obtain the pose information of the head-mounted device in the three-dimensional space; the head pose is described by three Euler angles: pitch angle α(t), yaw angle β(t), and roll angle γ(t); the Euler angle data output by the pose sensor at time t is set as vector wherein , , .

[0069] The preprocessing module includes:

[0070] The image denoising sub-module adopts an adaptive median filtering algorithm to perform denoising processing on the raw image data Iraw(t); for each pixel point (i, j) in the image, the median value of the pixel values in the k x k neighborhood centered on the pixel point is taken as the pixel value of the pixel point after denoising; the image data after denoising is set as The denoising process is represented as:

[0071]

[0072] wherein, represents the median value operation, and Z represents the set of integers;

[0073] The pose data filtering sub-module applies a Kalman filtering algorithm to filter the Euler angle data E(t) acquired by the head pose sensor; the state equation and the observation equation of the Kalman filtering algorithm are defined as:

[0074]

[0075] wherein, is the system state vector, including Euler angles and their angular velocities; A is the state transition matrix, describing the relationship between the system state at different times; B is the control input matrix; U(t) is the control input vector; W(t) is the process noise vector, which is assumed to follow a Gaussian distribution with a mean of 0 and a covariance of Q; Z(t) is the observation vector, i.e., the actual measurement of the Euler angle data by the sensor; C is the observation matrix; V(t) is the observation noise vector, which is assumed to follow a Gaussian distribution with a mean of 0 and a covariance of R. The Kalman filtering algorithm iteratively calculates the optimal state estimation value through the prediction and update steps , and then extracts the filtered Euler angle data .

[0076] The image stabilization module includes:

[0077] A coordinate transformation submodule constructs a rotation transformation matrix R(t). First, the Euler angles are converted into a rotation matrix form, and it is assumed that the rotation angles around the x-axis, y-axis, and z-axis are 、 、 The corresponding rotation matrices are , , Then we have:

[0078]

[0079]

[0080]

[0081] The total rotation transformation matrix R(t) is: ;

[0082] An image warping correction submodule uses the rotation transformation matrix R(t) to perform warping correction on the denoised image data . For each pixel point (i, j) in the image, the new coordinates (i', j') in the corrected image are calculated by the following formula:

[0083]

[0084] where is the inverse matrix of the rotation transformation matrix . Since coordinate transformation may cause pixel points to be mapped to non-integer positions, a bilinear interpolation algorithm is used to calculate the pixel values of the corrected image at integer coordinates. Let the corrected image data be . Then for any integer coordinates (m, n), we have:

[0085]

[0086] wherein , are the coordinates of the four nearest pixels to (m, n), , .

[0087] wherein the post-processing module comprises:

[0088] an image enhancement submodule that adjusts the contrast of the image using a histogram equalization algorithm to make the gray level distribution of the image more uniform; let the gray level histogram of the image be denoted as , then the cumulative distribution function (CDF) is:

[0089]

[0090]

[0091] wherein L is the number of gray levels of the image; the pixel value of the image after histogram equalization is calculated by the following formula:

[0092]

[0093] wherein denotes a floor operation;

[0094] an image fusion submodule that uses a weighted fusion algorithm to adaptively determine the fusion weight according to the stability degree and the detail features of the image; let the fusion weight coefficient be denoted as , and . Then the final image data after fusion is denoted as: .

[0095] wherein the control and optimization module comprises:

[0096] a parameter adjustment submodule that determines the optimal parameter value by minimizing an evaluation index such as the mean square error or the structural similarity index; let the mean square error function be defined as:

[0097] ;

[0098] wherein is the ideal non-shaking image data; the parameter combination that minimizes the MSE is found by an iterative search algorithm.

[0099] Embodiment 2:

[0100] A filtering stabilization algorithm based on a head-mounted display, in this embodiment, on the basis of Embodiment 1, the head-mounted display system further comprises: a display compensation module in a shake-proof mode, the display compensation module in the shake-proof mode comprising:

[0101] A motion vector calculation sub-module calculates the motion vector of the display picture according to the data changes of the head pose sensor in the adjacent two time intervals after the anti-shake mode is triggered. Assuming that the rotation angle change of the head in the time interval Δt is Δα, Δβ, Δγ, and the translation change is Δx, Δy, Δz, the motion vector V is represented as:

[0102]

[0103] The motion vector is used to determine the compensation direction and amplitude of the display picture subsequently.

[0104] A compensation strategy determination sub-module determines the appropriate compensation strategy according to the size and direction of the motion vector. For example, when the component of the motion vector in a certain direction is larger, larger compensation is performed in the direction; when the motion vector is smaller, smaller compensation or no compensation is performed. The specific compensation strategy can be adjusted and optimized through experiments and experience.

[0105] The compensation strategy includes setting different compensation weight coefficients according to the module length of the motion vector. The larger the module length, the larger the weight coefficient. For example, the compensation weight coefficient W is defined as wherein is a monotonically increasing function (such as a linear function, an exponential function, etc.), and When W < 1, no compensation is performed; when W > 1, maximum compensation is performed.

[0106] In addition, different compensation direction priorities can also be considered according to the direction of the motion vector. For example, the horizontal direction motion which is more sensitive to the human eye is given higher priority and larger compensation amplitude.

[0107] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can make equivalent replacements or changes to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.​​

Claims

1. A head-mounted system, characterized by The system comprises: a data acquisition module, which acquires original image data from a high-resolution image sensor built in a head-mounted device and synchronously acquires data of a head posture sensor to obtain posture information of the head-mounted device in a three-dimensional space; a preprocessing module, which performs denoising processing on the original image data by using an adaptive median filtering algorithm to suppress random noise generated in the image acquisition process; a window size of the adaptive median filtering is dynamically adjusted according to local statistical characteristics of the image; and a Kalman filtering algorithm is used to filter Euler angle data collected by the head posture sensor to remove high-frequency noise and errors of the posture data; an image stabilization module, which constructs a rotation transformation matrix from a world coordinate system to a coordinate system of the head-mounted device according to the filtered head posture data; and uses the rotation transformation matrix to perform distortion correction on the denoised image data to compensate for image deformation caused by head shaking; a post-processing module, which performs enhancement processing on the image after the stabilization processing; adjusts contrast of the image by using a histogram equalization algorithm; and performs fusion processing on the enhanced image and the original image to improve detail information and stability of the image; a control and optimization module, which dynamically adjusts parameters of the filtering algorithm according to an image stabilization effect and system performance indicators; a data storage and transmission module, which transmits the processed image data to the head-mounted device for display; the preprocessing module comprises: The image denoising sub-module adopts an adaptive median filter algorithm to perform denoising processing on the original image data Iraw(t). For each pixel point (i, j) in the image, the median value of the pixel values in the k x k neighborhood centered on the pixel point is taken as the pixel value of the pixel point after denoising. Let the image data after denoising be Iden(t). The denoising process is represented as: ； wherein denotes the median operation, Z denotes a set of integers; the preprocessing module further comprises: a posture data filtering submodule, which uses a Kalman filtering algorithm to filter Euler angle data E(t) collected by the head posture sensor, and state equations and observation equations of the Kalman filtering algorithm are defined as: ； wherein, is the system state vector, including Euler angles and their angular velocities; A is the state transition matrix, describing the relationship between the system states at different times; B is the control input matrix; U(t) is the control input vector; W(t) is the process noise vector, which is assumed to follow a Gaussian distribution with a mean of 0 and a covariance of Q; Z(t) is the observation vector, i.e., the actual measured Euler angle data by the sensor; C is the observation matrix; V(t) is the observation noise vector, which is assumed to follow a Gaussian distribution with a mean of 0 and a covariance of R; the Kalman filtering algorithm is iteratively calculated through the prediction and update steps to obtain the optimal state estimation value , and then the filtered Euler angle data are extracted.

2. The head-mounted system of claim 1, wherein, the data acquisition module comprises: An image sensor data acquisition sub-module is responsible for acquiring original image data from a high-resolution image sensor built in the head-mounted device; the resolution of the image sensor is M x N, and the original image data collected at time t is represented as a two-dimensional matrix wherein represents the pixel value collected by the image sensor at position (i, j) at time t, i ∈ {1, 2, …, M}, j ∈ {1, 2, …, N}. The head pose sensor data acquisition sub-module synchronously acquires data of the head pose sensor to obtain pose information of the head-mounted device in a three-dimensional space; the head pose is described by three Euler angles: a pitch angle a(t), a yaw angle β(t) and a roll angle γ(t); the Euler angle data output by the pose sensor at time t is a vector wherein , , .

3. The head-mounted system of claim 1, wherein, the image stabilization module comprises: The coordinate transformation submodule constructs a rotation transformation matrix R(t), converts Euler angles into a rotation matrix form first, and sets rotation around x-axis, y-axis and z-axis as , , Corresponding rotation matrices are , , Therefore, the following equation is obtained: ； ； ； The total rotational transformation matrix R(t) is then: .

4. The head-mounted system of claim 3, wherein, the image stabilization module further comprises: The image distortion correction sub-module utilizes a rotation transformation matrix R(t) to correct the denoised image data The distortion correction is performed, and for each pixel point (i, j) in the image, the new coordinates (i', j') in the corrected image are calculated by the following formula: ； wherein, is the inverse matrix of the rotation transformation matrix Since coordinate transformation can cause pixel points to be mapped to non-integer positions, a bilinear interpolation algorithm is used to calculate the pixel value of the corrected image at integer coordinates; let the corrected image data be Then, for any integer coordinates (m, n), there is: ； wherein , are the coordinates of the four pixels closest to (m, n), , .

5. The head-mounted system of claim 1, wherein, the post-processing module comprises: The image enhancement sub-module adopts a histogram equalization algorithm to adjust the contrast of the image, so that the gray level distribution of the image is more uniform; assuming that the gray histogram of the image is represented as The cumulative distribution function (CDF) is: ； ； where L is the number of gray levels of the image; the pixel value of the image after histogram equalization is calculated by the following equation: ； wherein denotes a floor operation.

6. The head-mounted system of claim 5, wherein, the post-processing module further comprises: The image fusion sub-module adopts a weighted fusion algorithm to adaptively determine the fusion weight according to the stability degree and the detail feature of the image; the fusion weight coefficient is set as , and ; the final image data after fusion is represented as .​ 7. The head-mounted system of claim 1, wherein, the control and optimization module comprises: a parameter adjustment submodule, which determines optimal parameter values by minimizing a mean square error or a structural similarity index evaluation indicator, and a mean square error function is defined as: ； wherein is the ideal non-shaking image data; the parameter combination that minimizes the MSE is found by an iterative search algorithm.

8. The head-mounted system of claim 1, wherein, The head-mounted system further comprises a display compensation module in an anti-shake mode, and the display compensation module in the anti-shake mode comprises: a motion vector calculation submodule, which calculates a motion vector of a display picture according to data changes of the head posture sensor in adjacent two time intervals after the anti-shake mode is triggered; a compensation strategy determination submodule, which determines a suitable compensation strategy according to a size and a direction of the motion vector; the compensation strategy comprises: setting different compensation weight coefficients according to a module length of the motion vector, and the larger the module length, the larger the weight coefficient.

Images