Filtering stabilization algorithm based on head-mounted display

Abstract

The invention discloses a filtering stabilization algorithm based on a head-mounted display, and relates to the technical field of head-mounted displays. The system is realized based on a head display system, and the system comprises a data acquisition module which obtains original image data from a built-in high-resolution image sensor of head display equipment and synchronously collects data of a head attitude sensor so as to obtain attitude information of the head display equipment in a three-dimensional space; and the preprocessing module is used for carrying out denoising processing on the original image data by adopting a self-adaptive median filtering algorithm. According to the method, an accurate rotation and translation matrix is constructed, accurate transformation from a world coordinate system to a head display equipment coordinate system is realized through reasonable matrix operation, and correct mapping of an image in a spatial position is ensured; according to the method, an efficient bilinear interpolation algorithm is used for image distortion correction, it is ensured that when image deformation caused by head shaking is processed, pixel values under new coordinates can be rapidly and accurately calculated, image blurring and distortion are effectively reduced, and the definition and accuracy of images are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of head mounted displays, and in particular to a filtering stabilization algorithm based on head mounted displays. Background Art

[0002] With the rapid development of immersive technologies such as VR and AR, head-mounted display devices have been widely used in many fields such as games, education, medical care, and industry. When users use head-mounted display devices, the natural movement of the head, such as rotation and tilt, will cause the image produced by the device to shake and blur, which not only reduces the user's visual experience, but also may cause dizziness, fatigue and other problems, seriously affecting the use effect and user experience of the head-mounted display device. Therefore, how to effectively solve the problem of image jitter caused by head shaking in head-mounted display devices has become a technical problem that needs to be solved urgently in this field.

[0003] Some existing head-mounted display devices use built-in inertial sensors (such as gyroscopes, accelerometers, etc.) to monitor the movement of the head and try to stabilize the image. However, these methods have many limitations. On the one hand, some methods only use simple filtering algorithms to process sensor data, which is difficult to accurately reflect the complex movement of the head, resulting in poor image stabilization and obvious jitter. On the other hand, although some complex stabilization algorithms have improved stability to a certain extent, they have high computational complexity and demanding performance requirements for hardware devices, making it difficult to meet real-time requirements, and may introduce additional delays, affecting the user's interactive experience. In addition, most traditional image stabilization methods do not fully consider the characteristics of user head movements in different scenarios and the characteristics of different types of image content, lack pertinence and adaptability, and cannot provide high-quality visual experience. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and propose a filtering stabilization algorithm based on a head display.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: A head-mounted display-based filtering stabilization algorithm is implemented based on a head-mounted display system, the system comprising: The data acquisition module acquires raw image data from the high-resolution image sensor built into the head display device and simultaneously acquires data from the head posture sensor to obtain posture information of the head display device in three-dimensional space; The preprocessing module uses an adaptive median filter algorithm to denoise the original image data to suppress the random noise generated during the image acquisition process; the window size of the adaptive median filter is dynamically adjusted according to the local statistical characteristics of the image; the Kalman filter algorithm is used to filter the Euler angle data collected by the head posture sensor to remove high-frequency noise and errors in the posture data; The image stabilization module constructs a rotation transformation matrix from the world coordinate system to the head display device coordinate system based on the filtered head posture data; the rotation transformation matrix is ​​used to perform distortion correction on the denoised image data to compensate for the image deformation caused by head shaking; The post-processing module performs enhancement processing on the image after stabilization processing; uses the histogram equalization algorithm to adjust the contrast of the image; and fuses the enhanced image with the original image to improve the image's detail information and stability; The control and optimization module dynamically adjusts the parameters of the filtering algorithm according to the image stabilization effect and system performance indicators; The data storage and transmission module transmits the processed image data to the head-mounted display device for display.

[0006] Preferably: the data acquisition module includes: The image sensor data acquisition submodule is responsible for acquiring raw image data from the high-resolution image sensor built into the head-mounted display device. Assuming the resolution of the image sensor is M×N, the raw image data collected at time t is represented as a two-dimensional matrix ,in represents the pixel value collected by the image sensor at position (i, j) at time t, i∈{1,2,…,M}, j∈{1,2,…,N}; The head posture sensor data acquisition submodule synchronously collects data from the head posture sensor to obtain the posture information of the head display device in three-dimensional space; the head posture is described by three Euler angles: pitch angle α(t), yaw angle β(t), and roll angle γ(t); let the Euler angle data output by the posture sensor at time t be vector ,in .

[0007] Preferably: the preprocessing module comprises: The image denoising submodule uses an adaptive median filtering algorithm to denoise the original image data Iraw(t). For each pixel point (i, j) in the image, the median of the pixel values ​​in the k×k neighborhood centered on it is used as the pixel value after denoising. Suppose the image data after denoising is , then the denoising process is expressed as: in, represents the median operation, and Z represents a set of integers.

[0008] Preferably: the preprocessing module further includes: The posture data filtering submodule uses the Kalman filter algorithm to filter the Euler angle data E(t) collected by the head posture sensor. The state equation and observation equation of the Kalman filter algorithm are defined as: ; in, is the system state vector, including the Euler angle and its angular velocity; A is the state transfer matrix, which describes the relationship between the system states at different times; B is the control input matrix; U(t) is the control input vector; W(t) is the process noise vector, which is assumed to obey a Gaussian distribution with a mean of 0 and a covariance of Q; Z(t) is the observation vector, i.e., the Euler angle data actually measured by the sensor; C is the observation matrix; V(t) is the observation noise vector, which is assumed to obey a Gaussian distribution with a mean of 0 and a covariance of R; The Kalman filter algorithm iteratively calculates through two steps of prediction and update to obtain the optimal state estimate. , and then extract the filtered Euler angle data .

[0009] Preferably: the image stabilization module comprises: The coordinate transformation submodule constructs the rotation transformation matrix R(t). First, the Euler angle is converted into a rotation matrix. Suppose the rotation around the x-axis, y-axis, and z-axis The corresponding rotation matrices are , then: ; ; ; Then the total rotation transformation matrix R(t) is: .

[0010] Preferably: the image stabilization module further includes: The image distortion correction submodule uses the rotation transformation matrix R(t) to denoise the image data. Distortion correction is performed. For each pixel point (i, j) in the image, its new coordinates (i′, j′) in the corrected image are calculated by the following formula: ; in, is the rotation transformation matrix The inverse matrix of the corrected image. Since the coordinate transformation may cause the pixel points to be mapped to non-integer positions, the bilinear interpolation algorithm is used to calculate the pixel values ​​of the corrected image at integer coordinates. Suppose the corrected image data is , then for any integer coordinates (m,n), we have: ; ; in are the coordinates of the four pixels closest to (m,n), .

[0011] Preferably: the post-processing module comprises: The image enhancement submodule uses the histogram equalization algorithm to adjust the contrast of the image to make the grayscale distribution of the image more uniform; the grayscale histogram of the image is expressed as , then the cumulative distribution function (CDF) is: ; ; Where L is the gray level of the image; the image pixel value after histogram equalization Calculated by the following formula: ; in Indicates a floor operation.

[0012] Preferably: the post-processing module further includes: The image fusion submodule adopts a weighted fusion algorithm to adaptively determine the fusion weight according to the stability and detail features of the image; the fusion weight coefficient is set to ,and ; The final image data after fusion is It is expressed as: .

[0013] Preferably: the control and optimization module includes: The parameter adjustment submodule determines the optimal parameter value by minimizing the mean square error or the structural similarity index evaluation index. The mean square error function is defined as: ; in, For ideal non-shaking image data, the parameter combination that minimizes the MSE is found through an iterative search algorithm.

[0014] Preferably, the head display system further includes: a display compensation module in an anti-shake mode, and the display compensation module in the anti-shake mode includes: The motion vector calculation submodule calculates the motion vector of the display image based on the data changes of the head posture sensor in two adjacent time intervals after the anti-shake mode is triggered; The compensation strategy determination submodule determines a suitable compensation strategy according to the size and direction of the motion vector; the compensation strategy includes: setting different compensation weight coefficients according to the modulus of the motion vector, the larger the modulus, the larger the weight coefficient.

[0015] The beneficial effects of the present invention are: 1. The present invention constructs precise rotation and translation matrices, and realizes accurate transformation from the world coordinate system to the head display device coordinate system through reasonable matrix operations, thereby ensuring the correct mapping of the image in spatial position; an efficient bilinear interpolation algorithm is used to correct image distortion, thereby ensuring that when processing image deformation caused by head shaking, the pixel values ​​under the new coordinates can be calculated quickly and accurately, effectively reducing image blur and distortion, and improving image clarity and accuracy.

[0016] 2. The present invention integrates a variety of high-precision sensors, which can collect the rotation and translation information of the head in real time and accurately, providing a reliable data basis for subsequent image stabilization processing; it adopts data filtering preprocessing technology to effectively remove sensor noise interference, ensure that the acquired head posture data is more accurate, and enable the system to capture subtle movements of the head more sensitively.

[0017] 3. The unique anti-shake mode triggering mechanism of the present invention can timely start the corresponding compensation strategy according to the intensity of head movement, effectively deal with various complex head shaking situations, and maintain stable image display. The present invention includes compensation weight adjustment and direction priority setting based on motion vectors, which can flexibly adjust the compensation amplitude and direction according to different movement situations, so that the shaking of the display screen in different directions can be targetedly compensated, greatly improving the stability and visual effect of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a processing flow chart of a head-mounted display-based filtering stabilization algorithm proposed by the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention is further described in detail below in conjunction with specific implementation methods.

[0020] Embodiment 1: A filtering stabilization algorithm based on a head-mounted display is implemented based on a head-mounted display system, the system comprising: The data acquisition module acquires raw image data from the high-resolution image sensor built into the head display device and simultaneously acquires data from the head posture sensor to obtain posture information of the head display device in three-dimensional space; The preprocessing module uses an adaptive median filter algorithm to denoise the original image data to suppress the random noise generated during the image acquisition process; the window size of the adaptive median filter is dynamically adjusted according to the local statistical characteristics of the image; the Kalman filter algorithm is used to filter the Euler angle data collected by the head posture sensor to remove high-frequency noise and errors in the posture data, thereby improving the accuracy and stability of posture estimation; The image stabilization module constructs a rotation transformation matrix from the world coordinate system to the head display device coordinate system based on the filtered head posture data; the rotation transformation matrix is ​​used to perform distortion correction on the denoised image data to compensate for the image deformation caused by head shaking; The post-processing module performs enhancement processing on the image after stabilization processing; uses the histogram equalization algorithm to adjust the contrast of the image; and fuses the enhanced image with the original image to improve the image's detail information and stability; The control and optimization module dynamically adjusts the parameters of the filtering algorithm according to the image stabilization effect and system performance indicators; The data storage and transmission module transmits the processed image data to the head-mounted display device for display; a high-speed data transmission interface is used to ensure the real-time and stability of data transmission.

[0021] Wherein, the data acquisition module includes: The image sensor data acquisition submodule is responsible for acquiring raw image data from the high-resolution image sensor built into the head-mounted display device. Assuming the resolution of the image sensor is M×N, the raw image data collected at time t is represented as a two-dimensional matrix ,in represents the pixel value collected by the image sensor at position (i, j) at time t, i∈{1,2,…,M}, j∈{1,2,…,N}; The head posture sensor data acquisition submodule synchronously collects data from the head posture sensor to obtain the posture information of the head display device in three-dimensional space; the head posture is described by three Euler angles: pitch angle α(t), yaw angle β(t), and roll angle γ(t); let the Euler angle data output by the posture sensor at time t be vector ,in .

[0022] Wherein, the preprocessing module includes: The image denoising submodule uses an adaptive median filtering algorithm to denoise the original image data Iraw(t). For each pixel point (i, j) in the image, the median of the pixel values ​​in the k×k neighborhood centered on it is used as the pixel value after denoising. Suppose the image data after denoising is , then the denoising process is expressed as: in, represents the median operation, and Z represents a set of integers; The posture data filtering submodule uses the Kalman filter algorithm to filter the Euler angle data E(t) collected by the head posture sensor. The state equation and observation equation of the Kalman filter algorithm are defined as: ; in, is the system state vector, including the Euler angle and its angular velocity; A is the state transfer matrix, which describes the relationship between the system states at different times; B is the control input matrix; U(t) is the control input vector; W(t) is the process noise vector, which is assumed to obey a Gaussian distribution with a mean of 0 and a covariance of Q; Z(t) is the observation vector, i.e., the Euler angle data actually measured by the sensor; C is the observation matrix; V(t) is the observation noise vector, which is assumed to obey a Gaussian distribution with a mean of 0 and a covariance of R; The Kalman filter algorithm iteratively calculates through two steps of prediction and update to obtain the optimal state estimate. , and then extract the filtered Euler angle data .

[0023] Wherein, the image stabilization module comprises: The coordinate transformation submodule constructs the rotation transformation matrix R(t). First, the Euler angle is converted into a rotation matrix. Suppose the rotation around the x-axis, y-axis, and z-axis The corresponding rotation matrices are , then: ; ; ; Then the total rotation transformation matrix R(t) is: ; The image distortion correction submodule uses the rotation transformation matrix R(t) to denoise the image data. Distortion correction is performed. For each pixel point (i, j) in the image, its new coordinates (i′, j′) in the corrected image are calculated by the following formula: ; in, is the rotation transformation matrix The inverse matrix of the corrected image. Since the coordinate transformation may cause the pixel points to be mapped to non-integer positions, the bilinear interpolation algorithm is used to calculate the pixel values ​​of the corrected image at integer coordinates. Suppose the corrected image data is , then for any integer coordinates (m,n), we have: ; ; in are the coordinates of the four pixels closest to (m,n), .

[0024] Wherein, the post-processing module comprises: The image enhancement submodule uses the histogram equalization algorithm to adjust the contrast of the image to make the grayscale distribution of the image more uniform; the grayscale histogram of the image is expressed as , then the cumulative distribution function (CDF) is: ; ; Where L is the gray level of the image; the image pixel value after histogram equalization Calculated by the following formula: ; in Indicates a round-down operation; The image fusion submodule adopts a weighted fusion algorithm to adaptively determine the fusion weight according to the stability and detail features of the image; the fusion weight coefficient is set to ,and ; The final image data after fusion is It is expressed as: .

[0025] Wherein, the control and optimization module includes: The parameter adjustment submodule determines the optimal parameter value by minimizing the mean square error or the structural similarity index evaluation index. The mean square error function is defined as: ; in, For ideal non-shaking image data, the parameter combination that minimizes the MSE is found through an iterative search algorithm.

[0026] Embodiment 2: A filtering stabilization algorithm based on a head-mounted display. In this embodiment, based on Embodiment 1, the head-mounted display system further includes: a display compensation module in an anti-shake mode, and the display compensation module in the anti-shake mode includes: The motion vector calculation submodule calculates the motion vector of the displayed image based on the data changes of the head posture sensor in two adjacent time intervals after the anti-shake mode is triggered. , the rotation angle change of the head is , the translation amount changes to , then the motion vector It is expressed as: ; The motion vector is used to subsequently determine the compensation direction and amplitude of the display picture.

[0027] The compensation strategy determination submodule determines the appropriate compensation strategy based on the size and direction of the motion vector. For example, when the component of the motion vector in a certain direction is large, a large compensation is performed in that direction; when the motion vector is small, a small compensation is performed or no compensation is performed. The specific compensation strategy can be adjusted and optimized through experiments and experience.

[0028] The compensation strategy includes: setting different compensation weight coefficients according to the length of the motion vector. The larger the length, the larger the weight coefficient. For example, define the compensation weight coefficient ,in is a monotonically increasing function (such as a linear function, exponential function, etc.), and .when No compensation will be made when , maximum compensation is performed.

[0029] In addition, it is also possible to consider setting different compensation direction priorities according to the direction of the motion vector. For example, a higher priority and a larger compensation amplitude may be given to horizontal motion, to which the human eye is more sensitive.

[0030] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A head-mounted display-based filtering stabilization algorithm, characterized in that: Based on the head display system, the system includes: The data acquisition module acquires raw image data from the high-resolution image sensor built into the head display device and simultaneously acquires data from the head posture sensor to obtain posture information of the head display device in three-dimensional space; The preprocessing module uses an adaptive median filter algorithm to denoise the original image data to suppress the random noise generated during the image acquisition process; the window size of the adaptive median filter is dynamically adjusted according to the local statistical characteristics of the image; the Kalman filter algorithm is used to filter the Euler angle data collected by the head posture sensor to remove high-frequency noise and errors in the posture data; The image stabilization module constructs a rotation transformation matrix from the world coordinate system to the head display device coordinate system based on the filtered head posture data; the rotation transformation matrix is ​​used to perform distortion correction on the denoised image data to compensate for the image deformation caused by head shaking; The post-processing module performs enhancement processing on the image after stabilization processing; uses the histogram equalization algorithm to adjust the contrast of the image; and fuses the enhanced image with the original image to improve the image's detail information and stability; The control and optimization module dynamically adjusts the parameters of the filtering algorithm according to the image stabilization effect and system performance indicators; The data storage and transmission module transmits the processed image data to the head-mounted display device for display.

2. The head-mounted display-based filtering stabilization algorithm according to claim 1, characterized in that: The data acquisition module comprises: The image sensor data acquisition submodule is responsible for acquiring raw image data from the high-resolution image sensor built into the head-mounted display device. Assuming the resolution of the image sensor is M×N, the raw image data collected at time t is represented as a two-dimensional matrix ,in represents the pixel value collected by the image sensor at position (i, j) at time t, i∈{1,2,…,M}, j∈{1,2,…,N}; The head posture sensor data acquisition submodule synchronously collects data from the head posture sensor to obtain the posture information of the head display device in three-dimensional space; the head posture is described by three Euler angles: pitch angle α(t), yaw angle β(t), and roll angle γ(t); let the Euler angle data output by the posture sensor at time t be vector ,in .

3. The head-mounted display-based filtering stabilization algorithm according to claim 1, characterized in that: The preprocessing module comprises: The image denoising submodule uses an adaptive median filtering algorithm to denoise the original image data Iraw(t). For each pixel point (i, j) in the image, the median of the pixel values ​​in the k×k neighborhood centered on it is used as the pixel value after denoising. Suppose the image data after denoising is , then the denoising process is expressed as: in, represents the median operation, and Z represents a set of integers.

4. The head-mounted display-based filtering stabilization algorithm according to claim 3, characterized in that: The preprocessing module also includes: The posture data filtering submodule uses the Kalman filter algorithm to filter the Euler angle data E(t) collected by the head posture sensor. The state equation and observation equation of the Kalman filter algorithm are defined as: ； in, is the system state vector, including the Euler angle and its angular velocity; A is the state transfer matrix, which describes the relationship between the system states at different times; B is the control input matrix; U(t) is the control input vector; W(t) is the process noise vector, which is assumed to obey a Gaussian distribution with a mean of 0 and a covariance of Q; Z(t) is the observation vector, i.e., the Euler angle data actually measured by the sensor; C is the observation matrix; V(t) is the observation noise vector, which is assumed to obey a Gaussian distribution with a mean of 0 and a covariance of R; The Kalman filter algorithm iteratively calculates through two steps of prediction and update to obtain the optimal state estimate. , and then extract the filtered Euler angle data .

5. The head-mounted display-based filtering stabilization algorithm according to claim 1, characterized in that: The image stabilization module comprises: The coordinate transformation submodule constructs the rotation transformation matrix R(t). First, the Euler angle is converted into a rotation matrix. Suppose the rotation around the x-axis, y-axis, and z-axis The corresponding rotation matrices are , then: ； ； ； Then the total rotation transformation matrix R(t) is: .

6. The head-mounted display-based filtering stabilization algorithm according to claim 5, characterized in that: The image stabilization module also includes: The image distortion correction submodule uses the rotation transformation matrix R(t) to denoise the image data. Distortion correction is performed. For each pixel point (i, j) in the image, its new coordinates (i′, j′) in the corrected image are calculated by the following formula: ； in, is the rotation transformation matrix The inverse matrix of the corrected image. Since the coordinate transformation may cause the pixel points to be mapped to non-integer positions, the bilinear interpolation algorithm is used to calculate the pixel values ​​of the corrected image at integer coordinates. Suppose the corrected image data is , then for any integer coordinates (m,n), we have: ； ； in are the coordinates of the four pixels closest to (m,n), .

7. The head-mounted display-based filtering stabilization algorithm according to claim 1, characterized in that: The post-processing module comprises: The image enhancement submodule uses the histogram equalization algorithm to adjust the contrast of the image to make the grayscale distribution of the image more uniform; the grayscale histogram of the image is expressed as , then the cumulative distribution function (CDF) is: ； ； Where L is the gray level of the image; the image pixel value after histogram equalization Calculated by the following formula: ； in Indicates a floor operation.

8. The head mounted display based filtering stabilization algorithm according to claim 7, characterized in that: The post-processing module also includes: The image fusion submodule adopts a weighted fusion algorithm to adaptively determine the fusion weight according to the stability and detail features of the image; the fusion weight coefficient is set to ,and ; The final image data after fusion is It is expressed as: .

9. The head-mounted display-based filtering stabilization algorithm according to claim 1, characterized in that: The control and optimization module includes: The parameter adjustment submodule determines the optimal parameter value by minimizing the mean square error or the structural similarity index evaluation index. The mean square error function is defined as: ； in, For ideal non-shaking image data, the parameter combination that minimizes the MSE is found through an iterative search algorithm.

10. The head mounted display based filtering stabilization algorithm according to claim 1, characterized in that: The head display system further includes: a display compensation module in an anti-shake mode, wherein the display compensation module in the anti-shake mode includes: The motion vector calculation submodule calculates the motion vector of the display image based on the data changes of the head posture sensor in two adjacent time intervals after the anti-shake mode is triggered; The compensation strategy determination submodule determines a suitable compensation strategy according to the size and direction of the motion vector; the compensation strategy includes: setting different compensation weight coefficients according to the modulus of the motion vector, the larger the modulus, the larger the weight coefficient.

Images