Image processing method and device, head-mounted equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and provides an image processing method and device, head-mounted equipment and a storage medium, and the method provides basic data for anti-shake processing through obtaining an original attitude track of an image sensor in a three-dimensional space. The original attitude track is smoothed, so that high-frequency jitter caused by rapid movement of equipment is effectively reduced, a more stable attitude track is obtained, and the instability of an image frame is reduced; the compensation transformation amount of each image frame is calculated based on the smooth attitude trajectory, and meanwhile, the rotation jitter and the translation jitter of the image frame are eliminated, so that the attitude compensation is more accurate and effective; the compensation transformation quantity is applied to the image frame to carry out pose compensation transformation, so that image jitter caused by motion is effectively reduced, the processed stable and clear image frame is obtained, and the jitter elimination effectiveness of the image processing technology on the image collected by the head-mounted equipment is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, head-mounted device, and storage medium. Background Technology

[0002] With the increasing popularity of head-mounted camera devices, video recording has become an important way for people to record beautiful moments and share life's little moments in daily life. However, during video recording, due to equipment shake, the photographer's hand tremor, or external interference (such as passing vehicles or wind), the video footage often becomes shaky, blurry, or visually uncomfortable. This not only affects the viewing experience but also reduces the accuracy and value of the video in subsequent analysis and applications.

[0003] In recent years, with the continuous development of digital electronics technology, in addition to traditional physical image stabilization methods such as optical and mechanical image stabilization, electric image stabilization (EIS) technology has also received increasing attention. EIS technology analyzes the displacement and rotation changes between video frames through software algorithms and performs spatial transformations on the image to achieve smooth video output. Due to its significant advantages of low cost and high flexibility, EIS has become the mainstream image stabilization solution.

[0004] While traditional image processing techniques (EIS) have improved video quality to some extent, their effectiveness in image stabilization for head-mounted devices is less than ideal. When the wearer rotates their head, the image sensor not only rotates but also experiences significant translational displacement due to the radius of rotation. Traditional EIS techniques, which rely solely on rotation compensation, cannot effectively eliminate this type of shaking, resulting in a substantial impact on video stability and clarity. Therefore, improving the effectiveness of image stabilization in head-mounted devices has become a pressing technical challenge. Summary of the Invention

[0005] This application provides an image processing method, apparatus, device, and storage medium, aiming to solve the technical problem that related image processing technologies are ineffective in eliminating jitter in images acquired by head-mounted devices.

[0006] In a first aspect, this application provides an image processing method, comprising: During the process of acquiring image frames based on the image sensor of the head-mounted device, the original attitude trajectory of the image sensor in three-dimensional space is obtained; The original attitude trajectory is smoothed to obtain a smooth attitude trajectory; Based on the smoothed posture trajectory, the compensation transformation amount corresponding to each image frame is determined; Based on the compensation transformation amount corresponding to each image frame, pose compensation is performed on each image frame in sequence to obtain the processed image frame.

[0007] Secondly, this application also provides an image processing apparatus, comprising: The original attitude trajectory acquisition module is used to acquire the original attitude trajectory of the image sensor in three-dimensional space during the process of the head-mounted device acquiring image frames based on the image sensor; A smooth attitude trajectory acquisition module is used to smooth the original attitude trajectory to obtain a smooth attitude trajectory; The compensation transformation amount acquisition module is used to determine the compensation transformation amount corresponding to each image frame based on the smoothed attitude trajectory. The pose compensation transformation module is used to sequentially perform pose compensation on each image frame based on the compensation transformation amount corresponding to each image frame, so as to obtain the processed image frame.

[0008] Thirdly, this application also provides a head-mounted device, the head-mounted device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the image processing method described above.

[0009] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the image processing method described above.

[0010] This application provides an image processing method, apparatus, device, and storage medium. The method acquires the original attitude trajectory of an image sensor in three-dimensional space, providing basic data for image stabilization. Smoothing the original attitude trajectory effectively reduces high-frequency jitter caused by rapid device movement, resulting in a more stable attitude trajectory and reduced image frame instability. Based on the smoothed attitude trajectory, the compensation transformation amount for each image frame is calculated, simultaneously eliminating rotational and translational jitter, making attitude compensation more accurate and effective. Applying the compensation transformation amount to the image frame for pose compensation transformation effectively reduces image jitter caused by motion, resulting in stable and clear image frames after processing, thus improving the effectiveness of image processing technology in eliminating jitter in images acquired by head-mounted devices. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a first embodiment of an image processing method provided in this application. Figure 2 This is a schematic diagram of the pose structure for translational displacement provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a first embodiment of an image processing apparatus provided in this application; Figure 4 This is a schematic block diagram of the structure of a head-mounted device provided in an embodiment of this application.

[0013] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0016] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0017] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a first embodiment of an image processing method provided in this application.

[0018] like Figure 1 As shown, the image processing method includes steps S101 to S105.

[0019] S101. During the process of the head-mounted device acquiring image frames based on the image sensor, the original attitude trajectory of the image sensor in three-dimensional space is obtained; In one embodiment, a gyroscope sensor can be integrated into the head-mounted device. This gyroscope is a sensor capable of measuring angular velocity, which can detect the rotational speed of the head-mounted device around three axes (X, Y, and Z axes).

[0020] Specifically, when the head-mounted device is started, the gyroscope sensor is initialized, and parameters such as the sampling rate and range of the gyroscope sensor are set. The parameters of the gyroscope sensor can use pre-set general parameters or be manually set according to user needs. Among them, the sampling rate refers to the number of times the gyroscope sensor collects angular velocity data per second. For example, a sampling rate of 1000Hz means that the gyroscope collects 1000 angular velocity samples per second.

[0021] Generally, the sampling rate of the gyroscope sensor can be set to be at least equal to or higher than the acquisition frame rate of the image frame sequence captured by the head-mounted device, so as to ensure that each image frame has one or more corresponding angular velocity data points, thereby enabling more accurate estimation of motion changes between image frames.

[0022] The frame rate refers to the number of image frames captured by the head-mounted device per second. For example, 30 FPS (Frame Per Second) means that the head-mounted device captures 30 image frames per second.

[0023] If the sampling rate of the gyroscope sensor is set to be equal to the frame rate of the image frame sequence captured by the head-mounted device, then by synchronously triggering the gyroscope sensor and the image sensor, at the moment the image sensor acquires an image frame, the gyroscope sensor synchronously acquires an angular velocity data point, so that each image frame corresponds to an angular velocity data point synchronously.

[0024] If the sampling rate of the gyroscope sensor is set higher than the frame rate of the image frame sequence captured by the head-mounted device, meaning that the gyroscope sensor synchronously and evenly collects a greater number of angular velocity data points during the time period of the image frame sequence captured by the head-mounted device (with at least one angular velocity data point in each image frame sequence), then data interpolation can be performed between each image frame to obtain a smoother motion estimate.

[0025] In some applications, the gyroscope's sampling rate and the frame rate of image acquisition can be dynamically adjusted based on the actual motion state. For example, the sampling rate and frame rate can be increased when the head-mounted device is in vigorous motion, and decreased when stationary to save resources.

[0026] Furthermore, based on a motion detection algorithm, the current motion state of the head-mounted device is detected; based on the current motion state, the sensor configuration parameters of the gyroscope sensor set in the head-mounted device are determined; based on the sensor configuration parameters, angular velocity data of the head-mounted device during the capture of an image frame sequence is collected; based on the angular velocity data, the attitude change of the image sensor in the head-mounted device in three-dimensional space is calculated to obtain the original attitude trajectory of the image sensor.

[0027] In one embodiment, an IMU (Inertial Measurement Unit) including a gyroscope sensor and an accelerometer can be configured in the head-mounted device. A motion detection algorithm is used to calculate and detect the readings of the gyroscope sensor and the accelerometer to determine the current motion state of the head-mounted device. For example, whether the head-mounted device is in motion can be determined by calculating thresholds for angular velocity and acceleration data.

[0028] In one embodiment, the current motion state may include a stationary state and multiple levels of motion states. Specifically, one or more operation monitoring thresholds can be set according to actual application requirements to distinguish between a stationary state and multiple different levels of motion states, wherein the multiple levels of motion states may include, for example, a slight motion state, a moderate motion state, and a vigorous motion state.

[0029] For example, slight movement can be the movement state when the wearer is stationary and slowly turns their head; moderate movement can be the movement state when the wearer walks slowly and keeps their head still or slowly turns; vigorous movement can be the movement state when the wearer runs or swings their head rapidly. It is understood that the definition and division of multi-level movement states can be set according to actual application needs. This embodiment is only an example for reference and is not intended to be a specific limitation on multi-level movement states.

[0030] In one embodiment, corresponding sensor configuration parameters, including the IMU sampling rate and the image sensor acquisition frame rate, are pre-configured for different motion states. For example, the IMU sampling rate and the image sensor acquisition frame rate are highest during vigorous motion, and lowest during stationary motion.

[0031] By continuously monitoring the angular velocity and acceleration data collected by the gyroscope sensor and accelerometer, a motion detection algorithm is used to determine the current motion state of the head-mounted device in real time. For example, if the angular velocity data is greater than or equal to a first angular velocity threshold and the acceleration data is less than a second acceleration threshold, or if the acceleration data is greater than or equal to the first acceleration threshold and the angular velocity data is less than the second acceleration threshold, the current motion state of the head-mounted device can be determined to be a slight motion state, where the second acceleration threshold is greater than the first acceleration threshold and the second angular velocity threshold is greater than the first angular velocity threshold. If the angular velocity data is at the second angular velocity threshold and the acceleration data is less than a third acceleration threshold, or if the acceleration data is greater than or equal to the second acceleration threshold and the angular velocity data is less than the third angular velocity threshold, the current motion state of the head-mounted device can be determined to be a moderate motion state, where the third acceleration threshold is greater than the second acceleration threshold and the third angular velocity threshold is greater than the second angular velocity threshold. And so on, to determine the current motion state of the head-mounted device.

[0032] Based on the current motion state, the sampling rate of the gyroscope sensor is dynamically adjusted and applied to the gyroscope sensor to obtain more accurate or sparser acceleration data. Simultaneously, the frame rate of the image sensor can also be dynamically adjusted based on the current motion state and applied to the image sensor to capture more or fewer image frames.

[0033] This embodiment reduces computational burden by dynamically adjusting sensor configuration parameters, adapts to the actual needs of image acquisition and angular velocity data under different motion states, avoids waste of computational and power resources, and improves the efficiency of head-mounted devices.

[0034] When the head-mounted device starts image acquisition or video recording and begins to acquire image frame sequences, the gyroscope sensor simultaneously begins to synchronously acquire continuous angular velocity data of the head-mounted device at a set sampling rate. In order to ensure the synchronization of angular velocity data with image frames, precise timestamp recording needs to be implemented during image capture and data acquisition so that the angular velocity data and image frame sequences can be synchronized according to the timestamp information.

[0035] In one embodiment, angular velocity data can be represented in the form of triples, i.e. In this triplet, each element represents the angular velocity along one axis, typically measured in degrees per second (° / s) or radians per second (rad / s). For example, suppose a head-mounted device's gyroscope sensor outputs the following angular velocity data at a given moment: X-axis angular velocity (2.5° / s), Y-axis angular velocity (-1.0° / s), and Z-axis angular velocity (0.5° / s). This indicates that the head-mounted device is rotating at a rate of 2.5 degrees per second around the positive X-axis, 1.0 degrees per second around the negative Y-axis, and 0.5 degrees per second around the positive Z-axis. Representing this angular velocity data as a triplet, (2.5, -1.0, 0.5), this triplet describes the head-mounted device's rotational rate around the three principal axes at a given moment.

[0036] The raw angular velocity data acquired by the gyroscope sensor may contain noise and biases; therefore, it can be preprocessed. Specifically, for noise, a low-pass filter can be applied to remove high-frequency noise, retaining the useful signal related to head movement. For bias, the output of the gyroscope sensor can be detected while the head-mounted device is stationary. If the gyroscope sensor outputs a non-zero angular velocity value, it indicates that the gyroscope sensor has a bias. Angular velocity data can be acquired and bias calculated while the head-mounted device is stationary, and sensor bias correction can be performed in subsequent data processing. The preprocessed angular velocity data is then used for subsequent attitude calculations of the head-mounted device.

[0037] In one embodiment, based on preprocessed angular velocity data, the continuous attitude change of the image sensor installed in the head-mounted device in three-dimensional space is calculated by integration to obtain the original attitude trajectory of the image sensor.

[0038] In practical applications, the original function of angular velocity data is usually not directly obtained in attitude estimation, and the integration interval may be continuous. Therefore, numerical integration methods can be used to approximate the integral of angular velocity data over time, thereby obtaining the attitude change.

[0039] Numerical integration methods can include the trapezoidal rule, Simpson's method, and Runge-Kutta method. For example, the trapezoidal rule divides the integration interval into multiple smaller intervals, approximates the area under the function curve in each interval using a trapezoid, and then sums the areas of all trapezoids to obtain an approximate value of the integral. Simpson's method is an improvement on the trapezoidal rule; it approximates the area under the function curve using a parabola (quadratic polynomial) in each smaller interval, thus obtaining a more accurate approximation of the integral. The Runge-Kutta method predicts the value of the function at the endpoint of the interval by calculating a weighted average of the function values ​​over each smaller interval.

[0040] In one embodiment, the triplet (angular velocity data) output by the gyroscope sensor:

[0041] in, Indicates that the gyroscope sensor is in Angular velocity data obtained from constant measurements. This refers to the transpose operation. It is a vector containing three components. , , respectively representing the image sensor in The instantaneous angular velocity around the X, Y, and Z axes of the camera coordinate system at all times.

[0042] The pose representation of an image sensor can be achieved using quaternions. (satisfy ; This refers to the transpose operation, used to represent and calculate rotations in three-dimensional space. A 3×3 rotation matrix can also be used. express.

[0043] In quaternion representation, Indicates the image sensor in The attitude at any given moment is represented using quaternions; , , , These are the four components of a quaternion. Among them... The real part represents the scalar part of the quaternion, which is used when the quaternion represents a rotation. A value close to 1 indicates almost no rotation, while A value close to 0 indicates a rotation close to 180 degrees; , , The imaginary part represents the vector component of the quaternion; these three components together form a three-dimensional vector. This three-dimensional vector is related to the direction of the rotation axis. The length (modulus) of a quaternion is related to the rotation angle; the larger the length, the larger the rotation angle. Quaternions satisfy the normalization condition as follows: ,Right now .

[0044] Before starting the calculations, different coordinate systems are first defined and calibrated, including the camera coordinate system (C) corresponding to the image sensor, the IMU coordinate system (I) corresponding to the gyroscope sensor, and the world coordinate system (W) corresponding to the head-mounted device.

[0045] Specifically, the origin of the camera coordinate system (C) is located at the optical center of the image sensor. The axis points along the optical axis towards the shooting scene. The axis points to the right side of the image frame. The axis points to the bottom of the image frame (following the right-hand rule). The IMU coordinate system (I) refers to the coordinate system of the gyroscope sensor itself, and the rotation matrix from the IMU coordinate system to the camera coordinate system needs to be obtained through extrinsic parameter calibration (factory calibration). Translation matrix This describes the position and orientation of the gyroscope sensor relative to the image sensor. The world coordinate system can be referenced to the camera coordinate system of the first frame, i.e. .

[0046] Preprocessed angular velocity data Transformation from IMU coordinate system to camera coordinate system:

[0047] in, Indicates the time of the gyroscope sensor Vector representation of angular velocity data obtained from time-lapse measurements; This represents the rotation matrix from the IMU coordinate system to the camera coordinate system; This represents the vector representation of the angular velocity data after transformation to the camera coordinate system at time t.

[0048] Generally, quaternions or Euler angles are used to represent the attitude during integration. Taking quaternions as an example, numerical integration is performed using quaternion differential equations:

[0049] in, The current attitude quaternion is a four-element vector containing one real part and three imaginary parts. This is used to describe the rotational state of an image sensor in three-dimensional space. This represents quaternion multiplication. This represents the vector representation of the angular velocity data after transformation to the camera coordinate system at time t. Representing attitude quaternions Over time The rate of change, or attitude change rate, reflects the change in the attitude of the image sensor. It describes the rate of attitude change for each image frame in the original attitude trajectory. Therefore, Used to represent the original attitude trajectory.

[0050] The quaternion differential equation reflects the process of image sensor attitude change caused by angular velocity data, and finally obtains the original attitude trajectory of image sensor in three-dimensional space. This original attitude trajectory reflects the spatial orientation change of the user's head during the movement process, providing a kinematic basis for subsequent fusion with visual features.

[0051] This embodiment converts the angular velocity data collected by the gyroscope into quaternions and uses numerical integration to predict the attitude change of the image sensor in three-dimensional space, thereby generating the original attitude trajectory. This not only corrects sensor noise and bias, but also synchronizes the angular velocity data with the image frame, ensuring the accuracy and real-time performance of the image stabilization process. This significantly improves the stability and reliability of the head-mounted device's image quality in dynamic environments.

[0052] S102. Smooth the original attitude trajectory to obtain a smooth attitude trajectory; In one embodiment, the original attitude trajectory contains high-frequency jitter, such as hand tremors or head tremors causing multiple jitters. Therefore, the calculated original attitude trajectory needs to be smoothed to eliminate high-frequency jitter components. The goal of smoothing is to reduce high-frequency noise (i.e., high-frequency jitter components) in attitude estimation while preserving important motion features as much as possible to improve the accuracy of subsequent pose compensation.

[0053] Specifically, the original attitude trajectory is preprocessed, including removing outliers and interpolation, to eliminate anomalies, breakpoints and noise caused by sensor defects, transmission errors or time misalignments, and to ensure the continuity and consistency of the data.

[0054] In one embodiment, the rotation matrix corresponding to the image frame at each time point in the original attitude trajectory is obtained; the rotation matrix corresponding to each image frame is smoothed by a filtering algorithm to obtain the filtered rotation matrix corresponding to each time point; the smoothed attitude value corresponding to each time point is extracted from each of the filtered rotation matrices, and the smoothed attitude trajectory is generated based on the smoothed attitude value.

[0055] One or more filtering algorithms, such as moving average filtering, Kalman filtering, or low-pass filtering, can be used to smooth the original attitude trajectory, generating a smoothed attitude trajectory. The filtering algorithm is then applied to the preprocessed original attitude trajectory to calculate the smoothed attitude value at each time point.

[0056] In essence, a smoothed attitude value refers to the smoothed attitude representation at a specific point in time. It is an element in the smoothed attitude trajectory and can be in the form of a rotation matrix, quaternion, or Euler angles. The smoothed attitude value reflects the rotation state of the image sensor relative to a reference coordinate system (such as the world coordinate system) at that specific point in time, and has already removed high-frequency jitter and gyroscope sensor noise. A smoothed attitude trajectory refers to the attitude change path over the entire time series after smoothing. It consists of a series of smoothed attitude values ​​at consecutive time points, each smoothed attitude value being a point in the trajectory. The smoothed attitude trajectory displays a set of smoothed attitude values ​​that change over time, and these smoothed attitude values ​​are the basic units that constitute the trajectory.

[0057] In one embodiment, as can be seen from the aforementioned steps, the original attitude trajectory contains the rotation matrix corresponding to the image frame at each time point. The selected filtering algorithm is applied to the filtering matrix at each time point. For example, when using a Kalman filter, it is necessary to define the state transition matrix and the observation matrix, as well as the covariance matrix of the process noise and the observation noise; for moving average filtering, the local average value of each rotation matrix is ​​calculated; for low-pass filtering, a filter is designed to remove high-frequency noise.

[0058] After filtering, the filtered rotation matrix corresponding to each time point is obtained. These matrices are smoother, reducing high-frequency jitter caused by gyroscope sensor (or IMU) noise. Smooth attitude values ​​are extracted from the filtered rotation matrix. These smooth attitude values ​​can be represented more intuitively by converting the rotation matrix into Euler angles or quaternions. The extracted smooth attitude values ​​are arranged in chronological order to generate a smooth attitude trajectory.

[0059] The process of extracting smoothed attitude values ​​from the filtered rotation matrix and converting them into Euler angles or quaternion representations is a way to transform the abstract rotation matrix into a more intuitive and easier-to-understand attitude representation.

[0060] Specifically, after smoothing the rotation matrix, the filtered rotation matrix corresponding to each time point is obtained, and the smoothed attitude value is extracted from each filtered rotation matrix.

[0061] Taking the conversion of a rotation matrix to Euler angles as an example: Determine the conversion order of the Euler angles; common orders include ZYX (yaw-pitch-roll), ZXY, etc. Use an appropriate mathematical formula to convert the filtered rotation matrix to Euler angles. For example, for the ZYX order, the conversion formula might be as follows:

[0062]

[0063]

[0064] in,( () represents the smoothed attitude value in Euler angles. Represents the elements in the rotation matrix, such as This represents the element in the first row and first column of the rotation matrix.

[0065] Alternatively, based on the computational requirements of the filtering algorithm, the original attitude values ​​at each time point can be directly extracted from the original attitude trajectory, and the original attitude values ​​can be smoothed using the filtering algorithm to output smooth attitude values.

[0066] Taking moving average filtering as an example, it's a simple time series smoothing technique that smooths data by calculating the average value within a certain time window. Specifically, a moving average window is first defined, and its size is set. The window size determines the degree of smoothing; the window size can be chosen based on application requirements and data characteristics. A larger window can smooth the data more effectively but may also cause signal delays and distortions. For each time point in the original attitude trajectory, the average value of the attitude data, including that time point and a certain number of time points before and after it, is calculated. For example, if the window size is 5, then for each time point, the average attitude value of the two time points before and after it (a total of 5 time points) is calculated. The calculated average value is used as the new attitude at that time point, thereby updating the entire original attitude trajectory, and the smoothed attitude trajectory is constructed with the new attitude corresponding to each time point.

[0067] Taking Kalman filtering as an example, Kalman filtering is a recursive filtering algorithm that can estimate the state of a dynamic system from a series of incomplete and noisy measurements, even in the presence of noise. First, a state-space model is defined, including state transition equations and observation equations. In attitude estimation, the state typically includes position, velocity, and acceleration. An initial state and a covariance matrix are set, where the initial state can be zero or an estimated value based on some prior knowledge, and the covariance matrix represents the uncertainty of the initial state. The state at the next time point is predicted based on the state transition equations, and the predicted state is updated using the observation equations and new measurements. Kalman filtering adjusts the state estimate by minimizing the error between the predicted state and the measured values. The state at each time point after Kalman filtering is the smoothed attitude output, resulting in a smoothed attitude trajectory.

[0068] This embodiment preprocesses the original attitude trajectory to ensure the continuity and consistency of the data. Then, a filtering algorithm is applied to eliminate gyroscope sensor noise and high-frequency jitter components. By extracting smooth attitude values ​​from the filtered rotation matrix, a smooth attitude trajectory is generated, providing more accurate and stable visual information for subsequent pose compensation and image stabilization. This improves the quality and reliability of images and videos from the head-mounted device in dynamic environments.

[0069] S103. Based on the smoothed attitude trajectory, determine the compensation transformation amount corresponding to each image frame; Most existing image processing (EIS) techniques employ gyroscope-based angular motion estimation, calculating the camera's pose change curve through integration, smoothing the pose trajectory, and finally compensating for video frames through image rotation. These methods typically assume that the camera's rotation center coincides with the imaging optical center, thus focusing primarily on correcting rotational errors. However, in certain applications, such as head-mounted devices (e.g., AR glasses), the imaging device is usually fixed in front of the head, with its rotation center located near the neck, rather than at the center of the camera's optical axis.

[0070] To address this technical problem, this embodiment calculates the rotation and translation compensation amounts for each image frame to construct a comprehensive compensation transformation. This transformation is then used to perform pose compensation on the image frames, eliminating image jitter caused by the misalignment of the device's rotation center and the imaging optical center. This not only corrects rotation errors but also considers translation errors, thereby significantly improving the stability and quality of images and videos from the head-mounted device in dynamic environments. The specific implementation process is described below.

[0071] Based on the smoothed pose trajectory, the rotation compensation and translation compensation for each image frame are calculated. The rotation compensation is calculated using the rotation matrix corresponding to the smoothed pose trajectory and the rotation matrix corresponding to the original pose trajectory. The translation compensation is calculated based on the spatial offset distance between the rotation center of the head-mounted device and the optical center of the image sensor, combined with the rotation angle in the smoothed pose trajectory. It can be understood that the compensation transformation includes both rotation and translation compensation.

[0072] Further, based on the smoothed attitude trajectory, a first rotation matrix corresponding to each image frame is determined; based on the original attitude trajectory, a second rotation matrix corresponding to each image frame is determined; based on the first rotation matrix and the second rotation matrix, the rotation compensation amount corresponding to each image frame is determined.

[0073] In one embodiment, a smoothed pose trajectory is obtained through Kalman filtering or other smoothing algorithms. This smoothed pose trajectory reflects the pose changes of the image sensor in the head-mounted device over a period of time. The smoothed pose trajectory includes a smoothed rotation matrix corresponding to each image frame. Information such as quaternions, Euler angles, position information (e.g., translation vectors), and timestamps can be directly extracted from the smoothed attitude trajectory. The first rotation matrix of the frame image is represented as: .

[0074] Similarly, the original attitude trajectory obtained through angular velocity integration contains the original rotation matrix corresponding to each image frame. Therefore, the first [object] can be extracted from the original attitude trajectory. The second rotation matrix of the frame image is represented as follows: .

[0075] For example, if the smoothed attitude trajectory and the original attitude trajectory are represented using quaternions, for the first... For each frame of the image, the rotation compensation amount is obtained by multiplying the first rotation matrix corresponding to the smoothed attitude trajectory with the second rotation matrix corresponding to the original attitude trajectory. :

[0076] in, It refers to the transpose of the second rotation matrix corresponding to the original pose trajectory, which is used to transform the image frame from the original pose to the smoothed pose.

[0077] This embodiment calculates the rotation compensation amount corresponding to each image frame by extracting the first rotation matrix in the smooth posture trajectory and the second rotation matrix in the original posture trajectory, thereby realizing pose compensation for the image frame, effectively reducing image jitter caused by device movement and improving image stability.

[0078] Further, the spatial offset distance between the rotation center of the head-mounted device and the optical center of the image sensor is obtained; based on the smooth posture trajectory, the rotation angle corresponding to each image frame is determined; based on the spatial offset distance and the rotation angle, the translation compensation amount is calculated.

[0079] The spatial offset distance, also known as the rotation radius, refers to the straight-line distance between the rotation center of the head-mounted device (such as the center of the wearer's neck or head) and the optical center of the image sensor. For example, the spatial offset distance can be determined through physical measurement, automatic calibration, or sensor measurement.

[0080] For physical measurement methods, the actual distance from the rotation center of the device to the image sensor can be measured. For example, the horizontal and vertical distances between the rotation center of the head-mounted device and the optical center of the image sensor can be measured with a ruler, and then the straight-line distance between the rotation center and the optical center can be calculated using the three-way theorem based on the horizontal and vertical distances.

[0081] In one embodiment, feature points are detected and tracked in a calibration image sequence to obtain the actual motion trajectory of the feature points; a mathematical model is constructed using the spatial offset distance between the rotation center of the head-mounted device and the optical center of the image sensor as the rotation radius, and the theoretical motion trajectory of the feature points is calculated through the mathematical model; an error function is defined to characterize the feature point position deviation between the actual motion trajectory and the theoretical motion trajectory; the rotation radius is adjusted to minimize the error function; when the change in the error function is lower than a preset threshold, the current rotation radius is determined as the spatial offset distance.

[0082] For automatic calibration methods, the rotation radius can be estimated by instructing the wearer to perform specific head movements, such as shaking their head left and right, and then analyzing the changes in feature points in the calibration image sequence. Specifically, while the wearer is wearing the head-mounted device, they are guided to perform specific head movements, such as shaking their head left and right. During these head movements, the head-mounted device continuously captures a series of calibration images, obtaining a calibration image sequence; simultaneously, it records inertial measurement data, such as angular velocity and acceleration, collected by sensors like gyroscopes and accelerometers. In the continuous calibration image sequence, computer vision algorithms (such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), or ORB (Oriented Fast and Rotated BRIEF)) are used to detect and track feature points. These feature points should be unique and reliably identifiable and trackable in consecutive image frames, such as a ball undergoing parabolic motion.

[0083] The motion trajectories of feature points in the calibration image sequence are analyzed, and the actual position of each feature point at each time point is calculated to construct the actual motion trajectory of the feature points. Sensor data (such as acceleration data, angular velocity data, etc.) are used to estimate the actual motion characteristics of the head, including rotation angle and velocity.

[0084] In one embodiment, the theoretical motion trajectory of the feature point due to the rotation of the head-mounted device is estimated based on the relationship between the actual motion characteristics of the head and the motion trajectory of the feature point.

[0085] A mathematical model is established to describe the evolution of the head-mounted device's rotational attitude over time. For rotational motion, the mathematical model can be represented using quaternion differential equations:

[0086] in, It is a quaternion. It is the angular velocity vector. This represents quaternion multiplication. It is the derivative of the quaternion with respect to time, representing the rate of attitude change. This mathematical model is used to guide how to update the state of the quaternion.

[0087] The mathematical model should include the rotation matrix and translation vector of the head-mounted device, as well as the spatial offset distance (rotation radius) between the rotation center and the optical center of the image sensor, to describe the influence of the rotation radius on the motion trajectory of the feature points.

[0088] Using quaternions Indicates time The rotation state at any given time can be specified. The initial quaternion corresponding to time 1 For this initial quaternion, the unit quaternion [1,0,0,0] represents no rotation. Set the time step. The time step determines the accuracy and computational cost of the integration calculation. A smaller time step can improve accuracy but increase computational cost.

[0089] Numerical methods (such as Euler integrals, Runge-Kutta methods, etc.) are used to integrate the rotational states represented by quaternions to solve for the theoretical trajectories of feature points. For example, using the Euler integral method:

[0090] in, Indicates time at The quaternion of rotation angle at any given time. Indicates time at A quaternion representing the rotation angle at any given moment; Indicates the time step, i.e., from Time's up The time interval; This is a coefficient used to adjust for the influence of angular velocity data. Indicates the current quaternion With angular velocity vector The product (usually represented as a quaternion), here This represents quaternion multiplication. Indicates time The angular velocity vector at time t is converted into quaternion form and then used to update the quaternion.

[0091] Repeat the above integral calculation for each time step, from From the initial time step to the final time step, the quaternion of the entire time series is constructed step by step. For the quaternion at each time step... , This refers to the transpose operation. The quaternion... Corresponding rotation matrix for:

[0092] Assume the initial position of the feature point in the world coordinate system is The spatial offset distance between the rotation center of the head-mounted device and the optical center of the image sensor is (Radius of rotation to be estimated).

[0093] For each time point The rotation matrix obtained using quaternion integration and spatial offset distance is Calculate the position of the feature point in the world coordinate system:

[0094] in, This indicates the position of the feature point in the world coordinate system. Representing quaternions The corresponding rotation matrix, Indicates spatial offset distance.

[0095] Known extrinsic parameters from device coordinate system to camera coordinate system and the rotation matrix obtained using quaternion integration Calculate the theoretical location of the feature points:

[0096] in, This indicates the position of the feature point in the world coordinate system. Representing quaternions The corresponding rotation matrix, It is a transpose operation. Indicates spatial offset distance. This indicates the theoretical position of the feature point in the camera coordinate system. This represents the known extrinsic parameters from the device coordinate system to the camera coordinate system. This represents the initial position of the feature point in the world coordinate system. The known extrinsic parameters from the device coordinate system to the camera coordinate system are also included. The position and orientation of the camera coordinate system relative to the device coordinate system are defined and can be obtained through calibration. Calibration can be achieved using relevant technical means in this field, which will not be described in detail in this embodiment.

[0097] Record the calculated theoretical position at each time point The above calculation process is repeated for each time point in the time series to obtain the theoretical motion trajectory of the head-mounted device.

[0098] Based on the theoretical and actual motion trajectories, the deviation between the theoretical and actual feature point positions at each time point is calculated. Then, an error function, such as mean squared error or mean absolute error, is defined to measure the deviation between the actual and theoretical motion trajectories. The error function is optimized to minimize the deviation between the theoretical and actual motion trajectories.

[0099] Choose an optimization algorithm, such as least squares, gradient descent, or genetic algorithm, and adjust the rotation radius through iterative optimization. The value of the value is determined to minimize the error function. In each iteration, a new rotation radius estimate is calculated based on the current rotation radius estimate and the gradient of the error function (i.e., the derivative of the error function with respect to the rotation radius). After each optimization execution, it is determined whether the optimization process has converged, i.e., whether the change in the error function is lower than a preset threshold (e.g., 1% of the initial value of the error function). If the change in the error function is lower than this preset threshold, the rotation radius is set to the current value. This refers to the spatial offset distance between the rotation center of the head-mounted device and the optical center of the image sensor.

[0100] This embodiment utilizes physical measurement or automatic calibration techniques to determine the rotation radius, and then simulates the theoretical motion trajectory of feature points through mathematical models and numerical integration methods. By comparing the theoretical trajectory and the actual trajectory, an error function is defined and optimized, ultimately accurately estimating the rotation radius to achieve accurate pose compensation for image frames, thereby improving the stability of images from the head-mounted device in dynamic scenes.

[0101] The rotation angle at each time point can be extracted from the smooth attitude trajectory. This angle can be obtained by applying the first rotation matrix. The angles are obtained by converting them to Euler angles or by extracting angles from quaternions. Rotation angles are usually expressed as rotations about three axes (usually the X, Y, and Z axes), corresponding to yaw, pitch, and roll, respectively.

[0102] Specifically, yaw angle It can be done through the first rotation matrix The elements were calculated to obtain:

[0103] in, , , It is the first rotation matrix The elements in the third row.

[0104] Pitch angle It can be calculated using the following formula:

[0105] in, , , It is the first rotation matrix The elements in the third row.

[0106] Roll angle It can be calculated using the following formula:

[0107] in, The first rotation matrix The first element in the second row; The first rotation matrix The first element in the first row.

[0108] In one embodiment, the rotation angle used in the translation compensation calculation formula refers to the rotation angle of the head-mounted device on a specific plane when it rotates around its center of rotation. In the head-mounted device, the specific plane is typically the plane where the wearer's neck or the center of their head is located.

[0109] Specifically, yaw angle Yaw is the rotation about an axis perpendicular to the ground (i.e., the object's vertical axis or Z-axis). The yaw angle describes the left and right rotation of the head, and its plane of rotation is the horizontal plane. Plane. Pitch angle. It refers to rotation about the horizontal axis (i.e., the left-right axis or X-axis of the object), while the pitch angle describes the vertical rotation of the head, and its corresponding plane of rotation is the vertical plane. Plane. Roll angle. It is the rotation about the longitudinal axis (i.e., the front-to-back axis or Y-axis of the object). The roll angle is used to describe the left and right roll of the head (i.e., the left and right swing of the target in front). Its corresponding plane of rotation is the horizontal plane. Plane. For example, if considering left and right head rotation, then choose... In a plane, the yaw angle is required. Calculate the translation compensation amount.

[0110] In practical applications, the head rotation of a wearer of a head-mounted device is usually not limited to a single plane. Therefore, the rotation angle corresponding to each image frame captured by the image sensor in a smooth attitude trajectory should include the yaw angle. Pitch angle and roll angle rotation angle vector :

[0111] in, This represents the rotation angle around the Z-axis, corresponding to the yaw angle; This represents the rotation angle around the Y-axis, corresponding to the pitch angle; This represents the rotation angle around the X-axis, corresponding to the roll angle.

[0112] Based on the above steps, the spatial offset distance and rotation angle corresponding to each image frame are calculated, and the translation compensation amount can then be calculated. Specifically, as shown... Figure 2 As shown, with the wearer's neck as the rotation center o, let the spatial offset distance (i.e., the rotation radius) between the rotation center and the optical center of the image sensor be . When the head rotates, the image sensor not only rotates with the head, but also changes due to the radius of rotation. This results in an additional translation. Assume the... The rotation angle of the smoothed camera pose relative to its original camera pose is: Then the translation compensation amount The translation compensation amount can be calculated using the following formula:

[0113] The translation compensation amount This indicates the translational displacement of the image sensor in space caused by head rotation.

[0114] The rotation compensation and translation compensation are integrated into a single transformation matrix to obtain the compensation transformation. :

[0115] The compensation transformation amount It is a homogeneous coordinate transformation matrix, where, Indicates the rotational compensation amount. This represents the translation compensation amount. The rotation compensation amount and the translation compensation amount together constitute the compensation transformation amount.

[0116] This embodiment extracts the rotation angle at each time point from the smoothed posture trajectory and combines it with the spatial offset distance between the rotation center of the head-mounted device and the optical center of the image sensor to calculate the image translation displacement caused by head rotation. This provides precise rotation and translation compensation for each frame of image, thereby significantly improving the image stabilization effect. Especially in dynamic environments, it can provide users with a clearer and more stable visual effect.

[0117] S105. Based on the compensation transformation amount corresponding to each image frame, pose compensation is performed on each image frame in sequence to obtain the processed image frame.

[0118] In one embodiment, during the image correction stage, based on the compensation transformation amount, a posture compensation transformation of rotation and translation is applied simultaneously to each image frame according to the stable pose, thereby obtaining a stable image frame sequence that remains visually stable and conforms to the rotational geometry of the human body.

[0119] For each image frame, use its corresponding compensation transform amount. A pose compensation transformation is performed to transform each pixel in the image frame from the original coordinate system to the compensated coordinate system. Specifically, for each pixel in the image frame, the following transformation formula is applied:

[0120] in, These are the pixel coordinates in the original image. Let K be the pixel coordinates in the transformed stable image, and K be the intrinsic parameter matrix of the image sensor. For the first The amount of frame compensation transformation.

[0121] Further, based on the compensation transformation amount corresponding to each image frame, pose compensation is sequentially performed on each image frame to obtain a pose-compensated image frame; non-integer pixel coordinates in each pose-compensated image frame are identified; based on an interpolation algorithm, interpolation calculation is performed on the pixel values ​​corresponding to each non-integer pixel coordinate to determine the interpolated pixel values ​​corresponding to each non-integer pixel coordinate; the interpolated pixel values ​​are applied to the non-integer pixel coordinates in each pose-compensated image frame to obtain the processed image frame.

[0122] During pose compensation transformation, the pixel coordinates after geometric transformation (rotation, translation) may not fall precisely on integer coordinates. Since image pixel values ​​are defined on integer coordinates, for non-integer pixel coordinates, interpolation algorithms (such as bilinear interpolation or bicubic interpolation) are needed to estimate the interpolated pixel value of the non-integer pixel coordinates by calculating a weighted average of the surrounding known pixel values.

[0123] In one embodiment, after applying pose compensation transformation (including rotation and translation transformation) to the image frame, each pixel in the transformed image frame is traversed, and the pixel coordinates are checked. Whether it is an integer. If any coordinate is not an integer, it is marked as a non-integer pixel coordinate. A list containing all non-integer pixel coordinates and their corresponding positions can be generated for subsequent interpolation processing.

[0124] Choose an appropriate interpolation method to interpolate the pixel values ​​at non-integer pixel coordinates. Interpolation methods can include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc.

[0125] In this algorithm, the nearest neighbor interpolation directly takes the pixel value at the integer coordinates closest to the target pixel as the interpolated pixel value. For example, if a pixel in an image frame is located at... The nearest integer coordinates are In nearest neighbor interpolation, the pixel coordinates in the image frame are taken as... The pixel value at that point is used as the pixel coordinate in the output stable image frame. The interpolated pixel value at the specified location.

[0126] The bilinear interpolation algorithm takes the four integer pixel coordinates closest to the target pixel and calculates the interpolated pixel value of the target pixel based on a distance-weighted average. For each non-integer pixel coordinate... Find the four nearest integer pixel coordinates around it. , , , And calculate non-integer pixel coordinates. Relative distance weights to these four points , usually used:

[0127]

[0128]

[0129]

[0130] Using these weights, a weighted average of the four pixel values ​​is calculated to determine the non-integer pixel coordinates. Interpolated pixel values ​​at:

[0131] in, Represents non-integer pixel coordinates Interpolated pixel values ​​at the location, , , , For non-integer pixel coordinates The four nearest integer pixel coordinates, , , , These are the pixel values ​​corresponding to these four integer pixel coordinates. These represent the relative distance weights corresponding to the four integer pixel coordinates.

[0132] For example, if a pixel in an image frame is located at The four integer coordinates of the pixels closest to this pixel are located at... , , , Let the pixel values ​​of these pixels be respectively , , , The formula for calculating bilinear interpolation is as follows:

[0133] in, Indicates that it is located at pixel coordinates The interpolated pixel values ​​of the pixels in the stable image frame. It refers to the location at pixel coordinates. The pixel coordinates are The distance weight of the pixels, It refers to the location at pixel coordinates. The pixel coordinates are The distance weight of the pixel, 0.3 means that the pixel at coordinates is The pixel coordinates are The distance weight of the pixel, 0.7 is located at pixel coordinates. The pixel coordinates are Distance weights of pixels.

[0134] The bicubic interpolation algorithm takes 16 integer pixel coordinates that are closest to the target pixel and uses a cubic polynomial function to calculate the interpolated pixel value of the target pixel:

[0135] in, and These are basis functions, typically expressed as cubic polynomials. express Rounding the axis coordinates down to the nearest integer. express The axis coordinates are rounded down to the nearest integer.

[0136] The calculated interpolated pixel values ​​are stored at their corresponding non-integer pixel coordinates. For each stable image frame after pose compensation transformation, interpolation is performed on the pixel values ​​at all its non-integer pixel coordinates. The processed stable image frames are then arranged in chronological order to form a stable image frame sequence.

[0137] Optionally, the stable image frame sequence is video encoded to generate the final stable video file.

[0138] This embodiment achieves pose compensation transformation of image frames by accurately calculating the rotation angle and translation compensation amount of the image sensor in the head-mounted device, thereby obtaining a visually stable image frame sequence that conforms to the rotational geometry of the human body. During the image correction stage, pose compensation transformations of rotation and translation are simultaneously applied to each image frame based on the compensation transformation amount, and non-integer pixel coordinates are processed using an interpolation algorithm to ensure image quality.

[0139] This embodiment provides an image processing method that accurately captures the motion state of a head-mounted device by acquiring angular velocity data during the shooting process. The angular velocity data is used to calculate the attitude change of the image sensor in three-dimensional space, thereby obtaining the original attitude trajectory and providing basic data for image stabilization. The original attitude trajectory is smoothed to eliminate noise and jitter, resulting in a more stable attitude trajectory and reducing image frame instability. Based on the smoothed attitude trajectory, rotation and translation compensation amounts are calculated for each image frame to generate a compensation transformation, simultaneously eliminating rotational and translational jitter in the image frames, making attitude compensation more accurate and effective. The compensation transformation is applied to the image frames to perform pose compensation transformation, thereby outputting stable image frames, effectively reducing image jitter caused by motion, resulting in a smoother and more stable final output sequence of stable image frames, significantly improving the effectiveness of image processing technology in eliminating jitter in images acquired by head-mounted devices.

[0140] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a first embodiment of an image processing apparatus provided in this application, which is used to perform the aforementioned image processing method.

[0141] like Figure 3 As shown, the image processing device 200 includes: an original attitude trajectory acquisition module 201, a smooth attitude trajectory acquisition module 202, a compensation transformation amount acquisition module 203, and an attitude compensation transformation module 204.

[0142] The original attitude trajectory acquisition module 201 is used to acquire the original attitude trajectory of the image sensor in three-dimensional space during the process of the head-mounted device acquiring image frames based on the image sensor. The smooth attitude trajectory acquisition module 202 is used to smooth the original attitude trajectory to obtain a smooth attitude trajectory. The compensation transformation amount acquisition module 203 is used to determine the compensation transformation amount corresponding to each image frame based on the smoothed attitude trajectory. The pose compensation transformation module 204 is used to perform pose compensation on each image frame sequentially based on the compensation transformation amount corresponding to each image frame, so as to obtain the processed image frame.

[0143] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing image processing method embodiments, and will not be repeated here.

[0144] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the head-mounted device shown.

[0145] Please see Figure 4 , Figure 4 This is a schematic block diagram of a head-mounted device provided in an embodiment of this application. The head-mounted device may be a server.

[0146] See Figure 4 The head-mounted device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0147] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any image processing method.

[0148] The processor provides computing and control capabilities to support the operation of the entire head-mounted device.

[0149] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform any image processing method.

[0150] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the head-mounted device to which the present application is applied. A specific head-mounted device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0151] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0152] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: During the process of acquiring image frames based on the image sensor of the head-mounted device, the original attitude trajectory of the image sensor in three-dimensional space is obtained; The original attitude trajectory is smoothed to obtain a smooth attitude trajectory; Based on the smoothed posture trajectory, the compensation transformation amount corresponding to each image frame is determined; Based on the compensation transformation amount corresponding to each image frame, pose compensation is performed on each image frame in sequence to obtain the processed image frame.

[0153] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the image processing methods provided in the embodiments of this application.

[0154] The computer-readable storage medium can be an internal storage unit of the head-mounted device described in the foregoing embodiments, such as the hard drive or memory of the head-mounted device. Alternatively, the computer-readable storage medium can be an external storage device of the head-mounted device, such as a plug-in hard drive, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard.

[0155] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: During the process of acquiring image frames based on the image sensor of the head-mounted device, the original attitude trajectory of the image sensor in three-dimensional space is obtained; The original attitude trajectory is smoothed to obtain a smooth attitude trajectory; Based on the smoothed posture trajectory, the compensation transformation amount corresponding to each image frame is determined; Based on the compensation transformation amount corresponding to each image frame, pose compensation is performed on each image frame in sequence to obtain the processed image frame.

2. The image processing method according to claim 1, characterized in that, The compensation transformation amount includes a rotation compensation amount; determining the compensation transformation amount corresponding to each image frame based on the smoothed attitude trajectory includes: Based on the smoothed pose trajectory, determine the first rotation matrix corresponding to each image frame; Based on the original pose trajectory, determine the second rotation matrix corresponding to each image frame; Based on the first rotation matrix and the second rotation matrix, the rotation compensation amount corresponding to each image frame is determined.

3. The image processing method according to claim 1, characterized in that, The compensation transformation amount includes a translation compensation amount; the step of calculating the compensation transformation amount corresponding to each image frame in the image frame sequence based on the smoothed attitude trajectory further includes: Obtain the spatial offset distance between the rotation center of the head-mounted device and the optical center of the image sensor; Based on the smooth posture trajectory, determine the rotation angle corresponding to each image frame; The translation compensation amount is determined based on the spatial offset distance and the rotation angle.

4. The image processing method according to claim 3, characterized in that, The step of obtaining the spatial offset distance between the rotation center of the head-mounted device and the optical center of the image sensor includes: Feature points are detected and tracked in a calibrated image sequence to obtain the actual motion trajectory of the feature points; Using the spatial offset distance between the rotation center of the head-mounted device and the optical center of the image sensor as the rotation radius, a mathematical model is constructed to calculate the theoretical motion trajectory of the feature point through the mathematical model; Define an error function, which is used to characterize the deviation of feature point positions between the actual motion trajectory and the theoretical motion trajectory; Adjust the rotation radius to minimize the error function; When the change in the error function is lower than a preset threshold, the current rotation radius is determined to be the spatial offset distance.

5. The image processing method according to claim 1, characterized in that, The step of smoothing the original attitude trajectory to obtain a smooth attitude trajectory includes: Obtain the rotation matrix corresponding to each image frame in the original pose trajectory; The rotation matrix corresponding to each image frame is smoothed by a filtering algorithm to obtain the filtered rotation matrix corresponding to each image frame. Extract the smooth pose value corresponding to each image frame from each of the filtered rotation matrices, and generate the smooth pose trajectory based on the smooth pose value.

6. The image processing method according to claim 1, characterized in that, The step of acquiring the original pose trajectory of the image sensor in three-dimensional space includes: The current motion state of the head-mounted device is detected based on a motion detection algorithm; Based on the current motion state, determine the sensor configuration parameters of the gyroscope sensor set in the head-mounted device; Based on the sensor configuration parameters, angular velocity data of the head-mounted device is collected during the process of capturing image frame sequences; Based on the angular velocity data, the attitude change of the image sensor in the head-mounted device in three-dimensional space is calculated to obtain the original attitude trajectory of the image sensor.

7. The image processing method according to claim 1, characterized in that, The step of sequentially performing pose compensation on each image frame based on the compensation transformation amount corresponding to each image frame to obtain the processed image frame includes: Based on the compensation transformation amount corresponding to each image frame, pose compensation is performed on each image frame in sequence to obtain the pose-compensated image frame. Identify the non-integer pixel coordinates in each pose-compensated image frame; Based on the interpolation algorithm, interpolation calculation is performed on the pixel values ​​corresponding to each of the non-integer pixel coordinates to determine the interpolated pixel values ​​corresponding to each of the non-integer pixel coordinates. The interpolated pixel value is applied to the non-integer pixel coordinates in each pose-compensated image frame to obtain the processed image frame.

8. An image processing apparatus, characterized in that, The image processing device includes: The original attitude trajectory acquisition module is used to acquire the original attitude trajectory of the image sensor in three-dimensional space during the process of the head-mounted device acquiring image frames based on the image sensor; A smooth attitude trajectory acquisition module is used to smooth the original attitude trajectory to obtain a smooth attitude trajectory; The compensation transformation amount acquisition module is used to determine the compensation transformation amount corresponding to each image frame based on the smoothed attitude trajectory. The pose compensation transformation module is used to sequentially perform pose compensation on each image frame based on the compensation transformation amount corresponding to each image frame, so as to obtain the processed image frame.

9. A head-mounted device, characterized in that, The head-mounted device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the image processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the image processing method as described in any one of claims 1 to 7.

Images