Video jitter determination method and video jitter processing method

Abstract

The invention discloses a video jitter determination method and a video jitter processing method, relates to the field of video image processing, and is applied to application scenes of fundus retina injection surgery, and the application scenes comprise an IMU (inertial measurement unit) and a microscope. The video jitter determination method comprises the following steps: acquiring a first initial acceleration and a first initial angular velocity of a microscope and a second initial acceleration and a second initial angular velocity of the head of a patient in real time through an IMU (Inertial Measurement Unit); extracting a first jitter frequency of the microscope and a second jitter frequency of the head of the patient based on the first initial acceleration, the first initial angular velocity, the second initial acceleration and the second initial angular velocity; a dither source is determined based on a first dither frequency of the microscope and a second dither frequency of the patient's head. According to the video jitter determination method, the multi-source jitter source in the actual fundus retina injection operation is comprehensively analyzed, the limitation of a single determination mode is effectively overcome, and a guarantee is provided for subsequently realizing multi-source jitter processing with higher distinguishing precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video image processing, in particular to a video jitter determination method and a video jitter processing method. BACKGROUND

[0002] Fundus retinal injection surgery is a high-precision ophthalmic surgery that operates on the retina and surrounding tissues. Due to the extremely small operation area, surgeons usually rely on surgical microscopes to obtain high-resolution real-time videos to guide the operation, postoperative analysis or for teaching training, etc. The clarity and stability of the surgical video are crucial to the success of the surgery, however, during the operation, the real-time surgical video is often disturbed by unexpected motion, i.e. jitter. These jitters not only affect the video quality and subsequent analysis, but also interfere with the accurate positioning of the injection needle, increasing the risk of damage to the retina.

[0003] The prior art mainly uses image stabilization algorithms and mechanical anti-shake systems to handle the jitter problem of ophthalmic surgery videos. Traditional image stabilization algorithms are usually based on digital image processing techniques, which detect and compensate for jitter in the video through methods such as optical flow analysis, feature point tracking, etc. Traditional mechanical anti-shake systems usually use mechanical anti-shake devices such as gyroscopic stabilization platforms to reduce the physical jitter of the microscope lens. However, the existing technology usually assumes that the jitter is mainly caused by a single source (such as microscope or patient movement), lacks comprehensive analysis capability for multi-source jitter in actual fundus retinal injection surgery, and cannot accurately separate jitter from different sources, thus limiting the de-jittering effect. SUMMARY

[0004] The present application aims to provide a video jitter determination method and a video jitter processing method to solve the above technical problems, comprehensively analyze the multi-source jitter in actual fundus retinal injection surgery, effectively overcome the limitations of single determination mode, and achieve more accurate jitter source, providing a guarantee for subsequent multi-source jitter processing with higher discrimination accuracy.

[0005] To solve the above problems, the present application provides a video jitter determination method, comprising the following steps:

[0006] real-time acquisition of the first initial acceleration and the first initial angular velocity of the microscope and the second initial acceleration and the second initial angular velocity of the patient's head through the IMU inertial measurement unit;

[0007] extracting the first jitter frequency of the microscope and the second jitter frequency of the patient's head based on the first initial acceleration, the first initial angular velocity, the second initial acceleration and the second initial angular velocity;

[0008] determining the jitter source based on the first jitter frequency of the microscope and the second jitter frequency of the patient's head.

[0009] The first initial acceleration, the first initial angular velocity of the microscope, the second initial acceleration and the second initial angular velocity of the patient's head are obtained in real time through the IMU in the scheme, the first jitter frequency of the microscope and the second jitter frequency of the patient's head are extracted through the obtained first initial acceleration, first initial angular velocity, second initial acceleration and second initial angular velocity, and the jitter source is determined through the first jitter frequency of the microscope and the second jitter frequency of the patient's head. Through the technical scheme, the jitters from the microscope and the patient's head in the actual fundus retina injection surgery can be accurately distinguished and comprehensively analyzed, the decomposition capability of the multi-frequency jitter is improved, then the motion of different sources is effectively separated in the frequency domain, so that the jitter source is more accurately determined, and the guarantee for the subsequent multi-source jitter processing with higher distinguishing precision is provided.

[0010] Further, the video jitter determination method provided by the application further comprises the following steps:

[0011] real-time video stream data is obtained through the microscope;

[0012] the optical flow vector field of all pixels in each frame of video is calculated based on the real-time video stream data;

[0013] contour extraction is performed based on the optical flow vector field of all pixels in each frame of video, the global motion amplitude and the contour motion amplitude are calculated;

[0014] whether the video stream is jittered is determined based on the global motion amplitude and the motion amplitude in the contour.

[0015] In the scheme, the optical flow vector field of all pixels in each frame of video is calculated based on the real-time video stream data obtained through the microscope, contour extraction is performed based on the optical flow vector field of all pixels in each frame of video, the global motion amplitude and the contour motion amplitude are calculated, and finally whether the video stream is jittered is determined based on the global motion amplitude and the motion amplitude in the contour. Through the technical scheme, whether the video captured by the microscope in the actual fundus retina injection surgery is locally jittered can be effectively identified, and the stability of the surgical video is effectively improved.

[0016] Further, the first jitter frequency of the microscope and the second jitter frequency of the patient's head are extracted based on the first initial acceleration, the first initial angular velocity, the second initial acceleration and the second initial angular velocity, comprising:

[0017] data preprocessing is performed based on the first initial acceleration, the first initial angular velocity, the second initial acceleration and the second initial angular velocity, to obtain a first target acceleration, a first target angular velocity, a second target acceleration and a second target angular velocity;

[0018] The first acceleration frequency spectrum, the first angular velocity frequency spectrum, the second acceleration frequency spectrum and the second angular velocity frequency spectrum are obtained by respectively performing fast Fourier transform on the first target acceleration, the first target angular velocity, the second target acceleration and the second target angular velocity.

[0019] The first jitter frequency of the microscope is obtained based on the first acceleration frequency spectrum and the first angular velocity frequency spectrum.

[0020] The second jitter frequency of the patient's head is obtained based on the second acceleration frequency spectrum and the second angular velocity frequency spectrum.

[0021] In the above scheme, the frequency components of the first target acceleration, the first target angular velocity, the second target acceleration and the second target angular velocity are extracted by fast Fourier transform, which effectively suppresses the noise interference in data transmission and effectively improves the jitter analysis efficiency.

[0022] Further, the first jitter frequency of the microscope and the second jitter frequency of the patient's head are used to determine the jitter source, including:

[0023] When the jitter frequency falls within the preset microscope jitter frequency range, it is judged that the jitter is from the microscope.

[0024] When the jitter frequency falls within the preset patient head jitter frequency range, it is judged that the jitter is from the patient's head.

[0025] In the above scheme, the microscope jitter is usually low-frequency vibration (such as 0.1-1Hz), and the patient's head jitter is usually medium-high frequency vibration (such as 1-10Hz). By presetting the frequency range of the microscope jitter and the patient's head jitter, the multi-source jitter can be decomposed, the actual fundus retinal surgery microscope and the patient's head micro-jitter can be effectively distinguished, and the jitter removal effect is effectively improved.

[0026] Further, the video jitter determination method provided by the application further includes the following steps:

[0027] The first acceleration amplitude and the first angular velocity amplitude are calculated based on the first target acceleration and the first target angular velocity.

[0028] The second acceleration amplitude and the second angular velocity amplitude are calculated based on the second target acceleration and the second target angular velocity.

[0029] The acceleration threshold comparison result is obtained by comparing the first acceleration amplitude and the second acceleration amplitude with the preset acceleration threshold.

[0030] The angular velocity threshold comparison result is obtained by comparing the first angular velocity amplitude and the second angular velocity amplitude with the preset angular velocity threshold.

[0031] when the acceleration threshold comparison result is that the first acceleration amplitude is greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result is that the first angular velocity amplitude is greater than or equal to the preset angular velocity threshold, and the video stream is jittered, it is confirmed that the microscope is jittered;

[0032] when the acceleration threshold comparison result is that the second acceleration amplitude is greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result is that the second angular velocity amplitude is greater than or equal to the preset angular velocity threshold, and the video stream is jittered, it is confirmed that the patient's head is jittered;

[0033] when the acceleration threshold comparison result is that the first acceleration and the second acceleration are both not greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result is that the first angular velocity and the second angular velocity are both not greater than or equal to the preset angular velocity threshold, and the video stream is jittered, it is confirmed that the video light flow is jittered;

[0034] when the acceleration threshold comparison result is that the first acceleration and the second acceleration are both greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result is that the first angular velocity and the second angular velocity are both greater than or equal to the preset angular velocity threshold, and the video stream is jittered, it is confirmed that the global jitter is confirmed.

[0035] In the above scheme, the jitter conditions of the microscope and the patient's head are further determined by calculating the acceleration and angular velocity amplitudes, and the jitter source in the actual fundus retinal injection surgery is distinguished by combining the jitter condition of the video stream. The above scheme provides an effective separation mechanism, fully utilizes the high robustness (not affected by optical factors) of the IMU and the high spatial resolution of the light flow, realizes the complementary advantages, effectively improves the effect of realizing the multi-source jitter distinction, and breaks through and overcomes the limitations of a single mode.

[0036] The application also provides a video jitter processing method, which performs jitter processing according to the jitter result determined by the video jitter determination method, and the jitter result includes microscope jitter, patient head jitter, video light flow jitter and global jitter. The video jitter processing method includes: when it is detected that the microscope is jittered, performing translation compensation and rotation compensation on the video stream; when it is detected that the patient's head is jittered, adjusting and aligning the video stream; and when it is detected that the video light flow is jittered, performing correction processing on the video stream.

[0037] In the above scheme, the multiple source jitter sources are targeted for jitter elimination measures, effectively solving the multiple jitter superimposed motion that may exist simultaneously in the actual fundus retinal injection surgery. By separately processing different jitter sources, such as globally compensating the jitter data of the microscope and the patient's head, and locally correcting the video stream using optical flow data. This mechanism of effectively separating jitter sources and compensating different jitter sources effectively overcomes the difference in jitter characteristics and ensures efficient and stable jitter elimination effect.

[0038] Further, when detecting that the microscope is jittered, the translation compensation and rotation compensation on the video stream include:

[0039] calculating a first motion trajectory of the microscope based on the first target acceleration and the first target angular velocity;

[0040] calculating a first motion offset and a first rotation angle of the microscope based on the first motion trajectory;

[0041] performing translation compensation on the video stream based on the first motion offset;

[0042] performing rotation compensation on the video stream based on the first rotation angle.

[0043] In the above scheme, the jitter of the microscope often occurs in the horizontal or vertical direction and has a large displacement order of magnitude (such as millimeter level), and through translation compensation and rotation compensation, dynamic correction of jitter of the microscope in actual fundus retinal injection surgery is effectively realized, and the jitter elimination effect is greatly improved.

[0044] Further, when detecting that the patient's head is jittered, the adjustment alignment on the video stream includes:

[0045] calculating a second motion trajectory of the patient's head based on the second target acceleration and the second target angular velocity;

[0046] calculating a second motion offset of the patient's head based on the second motion trajectory;

[0047] performing adjustment alignment on the video stream based on the second motion offset.

[0048] In the above scheme, the global offset of the patient's head is directly mapped using IMU data, and by presetting the center point position of the surgical field of view, the video stream is adjusted and aligned in combination with the global offset of the patient's head, and the center point position is updated in real time, effectively solving the jitter of the patient's head in actual fundus retinal injection surgery and greatly improving the jitter elimination effect for the patient's head.

[0049] Further, when detecting that the video optical flow is jittered, the correction processing on the video stream includes:

[0050] extracting a contour based on the optical flow vector field of all pixels in each frame of video, calculating the average value of the optical flow field in the contour;

[0051] generating a correction region based on the average value of the optical flow field in the contour, and correcting the video stream based on the correction region.

[0052] In the above scheme, the video stream jitter is locally corrected by the optical flow average vector, which can meet the real-time requirements in actual fundus retinal injection surgery, and avoids complex calculation. While ensuring the de-jitter effect, the calculation amount is effectively reduced, and the delay effect when processing jitter is reduced

[0053] Further, the calculation of the first motion trajectory of the microscope based on the first target acceleration and the first target angular velocity comprises:

[0054] Based on the first target acceleration, a second integral is performed to obtain the real-time displacement of the microscope in different direction axes;

[0055] Based on the first target angular velocity, a first integral is performed to obtain the real-time rotation angle of the microscope around different direction axes;

[0056] Based on the real-time displacement of the microscope in different directions and the real-time rotation angle of the microscope around different direction axes, the first motion trajectory of the microscope is obtained.

[0057] Further, the calculation of the second motion trajectory of the patient's head based on the second target acceleration and the second target angular velocity comprises:

[0058] Based on the second target acceleration, a second integral is performed to obtain the real-time displacement of the patient's head in different direction axes;

[0059] Based on the second target angular velocity, a first integral is performed to obtain the real-time rotation angle of the patient's head around different direction axes;

[0060] Based on the real-time displacement of the patient's head in different directions and the real-time rotation angle of the patient's head around different direction axes, the second motion trajectory of the patient's head is obtained.

[0061] In the above scheme, the displacement and rotation angle of the microscope are obtained by analyzing and calculating the first target acceleration and the first target angular velocity data, and the displacement and rotation angle of the patient's head are obtained by analyzing and calculating the second target acceleration and the second target angular velocity. The high sampling rate of the IMU inertial measurement unit is used to capture continuous motion data in real time, which provides a basis for high time resolution for subsequent jitter detection.

[0062] Further, the video jitter processing method provided by the application further comprises:

[0063] When the acceleration threshold comparison result is that the first acceleration and the second acceleration are both greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result is that the first angular velocity and the second angular velocity are both greater than or equal to the preset angular velocity threshold, but the video stream does not occur jitter, it is confirmed that a fault occurs, and the IMU inertial measurement unit and the microscope need to be checked.

[0064] In the above scheme, the acceleration threshold comparison result, the angular velocity threshold comparison result and the real-time jitter condition of the video stream are used to capture the abnormality of the equipment used in the actual fundus retinal injection operation, so as to effectively maintain the safety of the equipment and reduce the maintenance cost. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 A flowchart of a method for determining frequency jitter of the microscope and the patient's head is provided for an embodiment of the present application;

[0066] Figure 2 A flowchart of a method for determining video stream jitter is provided for an embodiment of the present application;

[0067] Figure 3 A flowchart of a method for determining multi-source jitter is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0069] Please refer to Figure 1 The present embodiment provides a video jitter determination method, comprising the following steps:

[0070] Step S1: Real-time acquisition of the first initial acceleration and the first initial angular velocity of the microscope and the second initial acceleration and the second initial angular velocity of the patient's head by the IMU inertial measurement unit;

[0071] Step S2: Extraction of the first jitter frequency of the microscope and the second jitter frequency of the patient's head based on the first initial acceleration, the first initial angular velocity, the second initial acceleration and the second initial angular velocity;

[0072] Step S3: Determination of the jitter source based on the first jitter frequency of the microscope and the second jitter frequency of the patient's head.

[0073] In the embodiment, the first initial acceleration and the first initial angular velocity of the microscope and the second initial acceleration and the second initial angular velocity of the patient's head are obtained in real time through an IMU (inertial measurement unit); the first jitter frequency of the microscope and the second jitter frequency of the patient's head are extracted based on the obtained first initial acceleration, first initial angular velocity, second initial acceleration and second initial angular velocity; and the jitter source is determined based on the first jitter frequency of the microscope and the second jitter frequency of the patient's head. Through the technical solution, the jitters from the microscope and the patient's head in an actual fundus retina injection surgery can be accurately distinguished and comprehensively analyzed, the decomposition capability for multi-frequency jitters is improved, the motion from different sources is effectively separated in the frequency domain, and thus the jitter source can be more accurately determined, thereby providing a guarantee for subsequent multi-source jitter processing with higher distinguishing precision.

[0074] See Figure 2 The embodiment provides a video jitter determination method, and further includes the following steps:

[0075] Step S1: obtaining real-time video stream data through a microscope;

[0076] Step S2: calculating an optical flow vector field of all pixels in each frame of video based on the real-time video stream data;

[0077] Step S3: performing contour extraction based on the optical flow vector field of all pixels in each frame of video, and calculating a global motion amplitude and a contour motion amplitude;

[0078] Step S4: determining whether jitter occurs in the video stream based on the global motion amplitude and the motion amplitude in the contour.

[0079] In the embodiment, the optical flow vector field of all pixels in each frame of video is calculated based on real-time video stream data obtained through a microscope; the global motion amplitude and the contour motion amplitude are calculated by performing contour extraction based on the optical flow vector field of all pixels in each frame of video; and finally, whether jitter occurs in the video stream is determined based on the global motion amplitude and the motion amplitude in the contour. Through the technical solution, whether local jitter occurs in the video captured by the microscope in an actual fundus retina injection surgery can be effectively identified, and the stability of the surgical video is effectively improved.

[0080] Further, the first jitter frequency of the microscope and the second jitter frequency of the patient's head are extracted based on the first initial acceleration, the first initial angular velocity, the second initial acceleration and the second initial angular velocity, including:

[0081] data preprocessing is performed based on the first initial acceleration, the first initial angular velocity, the second initial acceleration and the second initial angular velocity, to obtain a first target acceleration, a first target angular velocity, a second target acceleration and a second target angular velocity;

[0082] obtaining a first acceleration frequency spectrum, a first angular velocity frequency spectrum, a second acceleration frequency spectrum and a second angular velocity frequency spectrum by performing fast Fourier transform on the first target acceleration, the first target angular velocity, the second target acceleration and the second target angular velocity respectively;

[0083] obtaining a first jitter frequency of the microscope based on the first acceleration frequency spectrum and the first angular velocity frequency spectrum;

[0084] obtaining a second jitter frequency of the patient's head based on the second acceleration frequency spectrum and the second angular velocity frequency spectrum.

[0085] In the embodiment, the frequency components of the first target acceleration, the first target angular velocity, the second target acceleration and the second target angular velocity are extracted by fast Fourier transform, which effectively suppresses the noise interference in data transmission and effectively improves the jitter analysis efficiency.

[0086] Further, the first jitter frequency of the microscope and the second jitter frequency of the patient's head are used to determine the jitter source, including:

[0087] when the jitter frequency falls within a preset microscope jitter frequency range, it is determined that the jitter is from the microscope;

[0088] when the jitter frequency falls within a preset patient head jitter frequency range, it is determined that the jitter is from the patient's head.

[0089] In the embodiment, the microscope jitter is usually low-frequency vibration (such as 0.1-1Hz), and the patient's head jitter is usually medium-high frequency vibration (such as 1-10Hz). By presetting the frequency ranges of the microscope jitter and the patient's head jitter, the multi-source jitter can be decomposed, the actual microscopic and patient's head jitter in the fundus retinal surgery can be effectively distinguished, and the jitter removal effect is effectively improved.

[0090] Please refer to Figure 3 The embodiment provides a video jitter determination method, which further includes the following steps:

[0091] Step S1: calculating a first acceleration amplitude and a first angular velocity amplitude based on the first target acceleration and the first target angular velocity;

[0092] Step S2: calculating a second acceleration amplitude and a second angular velocity amplitude based on the second target acceleration and the second target angular velocity;

[0093] Step S3: comparing the first acceleration amplitude and the second acceleration amplitude with a preset acceleration threshold to obtain an acceleration threshold comparison result;

[0094] Step S4: comparing the first angular velocity amplitude and the second angular velocity amplitude with a preset angular velocity threshold value to obtain an angular velocity threshold comparison result;

[0095] Step S5: determining the shaking condition based on the acceleration threshold comparison result, the angular velocity threshold comparison result and the video stream shaking condition.

[0096] Specifically, the determination of the shaking condition based on the acceleration threshold comparison result, the angular velocity threshold comparison result and the video stream shaking condition includes:

[0097] When the acceleration threshold comparison result is that the first acceleration amplitude is greater than or equal to the preset acceleration threshold value, the angular velocity threshold comparison result is that the first angular velocity amplitude is greater than or equal to the preset angular velocity threshold value, and the video stream is shaking, it is determined that the microscope is shaking;

[0098] When the acceleration threshold comparison result is that the second acceleration amplitude is greater than or equal to the preset acceleration threshold value, the angular velocity threshold comparison result is that the second angular velocity amplitude is greater than or equal to the preset angular velocity threshold value, and the video stream is shaking, it is determined that the patient's head is shaking;

[0099] When the acceleration threshold comparison result is that the first acceleration and the second acceleration are both not greater than or equal to the preset acceleration threshold value, the angular velocity threshold comparison result is that the first angular velocity and the second angular velocity are both not greater than or equal to the preset angular velocity threshold value, and the video stream is shaking, it is determined that the video stream is shaking;

[0100] When the acceleration threshold comparison result is that the first acceleration and the second acceleration are both greater than or equal to the preset acceleration threshold value, the angular velocity threshold comparison result is that the first angular velocity and the second angular velocity are both greater than or equal to the preset angular velocity threshold value, and the video stream is shaking, it is determined that the shaking is global.

[0101] In this embodiment, the shaking conditions of the microscope and the patient's head are further determined by calculating the acceleration and angular velocity amplitudes, and the source of shaking in the actual fundus retinal injection surgery is distinguished by combining the shaking condition of the video stream. The above scheme provides an effective separation mechanism, which fully utilizes the high robustness of IMU (not affected by optical factors) and the high spatial resolution of optical flow, realizes complementary advantages, effectively improves the effect of realizing multi-source shaking differentiation, and breaks through and overcomes the limitations of single mode.

[0102] The embodiment also provides a jitter processing method according to the jitter result determined by the video jitter determination method. The jitter result includes microscope jitter, patient head jitter, video optical flow jitter and global jitter. The video jitter processing method includes: when detecting that the microscope is jittered, performing translation compensation and rotation compensation on the video stream; when detecting that the patient head is jittered, adjusting and aligning the video stream; and when detecting that the video optical flow is jittered, performing correction processing on the video stream.

[0103] In the embodiment, the targeted jitter elimination measures are taken for the multiple jitter sources, and the multiple jitter superimposed motions possibly existing simultaneously in the actual fundus retina injection surgery are effectively solved. By processing different jitter sources respectively, for example, the global compensation is performed on the jitter data of the microscope and the patient head, and the local correction is performed on the video stream using the optical flow data. The mechanism of effectively separating jitter sources and taking targeted compensation for different jitter sources effectively overcomes the jitter characteristic difference, and ensures efficient and stable jitter elimination effect.

[0104] Further, when detecting that the microscope is jittered, the translation compensation and the rotation compensation on the video stream include:

[0105] calculating a first motion trajectory of the microscope based on the first target acceleration and the first target angular velocity;

[0106] calculating a first motion offset and a first rotation angle of the microscope based on the first motion trajectory;

[0107] performing translation compensation on the video stream based on the first motion offset;

[0108] performing rotation compensation on the video stream based on the first rotation angle.

[0109] In the embodiment, the jitter of the microscope often occurs in the horizontal or vertical direction and has a large displacement order (such as millimeter level), and the dynamic correction of the jitter of the microscope in the actual fundus retina injection surgery is effectively realized by the translation compensation and the rotation compensation, and the jitter elimination effect is greatly improved.

[0110] Further, when detecting that the patient head is jittered, the adjustment and alignment of the video stream include:

[0111] calculating a second motion trajectory of the patient head based on the second target acceleration and the second target angular velocity;

[0112] calculating a second motion offset of the patient head based on the second motion trajectory;

[0113] performing adjustment and alignment on the video stream based on the second motion offset.

[0114] In the embodiment, the global offset of the patient's head is directly mapped by using the IMU data, the center point position of the preset surgical field is combined with the global offset of the patient's head to adjust and align the video stream, and the center point position is updated in real time, so that the shaking of the patient's head in the actual fundus retinal injection surgery is effectively solved, and the de-shaking effect for the patient's head is greatly improved.

[0115] Further, when the video light flow is detected to shake, the video stream is corrected, and the method comprises the steps that:

[0116] The contour is extracted based on the light flow vector field of all pixels in each frame of video, and the average value of the light flow field in the contour is calculated.

[0117] The correction area is generated based on the average value of the light flow field in the contour, and the video stream is corrected based on the correction area.

[0118] In the embodiment, the video stream shaking is locally corrected by the light flow average vector, so that the real-time requirement in the actual fundus retinal injection surgery is met, and complex calculation is avoided. While ensuring the de-shaking effect, the calculation amount is effectively reduced, and the delay effect when processing shaking is reduced.

[0119] Further, the first motion trajectory of the microscope is calculated based on the first target acceleration and the first target angular velocity, and the method comprises the steps that:

[0120] The real-time displacement of the microscope in different direction axes is obtained by twice integration based on the first target acceleration;

[0121] The real-time rotation angle of the microscope around different direction axes is obtained by once integration based on the first target angular velocity;

[0122] The first motion trajectory of the microscope is obtained based on the real-time displacement of the microscope in different directions and the real-time rotation angle of the microscope around different direction axes.

[0123] Further, the second motion trajectory of the patient's head is calculated based on the second target acceleration and the second target angular velocity, and the method comprises the steps that:

[0124] The real-time displacement of the patient's head in different direction axes is obtained by twice integration based on the second target acceleration;

[0125] The real-time rotation angle of the patient's head around different direction axes is obtained by once integration based on the second target angular velocity;

[0126] The second motion trajectory of the patient's head is obtained based on the real-time displacement of the patient's head in different directions and the real-time rotation angle of the patient's head around different direction axes.

[0127] In the embodiment, the displacement and rotation angle of the microscope are obtained by analyzing and calculating the first target acceleration and first target angular velocity data, and the displacement and rotation angle of the patient's head are obtained by analyzing and calculating the second target acceleration and second target angular velocity, and the high sampling rate of the IMU inertial measurement unit is used to capture continuous motion data in real time, which provides a basis for high time resolution for subsequent jitter detection.

[0128] Further, the video jitter processing method provided in the embodiment further includes: when the acceleration threshold comparison result is that the first acceleration and the second acceleration are both greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result is that the first angular velocity and the second angular velocity are both greater than or equal to the preset angular velocity threshold, but the video stream does not occur jitter, it is confirmed that a fault occurs, and the IMU inertial measurement unit and the microscope need to be checked.

[0129] In the embodiment, the acceleration threshold comparison result, the angular velocity threshold comparison result and the real-time jitter of the video stream are used to capture abnormalities of the equipment used in the actual fundus retinal injection surgery, effectively maintain the safety of the equipment and reduce the maintenance cost.

[0130] In order to more clearly illustrate the video jitter determination method provided by the above scheme, and the jitter processing method provided by the jitter result determined according to the video jitter determination method, and highlight its technical advantages, the following will be described with an actual example.

[0131] In order to meet the high-precision requirements of the actual fundus retinal injection surgery, the sampling frequency of the IMU inertial measurement unit used needs to reach more than 100 Hz to capture high-frequency jitter, and the module volume needs to be small enough (size < 10 mm x 10 mm x 2 mm, weight < 1 g) to be integrated into the microscope and the patient head wearing device without affecting the operation or patient comfort. The IMU inertial measurement unit used adopts a low-power mode (preferably working current < 5 mA), supports I2C or SPI interface, and is convenient for communication with the control unit of the surgery system. In addition, the accelerometer and gyroscope in the IMU inertial measurement unit need to have low noise characteristics (preferably accelerometer noise < 100 μg / √Hz, gyroscope noise < 0.01° / s / √Hz) to ensure accurate detection of small jitter signals.

[0132] In the embodiment, Bosch BMI160 inertial measurement module with an acceleration range of ±2g, a gyroscope range of ±125° / s and a sampling rate of up to 400Hz is preferably used to measure the related data of the microscope and the patient's head.

[0133] The microscope IMU inertial measurement unit is fixed and installed on the lens holder or main body structure of the surgical microscope by high-strength medical glue or a micro clamping device, close to the lens to directly measure the physical movement of the lens, and the coordinate system of the microscope IMU inertial measurement unit needs to be aligned with the optical axis of the microscope during installation to avoid coordinate conversion error; wherein the X, Y and Z axes of the microscope IMU inertial measurement unit correspond to the horizontal, vertical and depth directions of the microscope respectively.

[0134] In this embodiment, the microscope IMU inertial measurement unit is used to monitor the physical movement of the microscope in real time, including mechanical vibration caused by the operating table or the external environment, displacement caused by the operator adjusting the position of the microscope, etc.

[0135] The patient head IMU inertial measurement unit is fixed on the wearable device by medical grade headband or adhesive, and the patient head IMU inertial measurement unit is installed on the patient's head through the wearable device (such as headband), and the patient head IMU inertial measurement unit should be as close to the center of gravity of the head (such as the forehead or temple position) as possible to accurately capture the micro movement of the head and eyeball; wherein the X axis of the patient head IMU inertial measurement unit corresponds to the front and back direction of the patient's head, the Y axis corresponds to the left and right direction of the patient's head, and the Z axis corresponds to the up and down direction of the patient's head.

[0136] In this embodiment, the patient head IMU inertial measurement unit is used to capture the micro movement of the head or eyeball, such as the head swing caused by the patient's discomfort or the natural movement of the eyeball, etc., which may cause global jitter in the actual fundus retinal injection surgery video.

[0137] After the installation and deployment are completed, it is tested whether the IMU inertial measurement unit data has abnormal fluctuation by slightly moving the microscope or simulating the patient's head micro movement, and the baseline parameters of the IMU inertial measurement unit under different temperature (20-30℃) and humidity (40-60%) conditions are recorded before the operation starts.

[0138] The first initial acceleration and the first initial angular velocity of the microscope and the second initial acceleration and the second initial angular velocity of the patient's head are obtained in real time by the IMU inertial measurement unit. Wherein, the first initial acceleration includes a x1 (t), a y1 (t), a z1 (t), the first initial angular velocity includes w x1 (t), w y1 (t), w z1 (t), the second initial acceleration includes a x2 (t), a y2 (t), a z2 (t) and the second initial angular velocity includes w x2 (t), w y2(t), w z2 (t); wherein the first initial acceleration and the second initial acceleration are in units of m / s 2 , for reflecting linear motion of the device (such as translational jitter of a microscope or displacement of a patient's head); the first initial angular velocity and the second initial angular velocity are in units of rad / s, for reflecting rotational motion of the device (such as tilting of a microscope or rotation of a patient's head).

[0139] performing data preprocessing based on the first initial acceleration, the first initial angular velocity, the second initial acceleration, and the second initial angular velocity to obtain a first target acceleration, a first target angular velocity, a second target acceleration, and a second target angular velocity; wherein the data preprocessing comprises:

[0140] performing smoothing processing on the first initial acceleration, the first initial angular velocity, the second initial acceleration, and the second initial angular velocity using a low-pass filter (with a cutoff frequency preferably being 10 Hz);

[0141] subtracting zero-point bias correction from the obtained data based on reference parameters of the IMU inertial measurement unit under different temperature (20-30℃) and humidity (40-60%) conditions, i.e., acceleration, angular velocity zero-point offset in a stationary state;

[0142] In addition, if the IMU inertial measurement unit coordinate system is inconsistent with the global coordinate system of the surgical system, coordinate conversion also needs to be performed on the obtained real-time data.

[0143] In the embodiment, by performing data preprocessing on the obtained real-time data, noise is effectively removed, and the accuracy of jitter determination is greatly improved.

[0144] performing fast Fourier transform based on the first target acceleration, the first target angular velocity, the second target acceleration, and the second target angular velocity to extract a jitter frequency; wherein the first target acceleration comprises a x1 '(t), a y1 '(t), a z1 '(t), the first target angular velocity comprises w x1 '(t), w y1 '(t), w z1 '(t), the second target acceleration comprises a x2 '(t), a y2 '(t), a z2 '(t), the second target angular velocity comprises w x2 '(t), w y2 '(t), w z2 '(t).

[0145] Taking the first target acceleration in the X-axis direction as an example, the specific formula for extracting the frequency spectrum data by fast Fourier transform is as follows:

[0146] A x1 (f)=FFT[a x1 '(t)]

[0147] Wherein, A x1 (f) is the frequency spectrum data of the first target acceleration in the X-axis direction, with the unit of m / s 2 / Hz.

[0148] Taking the first target angular velocity in the X-axis direction as an example, the specific formula for extracting the frequency spectrum data by fast Fourier transform is as follows:

[0149] Ω x1 (f)=FFT[w x1 '(t)]

[0150] Wherein, Ω x1 (f) is the frequency spectrum data of the first target angular velocity in the X-axis direction, with the unit of rad / s / Hz.

[0151] The time domain signal is converted into a frequency domain signal by fast Fourier transform, to obtain the frequency spectrum data of the first target acceleration, the first target angular velocity, the second target acceleration and the second target angular velocity; by analyzing the frequency spectrum, the main frequency range of the jitter is determined (preferably, the microscope jitter is concentrated in the low frequency range of 0.1-1 Hz, and the patient head jitter is concentrated in the medium-high frequency range of 1-10 Hz), when the jitter frequency falls within the preset microscope jitter frequency range, it is judged that the jitter is from the microscope; when the jitter frequency falls within the preset patient head jitter frequency range, it is judged that the jitter is from the patient head.

[0152] In this embodiment, the frequency ranges of the preset microscope jitter and patient head jitter can be used to decompose the multi-source jitter, effectively distinguish the slight jitters of the microscope and the patient head in the actual fundus retina surgery, and further effectively improve the jitter removal effect.

[0153] The first acceleration amplitude and the first angular velocity amplitude are calculated based on the first target acceleration and the first target angular velocity; the second acceleration amplitude and the second angular velocity amplitude are calculated based on the second target acceleration and the second target angular velocity.

[0154] Taking the data of the first target acceleration in the X-axis direction as an example, the specific calculation formula is as follows:

[0155]

[0156] Wherein |a| is the amplitude of the first target acceleration at a certain time, a x1'、a x2 '、a x3 ' is the acceleration value of the first target acceleration along the X-axis, Y-axis, and Z-axis at a certain moment;

[0157] Taking the data of the first target angular velocity in the X-axis direction as an example, the specific calculation formula is as follows:

[0158]

[0159] Where |w| is the amplitude of the first target angular velocity at a certain moment, w x1 '、w x2 '、w x3 ' is the acceleration value of the first target angular velocity along the X-axis, Y-axis, and Z-axis at a certain moment;

[0160] Comparing the first acceleration amplitude and the second acceleration amplitude with a preset acceleration threshold to obtain an acceleration threshold comparison result;

[0161] Based on the comparison between the first angular velocity amplitude and the second angular velocity amplitude and the preset angular velocity threshold, an angular velocity threshold comparison result is obtained; specifically:

[0162] When the first acceleration amplitude is greater than or equal to a preset acceleration threshold, it is determined that the microscope is shaking;

[0163] When the second acceleration amplitude is greater than or equal to a preset acceleration threshold, it is determined that the patient's head is shaking;

[0164] When the first angular velocity amplitude is greater than or equal to a preset angular velocity threshold, it is determined that the microscope is shaking;

[0165] When the second angular velocity amplitude is greater than or equal to a preset angular velocity threshold, it is determined that the patient's head is shaking;

[0166] In this embodiment, by setting a threshold in advance (preferably the microscope acceleration threshold is T a1 = 0.2g, the microscope angular velocity threshold is T w1 =1.0° / s, the patient's head acceleration threshold is T a2 =0.15g, the patient's head angular velocity threshold is T w2 =0.8° / s), which can effectively determine the real-time jitter of the microscope and the patient's head.

[0167] Acquire real-time video stream data through a microscope (preferably with a frame rate of 30 to 60 fps and a resolution of 1080p) and pre-process the video stream;

[0168] Wherein, the preprocessing includes:

[0169] The acquired real-time video stream is converted into a gray image, and the image of the t-th frame is subjected to gray processing to reduce the calculation complexity, and the specific calculation formula is as follows:

[0170] I t (x,y)=0.299R+0.587G+0.114B

[0171] wherein the I t (x,y) is the gray image of the t-th frame, and the R, G and B are respectively the red, green and blue channels of the pixel;

[0172] The gray image is subjected to Gaussian filtering through convolution operation, so as to reduce the interference of noise on the calculation of the optical flow, and the specific calculation formula is as follows:

[0173] I t '(x,y)=I t (x,y)*G(x,y,σ)

[0174] wherein the I t '(x,y) is the t-th frame gray image subjected to Gaussian filtering, and the G(x,y,σ) is a two-dimensional Gaussian function expression.

[0175] In the embodiment, the data preprocessing is performed on the real-time video stream data, so as to effectively reduce the calculation complexity and the interference of noise on the subsequent optical flow calculation.

[0176] The optical flow vector field of all pixels in each frame of video is calculated based on the real-time video stream data, that is, for two continuous frames of video stream data, the optical flow vector field of all pixels in each frame of video is calculated through an optical flow algorithm (preferably a dense optical flow algorithm), and the specific calculation formula is as follows:

[0177]

[0178] wherein the [I t (x,y)] x ', [I t (x,y)] y ', [I t (x,y)] t ' are respectively the partial derivatives of the gray image of the t-th frame in the x direction, the y direction and the t direction, and the [I t+1 (x,y)] x ', [I t+1 (x,y)] y ', [I t+1 (x,y)] trespectively, are partial derivatives of the gray image of the t+1th frame in x direction, y direction and t direction, and (u, v) is an optical flow vector of a pixel in x direction and y direction;

[0179] The optical flow vector field of all pixels in each frame of video is calculated layer by layer from low resolution to high resolution, and a displacement vector is calculated by using a polynomial expansion to approximate a local image area, and finally an optical flow field V(x, y) = [u(x, y), v(x, y)] is output.

[0180] The contour extraction is performed based on the optical flow vector field of all pixels in each frame of video, and a global motion amplitude and a contour motion amplitude are calculated, wherein the global motion amplitude is calculated according to the following formula:

[0181]

[0182] Wherein, the M global is the global motion amplitude, and the N k is the Kth pixel in the whole image;

[0183] The contour motion amplitude is calculated according to the following formula:

[0184]

[0185] Wherein, the M local,k is the motion amplitude in the contour, and the C k is the Kth pixel in the contour image;

[0186] The video stream is determined to be jittered based on the global motion amplitude and the motion amplitude in the contour, and if M global <M local,k It is indicated that the average jitter of the contour in the lens is greater than the global jitter, and the video stream is jittered.

[0187] In the embodiment, the optical flow vector field of all pixels in each frame of video is calculated by acquiring real-time video stream data by a microscope; the contour extraction is performed based on the optical flow vector field of all pixels in each frame of video, and the global motion amplitude and the contour motion amplitude are calculated; finally, the video stream is determined to be jittered based on the global motion amplitude and the motion amplitude in the contour. Through the above technical solution, whether the video captured by the microscope in the actual fundus retinal injection surgery is locally jittered can be effectively identified, and the stability of the surgery video is effectively improved.

[0188] By combining the acceleration threshold comparison result and the angular velocity threshold comparison result in the above-mentioned scheme, the jitter that can occur in the actual fundus retinal injection surgery is analyzed, and whether the jitter occurs is determined; the specific analysis is as follows:

[0189] When the acceleration threshold comparison result is that the second acceleration amplitude is greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result is that the second angular velocity amplitude is greater than or equal to the preset angular velocity threshold, and the video stream is jittered, it is confirmed that the patient's head is jittered;

[0190] When the acceleration threshold comparison result is that the first acceleration and the second acceleration are both not greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result is that the first angular velocity and the second angular velocity are both not greater than or equal to the preset angular velocity threshold, and the video stream is jittered, it is confirmed that the video light stream is jittered;

[0191] When the acceleration threshold comparison result is that the first acceleration and the second acceleration are both greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result is that the first angular velocity and the second angular velocity are both greater than or equal to the preset angular velocity threshold, and the video stream is jittered, it is confirmed that the global jitter is confirmed.

[0192] In the embodiment, by combining the jitter condition of the video stream to distinguish the jitter source in the actual fundus retina injection surgery, an effective separation mechanism is provided, the high robustness (not affected by optical factors) of the IMU and the high spatial resolution of the light stream are fully utilized, the complementary advantages are realized, the effect of realizing multi-source jitter distinction is effectively improved, and the limitation of a single mode is broken through and overcome.

[0193] According to the video jitter determination method, when it is determined that jitter occurs, the embodiment will perform jitter processing according to different jitter results. The jitter results include microscope jitter, patient head jitter, video light stream jitter, and global jitter. The specific processing method is as follows:

[0194] The first motion trajectory of the microscope is calculated based on the first target acceleration and the first target angular velocity. Taking the first target acceleration and the first target angular velocity in the X-axis direction as an example, the specific calculation formula is as follows:

[0195]

[0196] Wherein, the S x1 (t) is the displacement of the microscope along the X-axis direction, and the θ x1 (t) is the rotation angle of the microscope around the X-axis.

[0197] The second motion trajectory of the microscope is calculated based on the second target acceleration and the second target angular velocity. Taking the second target acceleration and the second target angular velocity in the X-axis direction as an example, the specific calculation formula is as follows:

[0198]

[0199] Wherein, the S x2(t) is the displacement of the patient's head along the X-axis direction, and the θ x2 (t) is the rotation angle of the patient's head around the X-axis.

[0200] In addition, in order to reduce the integral drift error, the acceleration and angular velocity data can also be fused by Kalman filtering to correct the speed and displacement estimation.

[0201] The first motion offset and the first rotation angle of the microscope are calculated based on the first motion trajectory, and the specific calculation formula is as follows:

[0202] ΔS x ≈K a ·S x1 (t), ΔS y ≈K a ·S y1 (t), Δθ≈K w ·θ z1 (t)

[0203] Wherein, the ΔS x is the first motion offset of the microscope along the X-axis direction, the K a is the proportional relationship between the measurement speed of the microscope IMU inertial measurement unit and the pixel space speed in the video frame, the ΔS y is the motion offset of the microscope along the Y-axis direction, the Δθ is the first rotation angle of the microscope, the θ z1 (t) is the rotation angle of the microscope around the Z-axis, and the K w is the proportional coefficient between the first target angular velocity and the rotation angle in the pixel space of the video frame.

[0204] The second motion offset of the patient's head is calculated based on the second motion trajectory, and the specific calculation formula is as follows:

[0205] ΔS head,x ≈K a ·S x2 (t), ΔS head,y ≈K h ·S y2 (t)

[0206] Wherein, the ΔS head,x is the second motion offset of the patient's head along the X-axis direction, the ΔS head,y is the second motion offset of the patient's head along the Y-axis direction, and the K h is the proportional relationship between the measurement speed of the patient's head IMU inertial measurement unit and the pixel space speed in the video frame.

[0207] The contour is extracted based on the optical flow vector field of all pixels in each frame of video, and the average value of the optical flow field in the contour is calculated; wherein the specific calculation formula is as follows:

[0208]

[0209] Wherein, the V local,k is the average value of the optical flow field in the contour, and the C k is the Kth pixel in the picture in the contour.

[0210] When the microscope is detected to be shaken, the video stream is compensated for translation and rotation; wherein the specific calculation formula of the translation compensation is as follows:

[0211] I t ”(x,y)=I t '(x-ΔS x ,y-ΔS y )

[0212] Wherein, the I t ”(x,y) is the tth frame of gray-scale image after translation transformation according to the first motion offset of the microscope;

[0213] The specific calculation formula of the rotation compensation is as follows:

[0214]

[0215] Wherein, the is the coordinate after rotation compensation, and the is the original coordinate.

[0216] Suppose the center point of the surgical field of view is (x c ,y c ), when the shaking of the patient's head is detected, the video stream is adjusted and aligned; wherein the specific calculation formula of the adjustment and alignment is as follows:

[0217] (x c ',y c ')=(x c -ΔS head,x ,y c -ΔS head,y )

[0218] Wherein the (x c ',y c ') is the center point of the surgical field of view after adjustment and alignment.

[0219] When the video light flow is detected to be jittered, a correction region is generated based on the average value of the light flow field within the contour, preferably using a bilinear interpolation method, and the video stream is corrected based on the correction region; wherein the specific calculation formula is as follows:

[0220] I t ”(x,y)=I t '(x-u avg ,y-v avg )

[0221] Wherein, the I t ”(x,y) is the tth frame gray image after correction processing according to the average value of the light flow field within the contour of the video stream.

[0222] In order to avoid the mutation of the correction region and the uncorrected region, the embodiment also uses linear interpolation feathering to perform boundary smoothing processing on the tth frame gray image after correction processing, and the specific calculation formula is as follows:

[0223] I t ”'(x,y)=a·I t ”(x,y)+(1-a)I t '(x,y)

[0224] Wherein, the I t ”'(x,y) is the tth frame gray image after boundary smoothing processing, the a is the feathering weight, the I t ”(x,y) is the tth frame gray image after correction processing according to the average value of the light flow field within the contour of the video stream, and the I t '(x,y) is the tth frame gray image after Gaussian filtering.

[0225] In the embodiment, by taking targeted de-jitter measures for multiple jitter sources, the superimposed motion of multiple jitters that may exist simultaneously in the actual fundus retinal injection surgery is effectively solved. By processing different jitter sources respectively, such as globally compensating the jitter data of the microscope and the patient's head, and locally correcting the video stream using light flow data. This mechanism of effectively separating jitter sources and taking targeted compensation for different jitter sources effectively overcomes the difference in jitter characteristics and ensures efficient and stable de-jitter effect.

[0226] The above is the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, these improvements and refinements are also considered to be within the scope of protection of the present application.

Claims

1. A method for determining video jitter, characterized in that: Applied to an application scenario of fundus retinal injection surgery, the application scenario includes an IMU inertial measurement unit and a microscope; the video jitter determination method includes: Acquire a first initial acceleration and a first initial angular velocity of the microscope and a second initial acceleration and a second initial angular velocity of the patient's head in real time through an IMU inertial measurement unit; extracting a first shake frequency of the microscope and a second shake frequency of the patient's head based on the first initial acceleration, the first initial angular velocity, the second initial acceleration, and the second initial angular velocity; A source of the vibration is determined based on a first vibration frequency of the microscope and a second vibration frequency of the patient's head.

2. The video jitter determination method according to claim 1, wherein: Also includes: Acquire real-time video streaming data through the microscope; Calculate the optical flow vector field of all pixels in each frame of video based on real-time video stream data; Contour extraction is performed based on the optical flow vector field of all pixels in each frame of video, and the global motion amplitude and contour motion amplitude are calculated; Whether jitter occurs in the video stream is determined based on the global motion amplitude and the motion amplitude within the contour.

3. The video jitter determination method according to claim 2, wherein: The extracting a first shake frequency of the microscope and a second shake frequency of the patient's head based on the first initial acceleration, the first initial angular velocity, the second initial acceleration, and the second initial angular velocity includes: performing data preprocessing based on the first initial acceleration, the first initial angular velocity, the second initial acceleration, and the second initial angular velocity to obtain a first target acceleration, a first target angular velocity, a second target acceleration, and a second target angular velocity; Obtaining a first acceleration spectrum, a first angular velocity spectrum, a second acceleration spectrum, and a second angular velocity spectrum by performing fast Fourier transform on the first target acceleration, the first target angular velocity, the second target acceleration, and the second target angular velocity, respectively; Obtaining a first jitter frequency of the microscope based on a first acceleration spectrum and a first angular velocity spectrum; A second vibration frequency of the patient's head is obtained based on the second acceleration spectrum and the second angular velocity spectrum.

4. The video jitter determination method according to claim 3, wherein: The determining of a source of the jitter based on a first jitter frequency of the microscope and a second jitter frequency of the patient's head includes: When the jitter frequency falls within a preset microscope jitter frequency range, it is determined that the jitter originates from the microscope; When the shaking frequency falls within a preset patient head shaking frequency range, it is determined that the shaking originates from the patient's head.

5. The video jitter determination method according to claim 4, characterized in that: Also includes: Calculating a first acceleration amplitude and a first angular velocity amplitude based on the first target acceleration and the first target angular velocity; Calculating a second acceleration amplitude and a second angular velocity amplitude based on the second target acceleration and the second target angular velocity; Comparing the first acceleration amplitude and the second acceleration amplitude with a preset acceleration threshold to obtain an acceleration threshold comparison result; Obtaining an angular velocity threshold comparison result based on comparing the first angular velocity amplitude and the second angular velocity amplitude with a preset angular velocity threshold; When the acceleration threshold comparison result shows that the first acceleration amplitude is greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result shows that the first angular velocity amplitude is greater than or equal to the preset angular velocity threshold, and the video stream shakes, it is determined that the microscope is shaking; When the acceleration threshold comparison result shows that the second acceleration amplitude is greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result shows that the second angular velocity amplitude is greater than or equal to the preset angular velocity threshold, and the video stream shakes, it is determined that the patient's head is shaking; When the acceleration threshold comparison result is that both the first acceleration and the second acceleration are not greater than or equal to the preset acceleration threshold, and the angular velocity threshold comparison result is that both the first angular velocity and the second angular velocity are not greater than or equal to the preset angular velocity threshold, and the video stream jitters, it is determined that the video optical flow jitters; When the acceleration threshold comparison result is that both the first acceleration and the second acceleration are greater than or equal to the preset acceleration threshold, the angular velocity threshold comparison result is that both the first angular velocity and the second angular velocity are greater than or equal to the preset angular velocity threshold, and the video stream jitters, global jitter is confirmed.

6. A video jitter processing method, characterized in that: Performing jitter processing on the jitter results determined by the video jitter determination method according to claim 5, wherein the jitter results include microscope jitter, patient head jitter, video optical flow jitter, and global jitter; The video jitter processing method includes: When it is detected that the microscope is shaking, performing translation compensation and rotation compensation on the video stream; When shaking of the patient's head is detected, adjusting and aligning the video stream; When jitter is detected in the video optical flow, correction processing is performed on the video flow.

7. The video jitter processing method according to claim 6, characterized in that: When the microscope is detected to be shaking, performing translation compensation and rotation compensation on the video stream includes: calculating a first motion trajectory of the microscope based on the first target acceleration and the first target angular velocity; calculating a first motion offset and a first rotation angle of the microscope based on the first motion trajectory; performing translation compensation on the video stream based on a first motion offset; The video stream is rotationally compensated based on a first rotation angle.

8. The video jitter processing method according to claim 6, characterized in that: When the patient's head is detected to be shaking, the video stream is adjusted and aligned, including: calculating a second motion trajectory of the patient's head based on the second target acceleration and the second target angular velocity; calculating a second motion offset of the patient's head based on the second motion trajectory; The video streams are adjusted and aligned based on the second motion offset.

9. The video jitter processing method according to claim 6, characterized in that: When jitter is detected in the video optical flow, correction processing is performed on the video flow, including: Contour extraction is performed based on the optical flow vector field of all pixels in each frame of video, and the average value of the optical flow field within the contour is calculated; A correction region is generated based on an average value of the optical flow field within the contour, and the video stream is corrected based on the correction region.

10. The video jitter processing method according to claim 7, characterized in that: The calculating the first motion trajectory of the microscope based on the first target acceleration and the first target angular velocity includes: Performing a second integration based on the first target acceleration to obtain the real-time displacement of the microscope on different direction axes; Performing an integration based on the first target angular velocity to obtain the real-time rotation angle of the microscope around axes in different directions; A first motion trajectory of the microscope is obtained based on the real-time displacement of the microscope in different directions and the real-time rotation angles of the microscope around axes in different directions.

11. The video jitter processing method according to claim 8, characterized in that: The calculating a second motion trajectory of the patient's head based on the second target acceleration and the second target angular velocity includes: Performing a secondary integration based on the second target acceleration to obtain real-time displacement of the patient's head in different direction axes; Performing an integration based on the second target angular velocity to obtain real-time rotation angles of the patient's head around axes in different directions; A second motion trajectory of the patient's head is obtained based on the real-time displacement of the patient's head in different directions and the real-time rotation angles of the patient's head around axes in different directions.

12. The video jitter processing method according to claim 6, characterized in that: Also includes: When the acceleration threshold comparison result is that the first acceleration amplitude and the second acceleration amplitude are both greater than or equal to the preset acceleration threshold, and the angular velocity threshold comparison result is that the first angular velocity amplitude and the second angular velocity amplitude are both greater than or equal to the preset angular velocity threshold, but the video stream does not jitter, a fault is confirmed and the IMU inertial measurement unit and microscope need to be inspected.

Images