Automatic focusing method and system for intelligent exterior mirror

Abstract

The invention discloses an automatic focusing method and system for an intelligent exterior mirror. The method comprises the following steps: step 1, starting an automatic focusing function; 2, determining a focusing target; step 3, moving for a section of stroke R; step 4, acquiring an image sequence S0 in the motion stroke; 5, calculating a correlation curve C0 for the S0, analyzing the C0 to obtain an abnormal subsequence S1, and processing the S0 and the S1 to obtain a sequence S2; step 6, filtering and denoising the S2 to obtain a sequence S3; step 7, calculating a definition curve C1 for S3; 8, the C1 is optimized to solve the maximum value, and the Pmax position corresponding to the maximum value is obtained; and step 9, moving to a Pmax position, and completing automatic focusing. According to the method, the abnormal subsequences of the image sequence are analyzed and processed, the filtering method is combined, and the input image is used for definition calculation, so that the stability of automatic focusing of the intelligent exterior mirror can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical devices, and relates to an automatic focusing method and system for an intelligent external endoscope. Background Art

[0002] In recent years, as an important auxiliary instrument for observing the surgical site, the external endoscope has been increasingly used in surgical operations. During the use of the external endoscope, first, the doctor drags the working arm, and the target tissue appears in the field of view. Then, the focusing ring of the lens group is manually adjusted to change the working distance of the lens group until the target image is clear. First, the traditional manual focusing of the external endoscope is inconvenient to operate. Moreover, during the automatic focusing movement stroke of the lens group motor of the electronic external endoscope, the acceleration and deceleration jitter inevitably occurs, resulting in an abnormal correlation curve of the local image sequence (such as Figure 1 ), which seriously affects the subsequent clarity calculation and finally automatically focuses on the wrong position. How to analyze and process the abnormal sequence caused by the jitter of the lens group motor, and how to calculate the clarity quickly and stably are the key technologies that the intelligent external endoscope urgently needs to overcome. Summary of the Invention

[0003] In view of this, in order to solve the above problems of the intelligent external endoscope, the present invention provides a method and system for automatic focusing of an intelligent external endoscope.

[0004] The specific solutions are as follows:

[0005] An automatic focusing method for an intelligent external endoscope includes the following steps:

[0006] Step 1. Start the automatic focusing function;

[0007] Step 2. The ROI module determines the focusing target;

[0008] Step 3. The motion control module moves a stroke R;

[0009] Step 4. The image acquisition module acquires the image sequence S0 within the motion stroke R;

[0010] Step 5. The abnormal subsequence processing module calculates the correlation curve C0 for the image sequence S0, analyzes the correlation curve C0 to obtain the abnormal subsequence S1, and processes the image sequence S0 and the abnormal subsequence S1 to obtain the sequence S2;

[0011] Step 6. The filtering module filters and denoises the sequence S2 to obtain the sequence S3;

[0012] Step 7. The clarity evaluation module calculates the clarity curve C1 for the sequence S3;

[0013] Step 8. The optimization module optimizes and solves the maximum value of the clarity curve C1 to obtain the Pmax position corresponding to the maximum value;

[0014] Step 9. The motion control module moves to the Pmax position to complete autofocus.

[0015] Further, step 2 is specifically: by the user clicking on the target within the field of view of the external mirror on the screen, which is displayed in the form of a cross plus a rectangular frame, and confirmed as the focus target area.

[0016] Further, step 3 is specifically: the lens group moves a stroke R.

[0017] Further, step 4 is specifically: collect the image sequence S0 within the stroke R for subsequent analysis.

[0018] Further, step 5 is specifically: for the image sequence S0, calculate the correlation between adjacent images to obtain the correlation curve C0, analyze the correlation curve C0 to obtain the abnormal subsequence S1, and process the image sequence S0 and the abnormal subsequence S1 to obtain the sequence S2; among them, use the mutual information of the images as the correlation index; use wavelet transform to perform time-frequency analysis on the correlation curve C0, select Symlet wavelet for abnormal sequence analysis to obtain the abnormal subsequence S1; select the masking method for the abnormal subsequence S1 in the image sequence S0 to obtain the sequence S2.

[0019] Further, step 6 is specifically: use bilateral filtering to filter the sequence S2, remove noise while retaining edge information to obtain the sequence S3.

[0020] Further, step 7 is specifically: for the images in the sequence S3, use the Sobel gradient operator average amplitude statistical method to calculate the sharpness to obtain the sharpness curve C1 of the sequence S3.

[0021] Further, step 8 is specifically: for the sharpness curve C1, use the Levenberg-Marquardt method (abbreviated as LM) for optimization to obtain the Pmax position corresponding to the maximum value of the curve sharpness C1.

[0022] Further, step 9 is specifically: send Pmax to the motor control module, and the lens group moves into place to complete autofocus.

[0023] An intelligent external vision mirror automatic focusing system includes an ROI module, a motion control module, an image acquisition module, an abnormal subsequence processing module, a filtering module, a clarity evaluation module, and an optimization module. The ROI module is used to determine the focusing target and reduce the influence of the background in non-target areas on automatic focusing. The motion control module is responsible for controlling the movement of the lens group motor during the automatic focusing process. The image acquisition module is used to acquire image sequences and input them into the abnormal subsequence processing module. The abnormal subsequence processing module is used for processing abnormal subsequences, including correlation calculation, abnormal recognition, and abnormal processing. The filtering module is used to filter the input image sequences and remove noise points in the images. The clarity evaluation module is used to calculate the clarity curve for the input image sequences. The optimization module is used to optimize and solve the clarity curve.

[0024] The beneficial effects of the present invention are as follows:

[0025] The present invention can analyze and process the abnormal subsequences of image sequences, and then combine with the filtering method for the input images used in clarity calculation, improving the stability of the automatic focusing of intelligent external vision mirrors. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a diagram of an abnormal correlation curve.

[0027] Figure 2 It is a flowchart of the automatic focusing method.

[0028] Figure 3 It is a diagram of the focusing target area.

[0029] Figure 4 It is a diagram of the clarity curve. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The following further clarifies the present invention in conjunction with the drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and not to limit the scope of the present invention.

[0031] As Figure 2 shown, the present invention provides an external vision mirror automatic focusing system, including an ROI module, a motion control module, an image acquisition module, an abnormal subsequence processing module, a filtering module, a clarity evaluation module, and an optimization module. The specific focusing method is as follows:

[0032] Step 1. Start the automatic focusing function;

[0033] Step 2. The ROI module determines the focusing target;

[0034] Step 3. The motion control module moves a stroke R;

[0035] Step 4. The image acquisition module acquires the image sequence S0 within the motion stroke R;

[0036] Step 5. The abnormal subsequence processing module calculates the correlation curve C0 for the image sequence S0, analyzes the correlation curve C0 to obtain the abnormal subsequence S1, and processes the image sequence S0 and the abnormal subsequence S1 to obtain the sequence S2;

[0037] Step 6. The filtering module filters and denoises the sequence S2 to obtain the sequence S3;

[0038] Step 7. The sharpness evaluation module calculates the sharpness curve C1 for the sequence S3;

[0039] Step 8. The optimization module optimally solves the maximum value of the sharpness curve C1 to obtain the Pmax position corresponding to the maximum value;

[0040] Step 9. The motion control module moves to the Pmax position to complete autofocus.

[0041] In this embodiment, the ROI module is responsible for providing the function for the user to select the focusing target, reducing the influence of the background of the non-target area on autofocus. Optionally, the user clicks on the target object within the field of view of the external mirror on the screen, and represents the focusing target area in the form of a cross plus a rectangular frame (as Figure 3 shown), and by default, 4 / 5 of the field of view of the external mirror is used as the focusing target area.

[0042] In this embodiment, the motion control module is composed of a linear motor and a lens group. This module is responsible for controlling the motion of the motor during autofocus, coordinating with the functions of image sequence acquisition and motion in place.

[0043] In this embodiment, the image acquisition module is responsible for acquiring the image sequence S0, and S0 is input into the abnormal subsequence processing module.

[0044] In this embodiment, the abnormal subsequence processing module calculates the correlation curve C0 for S0, analyzes C0 to obtain the abnormal subsequence S1, and processes S0 and S1 to obtain the sequence S2. Further, this module includes correlation calculation, abnormal recognition, and abnormal processing. Among them, for correlation, first calculate the correlation of adjacent images of S0 to obtain the correlation curve C0, perform time-frequency analysis on C0, and identify the abnormal subsequence S1;

[0045] In this embodiment, the image correlation calculation methods usually include peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and mutual information (MI). Optionally, the correlation calculation method uses mutual information for calculation. The mutual information formula MI(A,B) of AB images is as follows:

[0046] MI(A,B) = H(A) + H(B) - H(A,B)

[0047]

[0048] where \(H(A)\) is the information entropy of image \(A\), \(H(B)\) is the information entropy of image \(B\), \(H(A,B)\) is the joint entropy of images \(A\) and \(B\), \(N\) is the number of gray levels, usually 256, and \(p\) i is the probability of a pixel with gray value \(i\) in the image, and \(p\) AB (a,b) is the probability that the gray value of the same pixel in image \(A\) is \(a\) and the gray value in image \(B\) is \(b\).

[0049] In this embodiment, anomaly analysis, that is, time-frequency analysis of the curve waveform, usually includes Fourier transform and wavelet transform. Fourier transform can analyze the frequency components but will lose the spatio-temporal domain information and cannot give the position of the subsequence; wavelet transform can not only analyze the frequency components but also retain the spatio-temporal domain information. Optionally, wavelet transform is used to perform time-frequency analysis on \(C0\) to obtain the anomaly frequency and the corresponding spatio-temporal information, and further obtain the anomaly subsequence \(S1\). Optionally, since the Symlet wavelet has good time and frequency resolution, the Symlet wavelet is selected in this embodiment to analyze the curve \(C0\) to obtain the anomaly subsequence \(S1\).

[0050] In this embodiment, for anomaly handling, the anomaly subsequence \(S1\) of \(S0\) is processed to obtain \(S2\). The anomaly handling method, optionally, by the masking method, the \(S1\) subsequence is masked in the \(S0\) sequence to obtain \(S2\), and the \(S2\) sequence is input to the filtering module.

[0051] In this embodiment, the filtering module is responsible for filtering the image to remove the noise points in the image. Common filtering operations include Gaussian, mean, median, bilateral filtering, etc. Optionally, bilateral filtering is used to suppress the noise points while maintaining the edges of the image of the \(S2\) sequence to obtain the \(S3\) sequence, and the \(S3\) sequence is input to the sharpness evaluation module. The principle of bilateral filtering is as follows:

[0052]

[0053] w(i,j,k,l) = d(i,j,k,l) * r(i,j,k,l)

[0054]

[0055] where \(i,j\) are the coordinates of the original image, \(k,l\) are the coordinates of the filtering window, \(g(i,j)\) is the gray value of the filtered output at the pixel position \((i,j)\), and \(f(k,l)\) is the gray value of the original image covered by the filtering window. \(w(i,j,k,l)\) is the weight of the filtering window, \(d(i,j,k,l)\) is the spatial domain kernel, and \(r(i,j,k,l)\) is the range domain kernel, where the window diameter is set to 9, and \(\sigma\)d = 75, σ r = 75.

[0056] In this embodiment, the clarity evaluation module is responsible for calculating the clarity of the S3 sequence images to obtain the C1 curve (as shown in Figure 4 ), and the C1 curve is input into the optimization module. The clarity calculation methods usually include the average amplitude statistical method based on the Sobel gradient operator, the method based on Fourier frequency domain analysis, and the method based on the difference analysis of sequential frame images. Optionally, the average amplitude statistical method of the Sobel gradient operator has the advantages of simple calculation and high real-time performance while meeting the index stability. The Sobel method is selected in this embodiment, and the principle of the Sobel operator is as follows:

[0057]

[0058] where I(x, y) is the pixel value of the original image at (x, y), G x (x, y) is the gradient value in the x direction, G y (x, y) is the gradient value in the y direction, and G(x, y) is the corresponding gradient amplitude.

[0059] In this embodiment, the optimization solving module optimizes and solves the maximum value of C1 to obtain the Pmax position corresponding to the maximum value. The optimization methods usually include the gradient descent method (GD), the Newton method, the Gauss-Newton method, the Levenberg-Marquardt method (LM for short), etc. Optionally, the C1 curve is optimized by the LM algorithm to obtain Pmax.

[0060] Finally, Pmax is input into the motor motion control module, and the motor moves to the Pmax position to complete the focusing.

[0061] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above embodiments, but also include the technical solutions composed of any combination of the above technical features. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. An automatic focusing method for an intelligent external vision mirror, characterized in that, Including the following steps: Step 1. Activate the autofocus function; Step 2. ROI module to determine the focusing target; Step 3. Motion control module to move a stroke R; Step 4. Image acquisition module to acquire the image sequence S0 within the motion stroke R; Step 5. Abnormal subsequence processing module to calculate the correlation curve C0 for the image sequence S0, analyze the correlation curve C0 to obtain the abnormal subsequence S1, and process the image sequence S0 and the abnormal subsequence S1 to obtain the sequence S2; Step 6. Filtering module to filter and denoise the sequence S2 to obtain the sequence S3; Step 7. Sharpness evaluation module to calculate the sharpness curve C1 for the sequence S3; Step 8. Optimization module to optimize and solve for the maximum value of the sharpness curve C1 to obtain the Pmax position corresponding to the maximum value; Step 9. Motion control module to move to the Pmax position to complete autofocus.

2. The automatic focusing method of an intelligent external vision mirror according to claim 1, characterized in that Step 2 specifically is: By the user clicking on the target within the field of view of the external mirror on the screen, which is displayed in the form of a cross plus a rectangular frame, and confirming it as the focused target area.

3. The automatic focusing method of an intelligent external vision mirror according to claim 2, characterized in that, Step 3 specifically is: The lens group moves a stroke R.

4. The automatic focusing method of an intelligent external vision mirror according to claim 3, characterized in that, Step 4 specifically is: Acquire the image sequence S0 within the stroke R for subsequent analysis.

5. The automatic focusing method of an intelligent external vision mirror according to claim 4, characterized in that, Step 5 specifically is: For the image sequence S0, calculate the correlation of adjacent images to obtain the correlation curve C0, analyze the correlation curve C0 to obtain the abnormal subsequence S1, and process the image sequence S0 and the abnormal subsequence S1 to obtain the sequence S2; Among them, use the mutual information of the images as the correlation index; use wavelet transform for time-frequency analysis of the correlation curve C0, select the Symlet wavelet for abnormal sequence analysis to obtain the abnormal subsequence S1; select the masking method for the abnormal subsequence S1 in the image sequence S0 to obtain the sequence S2.

6. The automatic focusing method of an intelligent external vision mirror according to claim 5, characterized in that, Step 6 specifically is: Use bilateral filtering to filter the sequence S2, removing noise while retaining edge information to obtain the sequence S3.

7. The automatic focusing method of an intelligent external vision mirror according to claim 6, wherein, Step 7 specifically is: For the images in the sequence S3, use the average amplitude statistics method of the Sobel gradient operator to calculate the sharpness to obtain the sharpness curve C1 of the sequence S3.

8. The automatic focusing method of an intelligent external vision mirror according to claim 7, wherein, Step 8 specifically is: Use the Levenberg-Marquardt method to optimize and solve for the sharpness curve C1 to obtain the Pmax position corresponding to the maximum value of the curve sharpness C1.

9. The automatic focusing method of an intelligent external vision mirror according to claim 8, characterized in that Step 9 specifically is: Send Pmax to the motor control module, and the lens group moves into place to complete autofocus.

10. An intelligent external vision mirror automatic focusing system, characterized in that, It includes an ROI module, a motion control module, an image acquisition module, an abnormal subsequence processing module, a filtering module, a sharpness evaluation module, and an optimization module. Among them, the ROI module is used to determine the focusing target and reduce the influence of the background in the non-target area on autofocus. The motion control module is responsible for controlling the movement of the lens group motor during the autofocus process. The image acquisition module is used to acquire image sequences and input them into the abnormal subsequence processing module. The abnormal subsequence processing module is used for abnormal subsequence processing, including correlation calculation, abnormality recognition, and abnormality handling. The filtering module is used to filter the input image sequence to remove noise points in the image. The sharpness evaluation module is used to calculate the sharpness curve for the input image sequence. The optimization module is used to optimize and solve the sharpness curve.

Images