Filter design method, filter IP, and horizontal synchronization time interval prediction method

Abstract

The present application relates to the field of signal processing, and discloses a filter design method, a filter IP, and a horizontal synchronization time interval prediction method. The filter design method comprises: performing statistical characteristic analysis on the basis of an abnormal horizontal synchronization signal time interval counting sequence, to obtain abnormal frequency spectrum characteristic distribution data; and determining a filtering coefficient of an FIR digital filter on the basis of the abnormal frequency spectrum characteristic distribution data.

Description

Filter design method, filter IP and line synchronization interval time prediction method

[0001] The present application claims priority to the Chinese patent application No. 202410467155.X filed on April 18, 2024 in the Chinese Patent Office and titled "Filter design method, filter IP and line synchronization interval time prediction method", the entire contents of which are incorporated herein by reference. The disclosure of all related applications that are cited herein and that are in the present application are hereby incorporated by reference in their entirety as part of the present application. TECHNICAL FIELD

[0002] The present application relates to the field of signal processing, more specifically, the present application relates to a filter design method, a filter IP and a line synchronization interval time prediction method. BACKGROUND

[0003] In the field of video images, the line synchronization signal is generally used as a trigger signal for the start of video image line sampling. When the sampling module of the system detects the rising edge of the line synchronization signal, it starts sampling a line of image. Therefore, whether the data of each line in the entire image can be aligned in the vertical direction is directly related to the quality of the synchronization signal. In actual systems, the synchronization signal is generally generated by a dedicated device. Due to factors such as batch differences of the device, the line synchronization signals generated by different batches of devices will usually have differences. In poor cases, low-frequency noise will be mixed into the synchronization signal, making the interval time between two adjacent synchronization signals not constant, showing changes in cycle length, which in turn affects the starting sampling time of each line of data, so that the data of each line in the entire image cannot be aligned, showing a wave-shaped image phenomenon. SUMMARY

[0004] A series of simplified concepts are introduced in the summary section, which will be further described in detail in the detailed description section. The summary section of the present application does not mean to attempt to limit the key features and necessary technical features of the claimed technical solutions, nor to determine the protection scope of the claimed technical solutions.

[0005] In order to eliminate the low-frequency jitter of the line synchronization signal, ensure the stability of the interval time of the line synchronization signal, and ensure the alignment of the video line data in the vertical direction, and improve the quality of the video image controlled by the line synchronization signal sampling, in a first aspect, the present application proposes a filter design method for abnormal line synchronization signals, comprising:

[0006] Statistical characteristic analysis is performed on the abnormal line synchronization signal interval time count sequence to obtain abnormal frequency spectrum characteristic distribution data;

[0007] The filter coefficients of the FIR digital filter are determined in combination with the above abnormal frequency spectrum characteristic distribution data.

[0008] Optionally, the statistical property analysis on the abnormal line synchronization signal interval time count sequence is used to obtain abnormal spectral characteristic distribution data, including:

[0009] The abnormal line synchronization signal interval time count sequence is subjected to spectral analysis to obtain spectral distribution data.

[0010] The image quality corresponding to the abnormal line synchronization signal interval time count sequence is determined.

[0011] The abnormal spectral characteristic distribution data is determined according to the spectral distribution data and the image quality.

[0012] Optionally, the spectral distribution data includes a maximum amplitude in a frequency domain and a frequency corresponding to the maximum amplitude.

[0013] Optionally, the image quality is determined according to an image resolution and an image resolution threshold.

[0014] Optionally, the filter coefficient of the FIR digital filter is determined in combination with the abnormal spectral characteristic distribution data, including:

[0015] The stopband frequency and the passband frequency are determined according to the abnormal spectral characteristic distribution data.

[0016] The Chebyshev filter is designed by using the equal-ripple best approximation method according to the stopband frequency, the passband frequency, and an empirical parameter, to obtain the filter coefficient.

[0017] In a second aspect, the application further provides a filter IP including a target filter coefficient, wherein the target filter coefficient is obtained by the filter design method for abnormal line synchronization signals according to any one of the first aspect.

[0018] In a third aspect, the application further provides a method for predicting line synchronization interval time based on a neural network, which is used for the filter IP in the second aspect, including:

[0019] The synchronization signal interval time is counted based on an internal clock of an FPGA chip to obtain a synchronization signal interval time count sequence.

[0020] The synchronization signal interval time count sequence is subjected to filter processing by the filter IP to obtain a to-be-processed synchronization signal interval count sequence.

[0021] The line synchronization signal interval time is predicted by using the to-be-processed synchronization signal interval count sequence and a self-attention-based long short-term memory network model to obtain a predicted line synchronization interval time.

[0022] Optionally, the above-mentioned line synchronization signal interval time prediction using the above-mentioned to-be-processed synchronization signal interval count sequence and the self-attention-based long short-term memory network model comprises:

[0023] The information at different times in the above-mentioned to-be-processed synchronization signal interval count sequence is dynamically assigned a weight according to its importance, and the prediction is combined with the weight.

[0024] Optionally, the above-mentioned counting of the synchronization signal time interval based on the FPGA chip internal clock comprises:

[0025] In the case of detecting the rising edge of the line synchronization signal, the counting of the FPGA chip internal clock is cleared and accumulated.

[0026] In the case of monitoring the next synchronization signal rising edge, the accumulation operation is stopped and the accumulated value is obtained.

[0027] The above-mentioned accumulation operation process is repeated, and a plurality of the above-mentioned accumulated values are used as the above-mentioned synchronization signal interval time count sequence.

[0028] Optionally, the above-mentioned self-attention-based long short-term memory network model is composed of a plurality of self-attention-based long short-term memory layers, each of which is composed of a plurality of long short-term memory units and a self-attention mechanism. Each long short-term memory unit is composed of a forget gate, an input gate, an output gate, and a unit state. The self-attention mechanism is connected to each of the long short-term memory units. The self-attention mechanism is used to calculate attention weights, and the attention weights are used to weight the sum of the outputs of the long short-term memory units to obtain the output.

[0029] In summary, the filter design method for abnormal line synchronization signal provided by the embodiment of the present application comprises: performing statistical characteristic analysis according to the abnormal line synchronization signal interval time count sequence to obtain abnormal frequency spectrum characteristic distribution data; and determining the filter coefficients of the FIR digital filter in combination with the above abnormal frequency spectrum characteristic distribution data. The filter design method for abnormal line synchronization signal provided by the embodiment of the present application can effectively remove or weaken the abnormal frequency spectrum components in the video synchronization signal, and can significantly improve the performance and reliability of the video image processing system. By accurately identifying and processing the abnormal part (for example, abnormal signal characteristics caused by noise, device difference or other interference sources) in the signal, the method can ensure that the normal signal part is not affected, thereby ensuring the high pertinence and efficiency of the processing process. The filter coefficients of the FIR digital filter obtained by the present application can be directly set in the FPGA chip, and the high-speed processing capability of the FPGA can be fully utilized to realize real-time filter processing of the synchronization signal. By performing filter processing through the obtained filter, the image quality can be effectively improved, and in particular, the problem of abnormal device causing line synchronization signal can be solved, thereby avoiding the problem of wave-shaped or misaligned line data in the finally displayed image.

[0030] The filter design method for abnormal line synchronization signal, filter IP and line synchronization interval time prediction method provided by the present application, other advantages, objects and features of the present application will be embodied in part through the following description, and will be understood by those skilled in the art through research and practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0031] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of preferred embodiments, and are not meant to limit the present specification. Moreover, the same reference numerals are used throughout the various drawings to designate identical elements. In the drawings:

[0032] FIG. 1 is a filter design flowchart for abnormal line synchronization signal provided by the embodiment of the present application;

[0033] FIG. 2 is a time domain diagram of abnormal line synchronization signal provided by the embodiment of the present application;

[0034] FIG. 3 is a local time domain diagram of abnormal line synchronization signal provided by the embodiment of the present application;

[0035] FIG. 4 is a frequency domain diagram of abnormal line synchronization signal provided by the embodiment of the present application;

[0036] FIG. 5 is a time domain diagram of filtered line synchronization signal provided by the embodiment of the present application;

[0037] FIG. 6 is a local time domain diagram of a filtered line synchronization signal according to an embodiment of the present application;

[0038] FIG. 7 is a frequency domain diagram of a filtered line synchronization signal according to an embodiment of the present application;

[0039] FIG. 8 is a diagram of frequency spectrum characteristics of a jittered data line synchronization signal according to an embodiment of the present application;

[0040] FIG. 9 is a diagram of a filter parameter sequence according to an embodiment of the present application;

[0041] FIG. 10 is a diagram of a filter frequency spectrum response according to an embodiment of the present application;

[0042] FIG. 11 is a diagram of a filter configuration interface according to an embodiment of the present application;

[0043] FIG. 12 is a flow diagram of a method of predicting line synchronization interval time based on a neural network according to an embodiment of the present application;

[0044] FIG. 13 is a diagram of a structure of a self-attention based long short-term memory network model according to an embodiment of the present application;

[0045] FIG. 14 is a diagram of a structure of an LSTM unit according to an embodiment of the present application;

[0046] FIG. 15 is a diagram of a line synchronization signal interval count sequence generated by a normal device according to an embodiment of the present application;

[0047] FIG. 16 is a diagram of a line synchronization signal interval count sequence generated by an abnormal device according to an embodiment of the present application;

[0048] FIG. 17 is a diagram of a line synchronization signal interval count sequence generated by an abnormal device after processing according to an embodiment of the present application;

[0049] FIG. 18 is a diagram of fiber end surface imaging obtained by a confocal scanning enteroscope system using an abnormal line synchronization signal generating device according to an embodiment of the present application;

[0050] FIG. 19 is a diagram of fiber end surface imaging obtained by a confocal scanning enteroscope system using an abnormal line synchronization signal generating device after processing according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application and above-mentioned drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the use of the terms so construed herein is merely for convenience and brevity and that one of the embodiments described herein could be practiced unless otherwise expressly so described herein. Also, the terms "comprise", "have" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, article, or apparatus that comprises a list of steps or units can not necessarily be limited to those steps or units, but can include other not expressly listed or inherent steps or units. The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all.

[0052] The line synchronization signal in a video image system is an important system signal for triggering the line sampling time, and the signal stability directly determines the image quality of the video image system. Due to the batch difference of the device for generating the line synchronization signal, in batch production, the line synchronization signal with uniform spectral characteristics cannot be obtained, and thus the system image quality is reduced. The conventional method for processing such problems is to perform incoming inspection on the incoming device, and to screen out devices with uniform and stable characteristics, so as to obtain uniform image quality. This method increases the production link, and obviously reduces the production efficiency.

[0053] Referring to FIG. 1, a filter design method for abnormal line synchronization signals provided by an embodiment of the present application can specifically include:

[0054] S110, performing statistical characteristic analysis according to the abnormal line synchronization signal interval time count sequence, to obtain abnormal spectral characteristic distribution data;

[0055] S120, determining the filter coefficient of the FIR digital filter in combination with the above-mentioned abnormal spectral characteristic distribution data.

[0056] Exemplarily, the statistical characteristic analysis is performed on the interval time count sequence of the synchronization signal based on the internal clock of the FPGA, so as to identify the abnormal spectral distribution in the signal, and to find the abnormal signal characteristics caused by noise, device difference or other interference sources. According to the obtained abnormal spectral distribution data, the filter coefficients of the FIR digital filter are determined, so as to design a filter capable of effectively removing or weakening the abnormal spectral components, and to process the abnormal part of the signal in a targeted manner without affecting the normal part. The filter coefficients can include parameters such as cutoff frequency, bandwidth, attenuation rate, etc. Then, the filter coefficients of the designed FIR filter are set in the FPGA chip, so as to realize the target filter, and the filtering processing can be performed in real time on the FPGA chip, fully utilizing the high-speed processing capability of the FPGA. By implementing the customized FIR filter on the FPGA, the system can efficiently perform real-time filtering on the synchronization signal, remove or reduce the abnormal signal components, and thus improve the performance and reliability of the entire video image processing system. Exemplarily, as shown in FIGS. 2 to 7, the spectral analysis result example graphs before and after the abnormal device filtering are given, and it can be seen that the quality after filtering is effectively improved. There is a low-frequency component in the row synchronization signal sequence of the abnormal device, and this low-frequency component will cause the data of each row in the finally displayed image to be unable to align, and the image to present a wavy shape.

[0057] In summary, the filter design method for abnormal row synchronization signal proposed in the embodiments of the present application can effectively remove or weaken the abnormal spectral components in the video synchronization signal, and can significantly improve the performance and reliability of the video image processing system. By accurately identifying and processing the abnormal part of the signal (for example, the abnormal signal characteristics caused by noise, device difference or other interference sources), the method can ensure that the normal signal part is not affected, thereby ensuring the high targeting and efficiency of the processing process. The filter coefficients of the FIR digital filter obtained by the present application can be directly set in the FPGA chip, and the high-speed processing capability of the FPGA can be fully utilized to realize real-time filtering processing on the synchronization signal. By filtering processing with the obtained filter, the image quality can be effectively improved, and in particular, the problem of row synchronization signal caused by abnormal devices can be solved, thereby avoiding the problem of wavy shape or row data misalignment in the finally displayed image.

[0058] In some examples, the statistical characteristic analysis on the abnormal row synchronization signal interval time count sequence to obtain the abnormal spectral characteristic distribution data includes:

[0059] The frequency spectrum analysis is performed on the abnormal row synchronization signal interval time count sequence to obtain the spectral distribution data;

[0060] The image quality corresponding to the abnormal row synchronization signal interval time count sequence is determined;

[0061] The abnormal spectral feature distribution data is determined according to the spectral distribution data and the image quality.

[0062] In some examples, the spectral distribution data includes a frequency domain maximum amplitude and a frequency domain maximum amplitude corresponding frequency.

[0063] For example, the frequency domain maximum amplitude represents the value of the point with the maximum amplitude (intensity) in the spectral distribution. The representation of the signal in the frequency domain is achieved by converting the time domain signal into the frequency domain signal, usually by Fourier transform. The frequency domain maximum amplitude indicates how strong the strongest frequency component in the signal is.

[0064] The frequency domain maximum amplitude corresponding frequency is the frequency value corresponding to the point with the maximum amplitude in the spectrum, which represents which frequency component is the most significant in the signal. The amplitude of this frequency is the largest, which means that the signal has the strongest energy concentration at this frequency.

[0065] In some examples, the image quality is determined according to the image resolution and the image resolution threshold.

[0066] For example, the evaluation of image quality is a complex and multi-dimensional task, which can be based on multiple criteria, including but not limited to image clarity, contrast, brightness, color saturation, and noise level, etc. The determination of image quality according to the image resolution and the image resolution threshold in this embodiment is a relatively simple and direct method. If the resolution of the image reaches or exceeds the set threshold, such image is usually considered to be "qualified" or "high quality". On the contrary, if the resolution of the image is lower than the threshold, it may be considered to be "unqualified" or "low quality".

[0067] Specifically, the image quality is determined according to the image resolution corresponding to the synchronization signal interval time count sequence and the preset resolution threshold information, and the image quality is used to determine whether there is jitter. The image resolution is used as a reference index to evaluate the image quality. The resolution threshold is a standard defined in advance, for example, it can be 5, which is used to determine whether the current image quality meets the expected level. By comparing the actual image resolution with the threshold, it can be determined whether the image meets the quality requirement. If the resolution is lower than the threshold, it may indicate that the image quality is poor. If the image meets the quality requirement, the jitter program is 0, and if the image does not meet the quality requirement, the jitter degree is 1. Table 1 is a possible jitter spectral feature statistical table:

[0068] Table 1

[0069] The statistical results are plotted according to frequency-amplitude, and different shapes are used to mark whether there is jitter, as shown in FIG. 8. It can be seen from the figure that the degree of jitter has obvious relevance with the signal spectrum. The line synchronization signal with jitter has a strong signal at a low frequency point, which may be the cause of the jitter.

[0070] In some examples, the determination of the filter coefficients of the FIR digital filter according to the abnormal spectrum characteristic distribution data described above comprises:

[0071] The stopband frequency and the passband frequency are determined according to the abnormal spectrum characteristic distribution data described above;

[0072] The Chebyshev filter is designed by using the equiripple optimal approximation method according to the stopband frequency, the passband frequency and the empirical parameter, so as to obtain the filter coefficients.

[0073] For example, the stopband frequency and the passband frequency are determined according to the abnormal spectrum distribution data that has been obtained. The stopband frequency refers to the frequency range that should be suppressed by the filter, and usually these frequencies contain noise or unwanted signal components. The passband frequency refers to the frequency range that should be retained by the filter, i.e. important signal components. By analyzing the abnormal spectrum distribution, the frequency intervals that need to be filtered out (stopband) and need to be retained (passband) can be identified. The abnormal spectrum distribution data shown in FIG. 8 can determine that the stopband frequency is 0.15 and the passband frequency is 0.2.

[0074] Suppose that the filter parameter of the finite impulse response (FIR) filter is h(n), then the system function of the filter is:

[0075] Where N is the order of the filter, and z represents a complex number. Here, h(n) represents a discrete sequence composed of FIR filter parameters, and the discrete-time sequence h(n) is transformed into the complex frequency domain by Z transform, which is convenient for spectrum analysis.

[0076] The corresponding Fourier frequency domain response function is:

[0077] Written in the form of amplitude response and phase response separation: H(e jw )=H g (w)e jθ(w)

[0078] Where H(e jw ) is called amplitude characteristic, and θ(w) is called phase characteristic.

[0079] The FIR filter can easily meet the strict linear phase characteristic, i.e. θ(w) is a linear function of w, while ensuring that the amplitude characteristic meets the technical requirements.

[0080] Chebyshev filter is designed using the equiripple optimal approximation method. The function scipy.signal.remez of Python can be used to realize the equiripple optimal approximation method. The example code for designing the filter is as follows: from scipy import signal taps=signal.remez(57,[0,0.15,0.2,0.5],[0,1]) w,h=signal.freqz(taps,[1],worN=512,fs=1)

[0081] wherein the first parameter numtaps=57 of the remez function is the number of taps, i.e. the order of the FIR filter; the second parameter bands=[0,0.15,0.2,0.5] is the frequency band of the filter, under the premise of normalized frequency, 0 is the minimum frequency, 0.5 is the maximum frequency, 0.15 and 0.2 are two special frequency points, and the frequency band has two parts: [0,0.15] and [0.2,0.5], and the effect of each part is determined by the third parameter; the third parameter disired=[0,1] indicates the gain of the frequency band, according to the order, the gain of the frequency band [0,0.15] is 0, which represents the spectral stop band, and the gain of the frequency band [0.2,0.5] is 1, which represents the spectral pass band. That is, the stop band frequency is 0.15, the pass band frequency is 0.2, and the empirical parameter is 57.

[0082] The return value taps represents the FIR filter parameter sequence, i.e. h(n), which is an array of 57 parameters as follows:

[0083] The filter parameter sequence h(n) of the above filter is shown in FIG. 9, and the spectral response of the filter is shown in FIG. 10. Then, according to the set FIR filter coefficients, the FIR digital filter is realized in FPGA. The present application can use the Xilinx ZYNQ7000 series FPGA, and the filter coefficients are set in the FIR Compiler configuration interface in the IDE, and then the IP is generated, as shown in FIG. 11. After the filter IP is generated, the interval time sequence of the synchronization signal obtained in step one is input into the filter IP, and the corresponding filtered digital sequence is obtained.

[0084] In the second aspect, the present application also provides a filter IP, comprising target filter coefficients, wherein the target filter coefficients are obtained by the filter design method for abnormal line synchronization signal according to any one of the first aspect.

[0085] In the third aspect, as shown in FIG. 12, the present application also provides a method for predicting line synchronization interval time based on a neural network, which is used for the filter IP according to the second aspect, comprising:

[0086] S210, count the time interval of the synchronization signal based on the internal clock of the FPGA chip to obtain a synchronization signal interval time count sequence;

[0087] For example, the FPGA chip uses its internal clock to count the time interval of the synchronization signal, thereby generating a count sequence of the synchronization signal interval time. This sequence includes normal and abnormal synchronization signal interval counts. Abnormal synchronization signal intervals can be caused by device abnormalities.

[0088] S220, filter the above synchronization signal interval time count sequence through the above filter IP to obtain a to-be-processed synchronization signal interval count sequence;

[0089] For example, the obtained synchronization signal interval time count sequence is processed through the target filter, thereby removing the interference components in the abnormal synchronization signal interval technical sequence, and the interference signals in the obtained to-be-processed synchronization signal interval count sequence are effectively removed.

[0090] S230, use the above to-be-processed synchronization signal interval count sequence and a self-attention-based long short-term memory network model to predict the line synchronization signal interval time to obtain a predicted line synchronization interval time.

[0091] For example, the obtained synchronization signal interval time count sequence is processed through the target filter, thereby removing the interference components in the abnormal synchronization signal interval technical sequence, and the interference signals in the obtained to-be-processed synchronization signal interval count sequence are effectively removed. The filtered synchronization signal interval time count sequence is sampled and further filtered to obtain historical signal data. Thereby, more valuable signal data for subsequent analysis and prediction is extracted. The historical signal data is analyzed using ST-LSTM (Self-Attention Long Short-Term Memory Network) to predict the line synchronization signal interval time. The LSTM model is good at processing and predicting time series data, and the self-attention mechanism can enhance the model's ability to identify important features in the time series. The goal is to accurately predict the line synchronization interval time at the next moment, especially in the presence of noise and abnormal conditions. According to the prediction result provided by the LSTM model, the FPGA chip performs prediction and reconstruction of the synchronization signal. Finally, the line synchronization signal is adjusted or reconstructed according to the predicted interval time to ensure the correct alignment and stability of the video image data.

[0092] In summary, the method of predicting line synchronization interval time based on neural network proposed in the embodiments of the present application can accurately capture the synchronization signal interval in the historical signal, including irregular intervals caused by device abnormalities, through accurate counting by the internal clock of the FPGA. The synchronization signal interval time counting sequence is filtered by the target filter, effectively removing noise and abnormal signal components, improving the quality of the signal and the overall quality and stability of the video image. The long short-term memory network based on self-attention is used to predict the line synchronization signal interval time, which can more accurately predict and adjust the line synchronization signal. This prediction method can better adapt to and correct the changes in the synchronization signal caused by device batch differences and other factors. Finally, the FPGA chip predicts and reconstructs the synchronization signal according to the predicted line synchronization interval time, which can significantly improve the stability and quality of the video image data. The present application combines high-precision hardware counting, advanced digital filtering technology, and deep learning models based on self-attention to provide a comprehensive and effective solution to the synchronization signal quality problem in the field of video images, improving the overall quality and stability of the image.

[0093] In some examples, the above-mentioned line synchronization signal interval time prediction using the above-mentioned to-be-processed synchronization signal interval counting sequence and the long short-term memory network model based on self-attention includes:

[0094] The information at different times in the above-mentioned to-be-processed synchronization signal interval counting sequence is dynamically assigned a weight depending on its importance, and the prediction is combined with the weight.

[0095] For example, the to-be-processed synchronization signal interval counting sequence is input into an LSTM network, which is responsible for capturing the time dependence of the sequence. At the same time, the self-attention mechanism is applied to the sequence, allowing the model to dynamically assign weights according to the importance of the information. Specifically, for each time point in the sequence, the model calculates the relationship strength between it and other time points in the sequence, and assigns attention weights according to these relationship strengths. In this way, when the model makes predictions, it not only considers the time dependence of the sequence, but also considers the importance of each part of the sequence.

[0096] The method proposed by the embodiments of the present application is particularly suitable for scenarios that require processing sequence data with complex time dependence and dynamically changing importance. In the context of line synchronization signal interval time prediction, this means that the model can more accurately predict future synchronization signal interval times because it not only considers the historical time series of the signal, but also considers the relative importance of information at different time points in the sequence. This is very helpful for optimizing synchronization signal processing in video image processing systems, especially when facing noise, device differences or other interference sources, it can provide more stable and accurate prediction results, thereby improving the overall performance and reliability of the system.

[0097] In some examples, the above-mentioned counting of the synchronization signal interval time based on the internal clock of the FPGA chip to obtain the synchronization signal interval time count sequence includes:

[0098] In the case of detecting the rising edge of the line synchronization signal, the counting of the above-mentioned internal clock of the FPGA chip is cleared and an accumulation calculation operation is performed;

[0099] In the case of monitoring the next synchronization signal rising edge, stop the accumulation calculation operation and obtain the accumulated value;

[0100] Repeat the above-mentioned accumulation calculation operation process, and use multiple accumulated values as the above-mentioned synchronization signal interval time count sequence.

[0101] For example, when the FPGA detects the rising edge of the line synchronization signal, its internal clock counter is cleared. The counting is cleared to ensure that the time interval is accurately measured from the start of each synchronization signal. The detection of the rising edge marks the start of a line of image sampling. After detecting the rising edge of the line synchronization signal, the FPGA starts to accumulate the time under the drive of the internal clock. The purpose of the accumulation calculation operation is to record the time interval between the current line synchronization signal rising edge and the next line synchronization signal rising edge. When the rising edge of the next line synchronization signal is monitored, the accumulation calculation stops, and the accumulated value at this time is recorded. This accumulated value represents the time interval between two consecutive line synchronization signals. The time interval is a key parameter in video image processing, which determines the sampling time of each line of image. The accumulation calculation operation is repeated, and the counting is restarted every time the rising edge of the line synchronization signal appears. Repeating this process can generate a series of accumulated values, each representing the time interval between two consecutive line synchronization signals, and this sequence provides detailed time interval data. The high precision of the internal clock of the FPGA can be used to measure the time interval very accurately. This method allows efficient time interval measurement in real-time environments, which is particularly important for video image processing.

[0102] In some examples, FIG. 15 is a sequence of normal device generation, and FIG. 16 is a sequence of abnormal device generation. There is a low-frequency wave coupling in the abnormal sequence, and the image after the method of the application is shown in FIG. 17, which removes the low-frequency wave. Using the method described in the application, the tolerance of the video image system to the signal quality difference caused by batch difference of the line synchronization signal generator can be increased, and the image quality can be finally guaranteed. As shown in FIG. 18 and FIG. 19, FIG. 18 is an optical fiber end surface imaging diagram obtained by using an abnormal line synchronization signal generator in a confocal scanning colonoscopy system. It can be seen that the image has obvious wave-shaped distortion at the edge. FIG. 19 is an optical fiber end surface imaging diagram obtained by processing the line synchronization signal generated by the abnormal device using the application. It can be seen that the wave distortion of the outer contour has been eliminated in the processed image, and the image is clear and intuitive, which illustrates the practical value of the application.

[0103] In some examples, the above-mentioned self-attention-based long short-term memory network model is composed of multiple self-attention-based long short-term memory layers, the above-mentioned self-attention-based long short-term memory layer is composed of multiple long short-term memory units and a self-attention mechanism, each long short-term memory unit is composed of a forget gate, an input gate, an output gate and a unit state, the above-mentioned self-attention mechanism is connected with each of the above-mentioned long short-term memory units, and the above-mentioned self-attention mechanism is used to calculate attention weights to obtain an output by weighted summation of the attention weights and the output of the long short-term memory unit.

[0104] In some examples, the above-mentioned self-attention-based long short-term memory network model is composed of multiple self-attention-based long short-term memory layers, the above-mentioned self-attention-based long short-term memory layer is composed of multiple long short-term memory units and a self-attention mechanism, each long short-term memory unit is composed of a forget gate, an input gate, an output gate and a unit state, the above-mentioned self-attention mechanism is connected with each of the above-mentioned long short-term memory units, and the above-mentioned self-attention mechanism is used to calculate attention weights to obtain an output by weighted summation of the attention weights and the output of the long short-term memory unit.

[0105] In some examples, the interval time of the line synchronization signal has strong nonlinearity and random characteristics, and the prediction effect error of the conventional prediction method is large. The LSTM network has strong time series prediction performance and nonlinearity processing capability, so the predicted data is obtained by selecting the long short-term memory network LSTM time series prediction, and the construction steps of the LSTM neural network model are as follows:

[0106] The interval time of line synchronization signal has strong nonlinearity and random characteristics, and the application provides a ST-LSTM (Self Attention-Long Short Term Memory) long short term memory network based on self attention, the traditional LSTM network mainly focuses on the input at the current time and the hidden state at the previous time, and the ST-LSTM network provided in the application adds a self attention mechanism in the network, so that the model can focus on the information of each position of the time sequence, dynamically allocate weights according to the importance of different position information, and make the model have better prediction performance.

[0107] The ST-LSTM network is mainly composed of a plurality of ST-LSTM layers, and the network architecture of the ST-LSTM layer is as shown in FIG. 13, the self attention mechanism is added to the output of each LSTM unit to calculate the attention weight, and finally the attention weight and the output of the LSTM unit are weighted and summed to obtain the output. In FIG. 13, x t is the input value of the network at the current time, M t is the unit state at the current time, h t is the output value of the network at the current time.

[0108] An ST-LSTM layer is composed of a plurality of LSTM units, each LSTM unit is composed of a forgetting gate, an input gate, an output gate and a unit state, and the structure is as shown in FIG. 13.

[0109] The calculation process of each part in the LSTM unit is as follows: L t = sigmoid (W l x t +W l h t-1 +b l ) F t = sigmoid (W f x t +W f h t-1 +b f ) S t = sigmoid (W s x t +W s h t-1 +b s ) M t = M t-1 L t +F t S t P t = sigmoid (W p x t +W p h t-1+b p ) h t =P t ·tanh(M t )

[0110] In the above formula, x t , h t , M t , L t , F t , S t , P t are respectively the input variable of the network at time t, the output variable of the network, the parameter of the state of the memory unit of the network, the forget gate, the input gate, the transmission intermediate state, and the output gate; sigmoid and tanh are activation functions, W l , W f , W s , W p are respectively the weight matrix between the corresponding gate and the input layer and the hidden layer, and b l , b f , b s , b p are respectively the corresponding bias vector. The input of the ST-LSTM network is the signal data at the historical time, which is a two-dimensional data matrix, the first dimension is the point information, and the second dimension is the time information, and the signal value at the next time is taken as the output (the input and the output are both the row synchronous signal interval time counting sequence),

[0111] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0112] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program code.

[0113] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in conjunction with the flowcharts.

[0114] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks or in conjunction with the flowcharts.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in conjunction with the flowcharts.

[0116] The embodiments of the present application also provide a computer program product, which comprises computer software instructions, when the computer software instructions are run on a processing device, the processing device executes the flow of the synchronization signal abnormal jitter elimination in the corresponding embodiments

[0117] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that the computer can store or be integrated into a data storage device such as a server, data center, etc. containing one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0119] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0120] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0121] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0122] If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods of various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0123] The above, the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. Industrial applicability

[0124] The filter design method provided by the embodiments of the present application can effectively remove or weaken abnormal spectral components in the video synchronization signal, and can significantly improve the performance and reliability of the video image processing system. Through the filter obtained by the filter design method, the image quality can be effectively improved, and the problem of abnormal devices causing line synchronization signals can be solved, thereby avoiding the problem of wave-shaped or misaligned line data in the finally displayed image. Therefore, the filter design method, the filter IP and the prediction method of the line synchronization interval time provided by the embodiments of the present application have industrial applicability.

Claims

1. A filter design method for abnormal line synchronization signal, characterized by, The method comprises the following steps: statistical characteristic analysis is performed on the abnormal line synchronization signal interval time count sequence to obtain abnormal spectral characteristic distribution data; filter coefficients of a FIR digital filter are determined in combination with the abnormal spectral characteristic distribution data.

2. The filter design method for abnormal line sync signal according to claim 1, wherein, The statistical characteristic analysis performed on the abnormal line synchronization signal interval time count sequence to obtain abnormal spectral characteristic distribution data comprises the following steps: spectrum analysis is performed on the abnormal line synchronization signal interval time count sequence to obtain spectrum distribution data; image quality corresponding to the abnormal line synchronization signal interval time count sequence is determined; the abnormal spectral characteristic distribution data is determined according to the spectrum distribution data and the image quality.

3. The filter design method for abnormal line sync signal according to claim 1 or 2, characterized in that, The spectrum distribution data comprises a maximum amplitude in a frequency domain and a frequency corresponding to the maximum amplitude in the frequency domain.

4. The filter design method for abnormal line sync signal according to claim 3, wherein, The image quality is determined according to image resolution and an image resolution threshold.

5. The filter design method for abnormal line sync signal according to claim 3, wherein, The filter coefficients of the FIR digital filter are determined in combination with the abnormal spectral characteristic distribution data, which comprises the following steps: a stopband frequency and a passband frequency are determined according to the abnormal spectral characteristic distribution data; a Chebyshev filter is designed by using an equal-ripple optimal approximation method according to the stopband frequency, the passband frequency and an empirical parameter to obtain the filter coefficients.

6. A filter IP, characterized by, The method comprises target filter coefficients, wherein the target filter coefficients are obtained by the filter design method for abnormal line synchronization signals according to any one of claims 1 to 5.

7. A method for predicting the line synchronization interval time based on a neural network, for the filter IP of claim 6, characterized in that, The method comprises the following steps: a synchronization signal interval is counted based on an internal clock of an FPGA chip to obtain a synchronization signal interval time count sequence; the synchronization signal interval time count sequence is filtered by the filter IP to obtain a to-be-processed synchronization signal interval count sequence; line synchronization signal interval time prediction is performed by using the to-be-processed synchronization signal interval count sequence and a self-attention-based long short-term memory network model to obtain predicted line synchronization interval time.

8. The method of predicting a row sync interval time based on a neural network according to claim 7, wherein, The line synchronization signal interval time prediction performed by using the to-be-processed synchronization signal interval count sequence and the self-attention-based long short-term memory network model comprises the following steps: information at different time points in the to-be-processed synchronization signal interval count sequence is dynamically assigned a weight according to the importance of the information, and prediction is performed in combination with the weight.

9. The method for predicting row synchronization interval time based on neural network according to claim 7, characterized in that: The synchronization signal interval is counted based on the internal clock of the FPGA chip to obtain the synchronization signal interval time count sequence, which comprises the following steps: in the case of detecting a rising edge of a line synchronization signal, the count of the internal clock of the FPGA chip is cleared and an accumulation calculation operation is performed; in the case of monitoring a next rising edge of a synchronization signal, the accumulation calculation operation is stopped and an accumulated value is obtained; the accumulation calculation operation process is repeated, and a plurality of accumulated values are taken as the synchronization signal interval time count sequence.

10. The method of claim 7, wherein, The self-attention-based long short-term memory network model is composed of multiple self-attention-based long short-term memory layers, each of which is composed of multiple long short-term memory units and a self-attention mechanism, each long short-term memory unit is composed of a forget gate, an input gate, an output gate and a cell state, the self-attention mechanism is connected with each long short-term memory unit, and the self-attention mechanism is used for calculating attention weights, and the attention weights are weighted and summed with the output of the long short-term memory unit to obtain the output.

Images