An algorithm for displacement extraction using speckle centroid array matching

Abstract

This invention discloses an algorithm for displacement extraction using speckle centroid array matching, comprising the following steps: S1, determining the number of speckles and the range of each speckle based on the grayscale information of each pixel in each row of the image; S2, calculating the centroid of each speckle using grayscale values ​​based on the range of each speckle, and forming an array of centroids for each row of speckles; S3, matching the centroids of speckles in adjacent rows based on the maximum displacement of adjacent rows, and forming a new array of matched centroids; S4, calculating the displacement information of adjacent rows based on the matched centroid array.

Description

Technical Field

[0001] This invention relates to the field of micro-motion sensing technology, specifically a method for extracting adjacent row displacements from video signals based on speckle centroid array matching, and then studying the motion characteristics of objects. Background Technology

[0002] A rolling shutter refers to a shutter scheme in CMOS image sensors that exposes and reads out each frame of an image sequentially from top to bottom. When the subject moves, it causes image blurring, known as the "jelly effect." Therefore, previous research mainly focused on solving, reducing, or eliminating image distortion through image processing. Meanwhile, some researchers have utilized the mechanism of line-by-line exposure, comparing changes in adjacent lines of the image to increase the effective sampling frequency to the order of the product of frames per second and the number of lines per frame, thus extracting motion information.

[0003] Researchers from MIT, the University of Ottawa, Hebei University of Technology, and the U.S. Naval Research Laboratory have all conducted research on extracting displacement or frequency information based on the exposure mechanism of rolling shutter cameras, achieving some research results. The main challenges currently faced are: firstly, the extracted vibration frequency is relatively low, generally below 1 kHz; secondly, the camera's rolling shutter direction must be parallel to the object's direction of motion, which limits the further widespread application of this technology; and thirdly, the measurement distance is relatively short, generally less than 5 meters.

[0004] To address the problems existing in motion detection of current rolling shutter cameras, the applicant introduced laser speckle, utilizing the modulation of the laser speckle by a vibrating object to achieve long-distance signal acquisition. The related detection system design has been patented (Application No.: CN202111454709.5, Publication No.: CN114156724A, Publication Date: 20220308). To extract displacement information from the comparison of information in adjacent rows, the applicant proposed a novel algorithm for displacement extraction based on speckle centroid array matching. Summary of the Invention

[0005] The purpose of this invention is to propose an algorithm for displacement extraction based on speckle centroid array matching. This method can take advantage of the high stability of speckle centroids and, based on the characteristic that different rows have different exposure times in rolling shutter camera imaging, extract the average displacement of adjacent rows by matching speckle centroid arrays, thereby establishing a transformation path from visual perception to actual motion detection.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An algorithm for displacement extraction using speckle centroid array matching includes the following steps:

[0008] S1. Based on the grayscale information of each pixel in each row of the image, determine the number of speckles and the range of each speckle.

[0009] S2. Based on the range of each speckle, calculate the centroid of each speckle using gray values, and form an array with the centroids of each row of speckles.

[0010] S3. Match the speckle centroids of adjacent rows based on the maximum displacement of adjacent rows, and the matched centroids form a new array.

[0011] S4. Calculate the displacement information of adjacent rows based on the matched centroid array.

[0012] Preferably, the process of constructing the number and range of speckle patterns in step S1 is as follows:

[0013] S11. Using image processing functions, convert the speckle image captured by the rolling shutter camera into a data array I(x, y), where x and y represent the column and row, respectively, and I is the gray value of the pixel defined by the coordinate.

[0014] S12. Calculate the average value of all pixels in the i-th row, set the gray values ​​of pixels with gray values ​​lower than the average value to zero, and define the new data array as I′(x, y).

[0015] S13. Points with a gray value of zero divide the y-th row into multiple speckle patterns, and the left and right boundaries of each speckle are defined as x, y ... l and x r .

[0016] Preferably, the specific process of step S2 is as follows:

[0017] S21. Based on the speckle boundary, calculate the centroid of each speckle using the following formula.

[0018]

[0019] In the formula, x l and x r These are the left and right boundaries of the speckle, respectively, and I′(x, y) is the gray level of the pixel;

[0020] S22. Sort the obtained centroid coordinates by pixel coordinate position to obtain the centroid coordinate index, and convert it into absolute centroid coordinates;

[0021] S23. For the centroid coordinates obtained in each row, take the row number as the ordinate and the absolute coordinate of the centroid as the abscissa to form a centroid array C(x′, y).

[0022] Preferably, the specific process of step S3 is as follows:

[0023] S31. Calculate the maximum displacement D of the adjacent rows based on the target's speed and the time difference between adjacent rows;

[0024] S32. Based on the maximum displacement D of the adjacent rows, for each speckle centroid, calculate its displacement with each speckle centroid in the next row. Those with a displacement greater than the maximum displacement are considered mismatched. The result of a match should be that the displacements of the matched centroids are the same within a certain error range.

[0025]

[0026] S33. Based on the matching results, delete the unmatched centroid elements and form a new array C′(x′, y) with the matched centroids.

[0027] Preferably, the specific steps for solving the adjacent row displacements in step S4 are as follows:

[0028] S41. The x-coordinate (absolute position) of the speckle centroid in the next row is subtracted from the x-coordinate of the centroid that matches it in the previous row. This subtraction is used as the displacement of the matching centroid, as shown in the following formula:

[0029] C x′，y+1 -C x′，y

[0030] S42. Perform the same operation on all centroids in the next row to find the displacement of each matching centroid;

[0031] S43. Sum the displacements of all matched centroids and divide by the number of matched centroids N to find the average displacement, which is the displacement between adjacent rows.

[0032]

[0033] By adopting the above technical solution, the present invention has the following advantages compared with the prior art:

[0034] 1. Based on the high stability of speckle centroids, this invention proposes a method for calculating the displacement of adjacent rows based on speckle centroid array matching, utilizing the continuity of the centroid and the translational nature of its displacement.

[0035] 2. This invention utilizes the stability characteristics of the speckle centroid. As long as the displacement of the speckle centroid has a component in the horizontal direction, the displacement between adjacent rows can be extracted, breaking through the limitation of the background technology that the direction of movement must be parallel to the direction of the roller blind.

[0036] 3. This invention utilizes the large number of speckle patterns and the matching of the centroid array of inter-row speckle patterns, using the average value of multiple displacements as the inter-row displacement, thereby improving the robustness of displacement information extraction. Attached Figure Description

[0037] Figure 1 This is a flowchart of the present invention;

[0038] Figure 2 This is a data processing flowchart of an embodiment of the present invention;

[0039] Figure 3 The graph shows the frequency measurement results and language measurement results output by the micro-motion extraction of this invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0041] Example

[0042] This invention mainly relates to an algorithm for displacement extraction using speckle centroid array matching. The following is a detailed description of this embodiment with reference to the accompanying drawings.

[0043] Please see Figure 1 An algorithm for displacement extraction using speckle centroid array matching includes the following steps:

[0044] S1. Based on the grayscale information of each pixel in each row of the image, determine the number of speckles and the range of each speckle.

[0045] S2. Based on the range of each speckle, calculate the centroid of each speckle using gray values, and form an array with the centroids of each row of speckles.

[0046] S3. Match the speckle centroids of adjacent rows based on the maximum displacement of adjacent rows, and the matched centroids form a new array.

[0047] S4. Calculate the displacement information of adjacent rows based on the matched centroid array.

[0048] Preferably, the process of constructing the number and range of speckle patterns in step S1 is as follows:

[0049] S11. Using MATLAB's image processing functions, convert the speckle image acquired by the rolling shutter camera into a data array I(x, y), where x and y represent the column and row, respectively, and I is the gray value of the pixel defined by the coordinate.

[0050] S12. Calculate the average value of all pixels in the i-th row, set the gray values ​​of pixels with gray values ​​lower than the average value to zero, and define the new data array as I′(x, y).

[0051] S13. Points with a gray value of zero divide the y-th row into multiple speckle patterns, and the left and right boundaries of each speckle are defined as x, y ... l and x r .

[0052] Preferably, the specific process of step S2 is as follows:

[0053] S21. Based on the speckle boundary, calculate the centroid of each speckle using the following formula.

[0054]

[0055] In the formula, x l and x r These are the left and right boundaries of the speckle, respectively, and I′(x, y) is the gray level of the pixel;

[0056] S22. Convert the obtained centroid coordinates into global coordinates of the absolute position of the pixel;

[0057] S23. For the centroid coordinates obtained in each row, take the row number as the ordinate and the absolute coordinate of the centroid as the abscissa to form a centroid array C(x′, y).

[0058] Preferably, the specific process of step S3 is as follows:

[0059] S31. Calculate the maximum displacement D of the adjacent rows based on the target's speed and the time difference between adjacent rows;

[0060] S32. Based on the maximum displacement D of the adjacent rows, for each speckle centroid, calculate its displacement with each speckle centroid in the next row. Those with a displacement greater than the maximum displacement are considered mismatched. The result of a match should be that the displacements of the matched centroids are the same within a certain error range.

[0061]

[0062] S33. Based on the matching results, delete the elements that do not match the centroids, and form a new array C′(x′, y) with the matching centroids.

[0063] Preferably, the specific steps for solving the adjacent row displacements in step S4 are as follows:

[0064] S41. The x-coordinate (absolute position) of the speckle centroid in the next row is subtracted from the x-coordinate of the centroid that matches it in the previous row. This subtraction is used as the displacement of the matching centroid, as shown in the following formula:

[0065] C x′，y -C x′，y

[0066] S42. Perform the same operation on all centroids in the next row to find the displacement of each matching centroid;

[0067] S43. Sum the displacements of all matched centroids and divide by the number of matched centroids N to find the average displacement, which is the displacement between adjacent rows.

[0068]

[0069] To provide a more intuitive understanding of this invention, a specific data processing procedure will be used as an example for illustration, such as... Figure 2 As shown. The experimental section first tested the realization of this invention in sensing micro-vibrations induced by sound. Based on the extraction of adjacent row micro-displacements through centroid array matching, the micro-displacements were sorted and normalized according to time sequence, and finally written into a speech file, as shown. Figure 3 As shown. Figure 3 The first result shows the detection result of a 1000Hz single-frequency signal, with a reconstructed frequency of 999.7Hz and a relative error of 0.03%. Figure 3 The program then displays a restored waveform of the speech, achieving a relatively high signal-to-noise ratio and intelligibility.

[0070] The results show that the present invention performs well in practical applications of establishing micro-displacement sensing based on speckle images, and can integrate the advantages of existing research on motion sensing, and has significant advantages in micro-motion sensing and system implementation.

[0071] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An algorithm for displacement extraction using speckle centroid array matching, characterized in that, Includes the following steps: S1. Based on the grayscale information of each pixel in each row of the image, determine the number of speckles and the range of each speckle, including: using MATLAB image processing functions to convert the speckle image acquired by the rolling shutter camera into a data array. Where x and y represent the column and row in the coordinate system, respectively, and I is the grayscale value of the pixel defined by that coordinate. The average value of all pixels in the i-th row is calculated, and pixels with grayscale values ​​lower than the average value are set to zero. The resulting new data array is defined as follows: Points with a grayscale value of zero divide the y-th row into multiple speckle patterns, and the left and right boundaries of each speckle are defined as follows: and ; S2. Based on the range of each speckle, calculate the centroid of each speckle using gray values, and form an array with the centroids of each row of speckles. S3. Based on the maximum displacement of adjacent rows during the actual target motion, match the speckle centroids of adjacent rows. The matched centroids form a new array. This includes: calculating the maximum displacement D of adjacent rows based on the target's motion velocity and the time difference between adjacent rows; calculating the displacement between each speckle centroid and each speckle centroid in the next row based on the maximum displacement D of adjacent rows; considering those greater than the maximum displacement as mismatched, and those less than or equal to the maximum displacement as matched; and based on the matching results, deleting elements without matched centroids and forming a new array of matched centroids. ; S4. Calculate the displacement information of adjacent rows based on the matched centroid array.

2. The algorithm for displacement extraction using speckle centroid array matching as described in claim 1, characterized in that, The specific process of step S2 is as follows: S21. Based on the speckle boundary, calculate the centroid of each speckle using the following formula. In the formula, and These are the left and right boundaries of the speckle pattern, respectively. It is the grayscale value of the pixel; S22. Convert the obtained centroid coordinates into global coordinates of the absolute position of the pixel; S23. For the centroid coordinates obtained in each row, form a centroid array with the row number as the ordinate and the absolute coordinate of the centroid as the abscissa.

3. The algorithm for displacement extraction using speckle centroid array matching as described in claim 1, characterized in that, The specific steps for solving the adjacent row displacement in step S4 are as follows: S41. Subtract the x-coordinate of the centroid of the next row from the x-coordinate of the centroid that matches it in the previous row, and use that as the displacement of the matching centroid. S42. Perform the same operation on all centroids in the next row to find the displacement of each matching centroid; S43. Sum the displacements of all matched centroids and divide by the number of matched centroids to find the average displacement, which is the displacement between adjacent rows.

Images