A distance measurement system and method based on moiré

Abstract

The present invention discloses a moiré-based distance measurement system and method, comprising: a moiré image processing module for processing an image captured by a camera with the screen as the main subject, and extracting a clear moiré binary image; a moiré feature extraction module for constructing a frequency vector based on the position of the pulse region in the moiré spectrum diagram, and extracting the moiré statistical features based on the parameters of the frequency vector; a distance measurement module for verifying the validity of the statistical features after physical unit conversion. If the features are valid, a model between the moiré statistical features and the distance is constructed to achieve distance measurement from the camera to the screen; if the features are invalid, a thumbnail downsampling method is used to obtain a valid moiré image, and then moiré feature extraction is performed again. The present invention utilizes the moiré formed by the superposition of the camera front-end color filter array and the screen projection pixels to achieve camera-to-screen distance perception, with extremely high distance measurement accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of camera ranging, and in particular relates to a moiré-based ranging system and method for measuring the distance from a camera device to a screen device. Background Art

[0002] In the fields of human-computer interaction applications, autonomous driving, industrial robot positioning, etc., the camera is required to estimate the depth of field of objects in the image while imaging, so as to realize a wide range of application services based on interactive distance.

[0003] Existing camera depth measurement technologies mainly include monocular camera ranging, two-sided stereo vision ranging, and structured light ranging. Monocular camera ranging relies on a priori calibration of the size of objects in the environment and lacks scalability. Binocular stereo vision ranging estimates depth of field based on binocular parallax. The algorithm is efficient and low-cost, but it is easily affected by ambient lighting and is not suitable for scenes with single color features and lack of texture. Structured light ranging uses a characterized light source to project onto objects, making up for the deficiency of two-sided vision ranging in scenes without inherent texture. However, structured light ranging relies on dedicated projection and receiving equipment and is relatively expensive. In addition, existing camera depth measurement technologies are difficult to meet the requirements of scenes with ultra-high sensitivity requirements for position measurement.

[0004] Therefore, there is an urgent need to realize an efficient, low-cost, ultra-high-precision camera ranging method. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a moiré-based ranging system and method to solve the problems of insufficient accuracy, high dependence on peripherals, and high cost of existing camera ranging technology.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A moiré-based distance measurement system of the present invention is applied to a camera device in a camera-screen interaction scenario, comprising: a moiré image processing module, a moiré feature extraction module, and a distance measurement module;

[0008] The moiré image processing module is used to process the screen-based image captured by the camera, and extract a clear moiré binary image using effective area cropping and image enhancement operations;

[0009] A moiré feature extraction module is configured to perform frequency domain analysis on the moiré binary image, construct a frequency vector based on the location of the pulse region in the moiré spectrum, extract the moiré statistical features based on the parameters of the frequency vector, convert the statistical features into physical units, and send them to the distance measurement module;

[0010] The distance measurement module verifies the validity of the statistical features after the physical unit conversion. If the features are valid, a model between the moiré statistical features and the distance is constructed to achieve the distance measurement from the camera to the screen. If the features are invalid, a valid moiré image is obtained through the thumbnail downsampling method, and then the moiré feature extraction is performed again.

[0011] Furthermore, the camera device includes but is not limited to a camera module with image acquisition and data transmission functions and a mobile device with a camera module.

[0012] Furthermore, the screen refers to a display screen with a regular pixel arrangement other than a silver screen, and differences in display technology, pixel geometry, and pixel arrangement of the screen device are allowed.

[0013] Furthermore, the image processing operations in the moiré image processing module specifically include:

[0014] The effective central area is selected from the original image for cropping to eliminate the image distortion and dark corners caused by the lens effect. The number of pixels in the cropped area is P. c The number of pixels on the shorter side of the image P m The ratio is denoted as σ;

[0015] Perform histogram equalization on the cropped image to enhance the image contrast;

[0016] Adopting multi-scale Gaussian function and adaptively adjusting image brightness value based on nonlinear gamma correction, it further reduces the dark corner situation of the image.

[0017] Perform image binarization and median filtering operations to remove image noise and obtain a clear moiré-free binary image.

[0018] Furthermore, the statistical characteristics of the moiré pattern include the spatial frequency f of the moiré pattern. m and the propagation direction of spatial frequency θ m , propagation direction θ m Orthogonal to the inherent fringe direction of the moiré pattern.

[0019] Furthermore, the frequency vector in the moiré feature extraction module refers to the vector connection from the origin to the pulse area in the moiré spectrum diagram, using polar coordinates (f m ,θ m ); its specific value is calculated by the Cartesian coordinates (u, v) of the pulse area:

[0020]

[0021]

[0022] Furthermore, the physical unit conversion in the moiré feature extraction module refers to converting the spatial frequency of the moiré from a pixel unit to an actual distance measurement unit measured in millimeters.

[0023] Furthermore, the distance measurement module needs to obtain the spatial frequency f of the screen to construct a model between the statistical characteristics of the moiré pattern and the distance. s , the spatial frequency f of the color filter array at the front end of the camera c and the camera focal length f; combined with the spatial frequency f of the moiré m and the propagation direction of the spatial frequency θ m , calculate the distance d from the camera to the screen:

[0024]

[0025] Furthermore, the distance measurement module is based on the spatial frequency f of the moiré pattern. m The validity of the statistical features of the moiré is judged by the range; if the spatial frequency of the moiré is too high, that is, the clarity of the moiré binary image extracted by the moiré image processing module is lower than the threshold T c At this time, adjacent moiré splines are distributed in the same pixel, resulting in multiple pairs of pulse areas with similar energy in the spectrum diagram. It is impossible to determine the target pulse area representing the statistical characteristics of the moiré based on the energy value. The effective moiré statistical characteristics need to be calculated through the thumbnail downsampling method.

[0026] Furthermore, the thumbnail downscaling method specifically includes:

[0027] Downsample the image at a sampling rate δ to generate a thumbnail;

[0028] Evaluate the clarity of the thumbnail and adjust the sampling rate δ until the image clarity reaches a local optimum;

[0029] Perform moiré statistical feature extraction on clear moiré thumbnails;

[0030] According to the statistical characteristics of the moiré patterns of the thumbnail and the sampling rate δ, the accurate features of the moiré pattern binary image extracted by the moiré pattern image processing module are reconstructed.

[0031] The present invention also provides a distance measurement method based on moiré patterns, based on the above system, comprising the following steps:

[0032] 1) Use a camera to capture video of the interactive screen to be measured at a fixed frame rate, and the video frame contains a moiré image with low spatial frequency;

[0033] 2) Perform moiré image processing on the video frame sequence and crop the effective area in the center of the image to extract a clear moiré binary image;

[0034] 3) Perform frequency domain transformation on the extracted moiré binary image, find the pulse area corresponding to the moiré feature from the spectrum, and extract the moiré statistical features based on the location of the pulse area;

[0035] 4) Verify the validity of the extracted moiré statistical features. If the features are invalid, perform thumbnail downsampling until accurate moiré statistical features are extracted.

[0036] 5) Obtain the spatial frequency of the screen and camera color filter array, calculate the distance from the camera to the screen based on the statistical characteristics of the moiré pattern, and output the ranging result of the corresponding frame.

[0037] Beneficial effects of the present invention:

[0038] 1. Ultra-high ranging accuracy: The distance from the camera to the screen is perceived using the moiré pattern formed by the superposition of the camera's front-end color filter array and the screen's projected pixels, achieving extremely high ranging accuracy. Any slight change in distance will cause a significant change in the moiré pattern characteristics. Therefore, the moiré-based ranging method of the present invention has ultra-high ranging accuracy.

[0039] 2. Low cost: No additional signal transmitting and receiving equipment is required. Only the software algorithm on the camera device side can be used to achieve real-time distance measurement of the camera relative to the interactive screen.

[0040] 3. Marker-free: Unlike traditional positioning methods based on visual markers, the present invention does not require prior calibration of feature points in three-dimensional space and has scalability across equipment and environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a diagram showing the architecture of a ranging system according to an embodiment of the present invention;

[0042] Figure 2 Schematic diagram of the distance measurement system of the present invention;

[0043] Figure 3 This is a schematic diagram of the principle of moiré image processing in the present invention;

[0044] Figure 4 Schematic diagram of the moiré feature extraction in the present invention;

[0045] Figure 5 Schematic diagram of the principle of calculating effective frequency characteristics in the present invention. DETAILED DESCRIPTION

[0046] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0047] Reference Figure 1 As shown, a distance measurement system based on moiré patterns of the present invention is applied to a camera device in a camera-screen interaction scenario, comprising: a moiré pattern image processing module, a moiré pattern feature extraction module, and a distance measurement module;

[0048] The moiré image processing module is used to process the screen-based image captured by the camera, and extract a clear moiré binary image using effective area cropping and image enhancement operations;

[0049] A moiré feature extraction module is configured to perform frequency domain analysis on the moiré binary image, construct a frequency vector based on the location of the pulse region in the moiré spectrum, extract the moiré statistical features based on the parameters of the frequency vector, convert the statistical features into physical units, and send them to the distance measurement module;

[0050] The distance measurement module verifies the validity of the statistical features after the physical unit conversion. If the features are valid, a model between the moiré statistical features and the distance is constructed to achieve the distance measurement from the camera to the screen. If the features are invalid, a valid moiré image is obtained through the thumbnail downsampling method, and then the moiré feature extraction is re-executed.

[0051] In a specific example, the camera device includes but is not limited to a camera module with image acquisition and data transmission functions and a mobile device with a camera module, such as a smart phone, a smart watch, a smart glasses, a head-mounted display device, a smart handle, a mobile robot, an intelligent robotic arm device, etc.

[0052] In a specific example, the screen refers to a display screen with a regular pixel arrangement other than a silver screen, and differences in display technology, pixel geometry, and pixel arrangement of the screen device are allowed; the display technology covers a variety of screen display technologies such as LCD, OLED, LED, etc.

[0053] The image processing operations in the moiré image processing module specifically include:

[0054] The effective central area is selected from the original image for cropping to eliminate the image distortion and dark corners caused by the lens effect. The number of pixels in the cropped area is P. c The number of pixels on the shorter side of the image P m The ratio is denoted as σ;

[0055] Perform histogram equalization on the cropped image to enhance the image contrast;

[0056] Adopting multi-scale Gaussian function and adaptively adjusting the image brightness value based on nonlinear gamma correction, it further reduces the image vignetting.

[0057] Perform image binarization and median filtering operations to remove image noise and obtain a clear moiré-free binary image.

[0058] Specifically, the statistical characteristics of the moiré pattern include the spatial frequency f of the moiré pattern. m and the propagation direction of spatial frequency θ m , propagation direction θ m Orthogonal to the inherent fringe direction of the moiré pattern.

[0059] Specifically, the frequency vector in the moiré feature extraction module refers to the vector connection from the origin to the pulse area in the moiré spectrum diagram, using polar coordinates (f m ,θ m ); its specific value is calculated by the Cartesian coordinates (u, v) of the pulse area:

[0060]

[0061]

[0062] Specifically, the physical unit conversion in the moiré feature extraction module refers to the conversion of the spatial frequency of the moiré from a pixel unit to an actual distance measurement unit measured in millimeters.

[0063] Specifically, the distance measurement module needs to obtain the spatial frequency f of the screen to construct the model between the statistical characteristics of the moiré pattern and the distance. s , the spatial frequency f of the color filter array at the front end of the camera c and the camera focal length f; combined with the spatial frequency f of the moiré m and the propagation direction of the spatial frequency θ m , calculate the distance d from the camera to the screen:

[0064]

[0065] Specifically, the distance measurement module is based on the spatial frequency f of the moiré pattern. m The validity of the statistical features of the moiré is judged by the range; if the spatial frequency of the moiré is too high, that is, the clarity of the moiré binary image extracted by the moiré image processing module is lower than the threshold T c At this time, adjacent moiré splines are distributed in the same pixel, resulting in multiple pairs of pulse areas with similar energy in the spectrum diagram. It is impossible to determine the target pulse area representing the statistical characteristics of the moiré based on the energy value. The effective moiré statistical characteristics need to be calculated through the thumbnail downsampling method.

[0066] Furthermore, the thumbnail downscaling method specifically includes:

[0067] Downsample the image at a sampling rate δ to generate a thumbnail;

[0068] Evaluate the clarity of the thumbnail and adjust the sampling rate δ until the image clarity reaches a local optimum;

[0069] Perform moiré statistical feature extraction on clear moiré thumbnails;

[0070] According to the statistical characteristics of the moiré patterns of the thumbnail and the sampling rate δ, the accurate features of the moiré pattern binary image extracted by the moiré pattern image processing module are reconstructed.

[0071] The present invention also provides a distance measurement method based on moiré patterns, based on the above system, comprising the following steps:

[0072] 1) Use a camera to capture video of the interactive screen to be measured at a fixed frame rate, and the video frame contains a moiré image with low spatial frequency;

[0073] 2) Perform moiré image processing on the video frame sequence and crop the effective area in the center of the image to extract a clear moiré binary image;

[0074] 3) Perform frequency domain transformation on the extracted moiré binary image, find the pulse area corresponding to the moiré feature from the spectrum, and extract the moiré statistical features based on the location of the pulse area;

[0075] 4) Verify the validity of the extracted moiré statistical features. If the features are invalid, perform thumbnail downsampling until accurate moiré statistical features are extracted.

[0076] 5) Obtain the spatial frequency of the screen and camera color filter array, calculate the distance from the camera to the screen based on the statistical characteristics of the moiré pattern, and output the ranging result of the corresponding frame.

[0077] Reference Figure 2 As shown, the principle of generating a low spatial frequency moiré image in step 1) is that the color filter array at the front end of the camera and the projection of the screen pixels on the camera color filter array are both high-frequency grids. The superposition of two high-frequency grids with close frequencies will produce a low spatial frequency moiré pattern, which will be displayed in the camera imaging.

[0078] Reference Figure 3 As shown, the moiré image processing method in step 2) is:

[0079] 21) Select the effective central area in the original image for cropping to eliminate the image distortion and dark corners caused by the lens effect. The number of pixels in the cropped area is P. c The number of pixels on the shorter side of the image P m The ratio is denoted as σ;

[0080] 22) Perform histogram equalization on the cropped image to enhance image contrast;

[0081] 23) Using multi-scale Gaussian function, based on nonlinear gamma correction, the image brightness value is adaptively adjusted to further reduce the dark corner situation of the image;

[0082] 24) Perform image binarization and median filtering operations to remove image noise and obtain a clear moiré-free binary image.

[0083] Reference Figure 4 As shown, the moiré spectrum diagram in step 3) contains a vector connection from the origin to the pulse highlight area. This vector is the frequency vector f of the moiré, which is used to calculate the statistical characteristics of the moiré. The specific method is:

[0084] 31) According to the principle that the foreground target is brighter than the background, the original spectrum is adaptively binarized to extract all the highlighted pulse areas C i ;

[0085] 32) Calculate the center coordinates of all pulse areas (u i , v i ), then the line connecting each pulse area and the origin constitutes a frequency vector; calculate the distance r between the center coordinates of all pulse areas and the origin i And the area S of each pulse region i , select the one that satisfies the relationship S i >ξ, ξ represents the minimum threshold for determining whether a pulse belongs to noise, and the pulse area closest to the origin is the target pulse area, and the vector from the origin to the target pulse highlight area is the target frequency vector f;

[0086] 33) The corresponding polar coordinates of the target frequency vector (f m ,θ m ) is the statistical characteristic of the moiré image: spatial frequency f m and the frequency propagation direction θ m , calculate the statistical characteristics according to the center coordinates (u, v) of the target pulse area:

[0087]

[0088]

[0089] 34) Spatial frequency f mRepresents the propagation direction θ in the effective clipping area m The number of fringe cycles that appear needs to be further multiplied by the coefficient Convert from pixel units to actual distance measurement units in millimeters; where S c Indicates the pixel size of the camera color filter array, P c The number of pixels that represent the side length of the cropped area.

[0090] Reference Figure 5 As shown, the judgment basis for verifying the validity of the moiré statistical feature in step 4) is the spatial frequency range of the moiré; if the spatial frequency of the moiré is too high, that is, the clarity of the moiré binary image extracted by the moiré image processing module is lower than the threshold T c At this time, adjacent moiré splines are distributed in the same pixel, resulting in multiple pairs of pulse regions with similar energies in the spectrum. It is impossible to accurately determine the target pulse region representing the statistical characteristics of the moiré based on the energy value. The thumbnail downsampling method is required to calculate the effective moiré statistical characteristics. The specific method is as follows:

[0091] 41) downsampling the image at a sampling rate δ to generate a thumbnail;

[0092] 42) The clarity of the thumbnail is evaluated according to the Brenner gradient function, and the sampling rate δ is adjusted using the Newton iteration method until the image clarity reaches a local optimum;

[0093] 43) Perform moiré statistical feature extraction on the clear moiré thumbnail and calculate the corresponding spatial frequency f t ;

[0094] 44) The operation of shrinking the image is equivalent to superimposing the original moiré image with a spatial frequency of δP c grid; according to the spatial frequency f of the thumbnail t and the spatial frequency δP of the superimposed grid c , calculate the accurate features of the original moiré binary image extracted by the moiré image processing module:

[0095] f m =f t +δP c .

[0096] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principles of the present invention. These improvements should also be considered as the scope of protection of the present invention.

Claims

1. A moiré-based ranging system, applied to a camera device in a camera-screen interaction scenario, characterized in that: include: Moiré image processing module, moiré feature extraction module and distance measurement module; The moiré image processing module is used to process the screen-based image captured by the camera, and extract a clear moiré binary image using effective area cropping and image enhancement operations; A moiré feature extraction module is configured to perform frequency domain analysis on the moiré binary image, construct a frequency vector based on the location of the pulse region in the moiré spectrum, extract the moiré statistical features based on the parameters of the frequency vector, convert the statistical features into physical units, and send them to the distance measurement module; A distance measurement module verifies the validity of the statistical features after the physical unit conversion. If the features are valid, a model is constructed between the moiré statistical features and the distance to achieve distance measurement from the camera to the screen. If the feature is invalid, a valid moiré image is obtained by thumbnail downscaling, and then the moiré feature extraction is performed again; The moiré statistical characteristics include the spatial frequency f of the moiré m and the propagation direction of spatial frequency θ m , propagation direction θ m Orthogonal to the inherent fringe direction of the moiré pattern; The frequency vector in the moiré feature extraction module refers to the vector connection from the origin to the pulse area in the moiré spectrum diagram, expressed in polar coordinates (f m ,θ m ); its specific value is calculated by the Cartesian coordinates (u, v) of the pulse area: The distance measurement module constructs a model between the statistical characteristics of the moiré pattern and the distance, which requires obtaining the spatial frequency f of the screen. s , the spatial frequency f of the color filter array at the front end of the camera c and the camera focal length f; combined with the spatial frequency f of the moiré m and the propagation direction of the spatial frequency θ m , calculate the distance d from the camera to the screen:

2. The moiré-based ranging system according to claim 1, wherein: The image processing operations in the moiré image processing module specifically include: The effective central area is selected from the original image for cropping to eliminate the image distortion and dark corners caused by the lens effect. The number of pixels in the cropped area is P. c The number of pixels on the shorter side of the image P m The ratio is denoted as σ; Perform histogram equalization on the cropped image to enhance the image contrast; Adopting multi-scale Gaussian function and adaptively adjusting image brightness value based on nonlinear gamma correction, it further reduces the dark corner situation of the image. Perform image binarization and median filtering operations to remove image noise and obtain a clear moiré-free binary image.

3. The moiré-based ranging system according to claim 1, wherein: The distance measurement module is based on the spatial frequency f of the moiré pattern. m The validity of the statistical features of the moiré is judged by the range; if the spatial frequency of the moiré is too high, that is, the clarity of the moiré binary image extracted by the moiré image processing module is lower than the threshold T c At this time, adjacent moiré splines are distributed in the same pixel, resulting in multiple pairs of pulse areas with similar energy in the spectrum diagram. It is impossible to determine the target pulse area representing the statistical characteristics of the moiré based on the energy value. The effective moiré statistical characteristics need to be calculated through the thumbnail downsampling method.

4. The moiré-based ranging system according to claim 1, wherein: The thumbnail downsampling method specifically includes: Downsample the image at a sampling rate δ to generate a thumbnail; Evaluate the clarity of the thumbnail and adjust the sampling rate δ until the image clarity reaches a local optimum; Perform moiré statistical feature extraction on clear moiré thumbnails; According to the statistical characteristics of the moiré patterns of the thumbnail and the sampling rate δ, the accurate features of the moiré pattern binary image extracted by the moiré pattern image processing module are reconstructed.

5. A distance measurement method based on moiré patterns, based on the system according to any one of claims 1 to 4, characterized in that: The steps are as follows: 1) Use a camera to capture video of the interactive screen to be measured at a fixed frame rate, and the video frame contains a moiré image with low spatial frequency; 2) Perform moiré image processing on the video frame sequence and crop the effective area in the center of the image to extract a clear moiré binary image; 3) Perform frequency domain transformation on the extracted moiré binary image, find the pulse area corresponding to the moiré feature from the spectrum, and extract the moiré statistical features based on the location of the pulse area; 4) Verify the validity of the extracted moiré statistical features. If the features are invalid, perform thumbnail downsampling until accurate moiré statistical features are extracted. 5) Obtain the spatial frequency of the screen and camera color filter array, calculate the distance from the camera to the screen based on the statistical characteristics of the moiré pattern, and output the ranging result of the corresponding frame.

Images