Bimodal image capturing method and device, equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, in particular to a bimodal image capturing method, device and equipment and a storage medium, and the method comprises the steps: controlling an infrared laser speckle projector to emit a highly random infrared speckle image to the surface of a to-be-measured target, so as to form light spot distribution on the surface of the to-be-measured target; driving an RGB camera to carry out image acquisition on the surface of the to-be-measured target forming the light spot distribution so as to obtain a fused image; and analyzing the fused image to obtain spectral characteristics and color channel information of the fused image, and separating according to the spectral characteristics and the color channel information of the fused image to obtain an infrared speckle image and a visual image. The defect that an existing monocular laser speckle projection system cannot obtain a visual image and an infrared speckle image at the same time is overcome, and therefore the technical problem that images are not aligned due to multiple times of shooting or device asynchronization in an existing system is solved.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision, and in particular to a dual-modal image capturing method, apparatus, device and storage medium. Background Art

[0002] High-precision depth measurement is one of the important research topics in the field of computer vision. At present, the mainstream depth measurement methods mainly include ToF (Time-of-Flight) and binocular stereo vision technology. ToF obtains the depth information of the target by measuring the flight time or phase change of modulated light. It has the advantages of fast measurement speed, small ambient light interference and strong long-distance measurement capability. However, the measurement accuracy of ToF is limited and can only reach the centimeter level, which is difficult to meet the needs of high-precision measurement tasks. The binocular stereo vision method obtains the disparity map and calculates the depth information by matching the image pairs taken by cameras from two different positions. Although this method achieves matching accuracy at the sub-pixel level, it performs poorly in scenes with sparse textures or large changes in lighting conditions, and has high computational complexity, which limits its use in real-time applications.

[0003] In order to address the limitations of ToF and binocular stereo vision technology, a projection system based on laser speckle images is further proposed. This system uses a laser to generate highly random speckle images through a diffraction grating, and an infrared camera is used to capture these speckle images, thereby reducing the impact of ambient light on the measurement and achieving real-time dynamic measurement. According to the number of cameras used in the system, the laser speckle projection system can be divided into a binocular speckle system and a monocular speckle system. The binocular laser speckle projection system combines the advantages of binocular stereo vision and enhances the feature information of texture-free areas through highly random speckle images, thereby significantly improving the matching accuracy and depth measurement accuracy. However, the binocular speckle system has a high manufacturing cost and a complex calibration step.

[0004] In contrast, the monocular laser speckle projection system only includes an infrared camera and a laser speckle projector, which has the advantages of compact system and low cost. However, since the monocular laser speckle projection system currently on the market usually only relies on infrared cameras to obtain speckle images, it has the defect of not being able to obtain visible images and infrared speckle images at the same time. Due to the inability to obtain synchronously, the system lacks synchronization information or has limited application scenarios.

[0005] Based on this, how to make the monocular laser speckle projection system overcome the defect of being unable to simultaneously acquire visible images and infrared speckle images, so as to avoid image misalignment caused by multiple shots or device asynchrony in the existing system, is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present application provides a dual-modal image capture method, device, equipment and storage medium to overcome the defect that the existing monocular laser speckle projection system cannot simultaneously acquire visible images and infrared speckle images, thereby avoiding the technical problem of image misalignment caused by multiple shooting or device asynchrony in the existing system.

[0007] The technical solution adopted by this application to solve its technical problem is: In a first aspect, the present application provides a dual-modal image capture method, which is applied to a dual-modal image capture system based on an RGB camera and an infrared laser speckle projector, wherein the RGB camera is an RGB camera without IR-CUT, and the method comprises: Controlling the infrared laser speckle projector to emit a highly random infrared speckle image to the surface of the target to be measured, so as to form a spot distribution on the surface of the target to be measured; The RGB camera is driven to collect images of the target surface to be measured where the light spot distribution is formed, so as to obtain a fused image; The fused image is analyzed to obtain the spectral characteristics and color channel information of the fused image, and the infrared speckle image and the visible image are separated according to the spectral characteristics and color channel information of the fused image.

[0008] Optionally, the step of analyzing the fused image to obtain spectral characteristics and color channel information of the fused image, and separating the infrared speckle image and the visible image according to the spectral characteristics and color channel information of the fused image includes: preprocessing the fused image; The preprocessed fused image is input into a generation network to generate an infrared speckle image. The generation network is a set of network architectures in a pre-trained generation adversarial network.

[0009] Optionally, the generation network includes an input layer, a convolutional layer, an encoder-decoder structure and an output layer; The input layer is used to receive the preprocessed fused image with a size of H×W×C, where H is the image height, W is the image width, and C is the number of channels: The convolution layer uses multi-layer convolution operations to gradually extract image features from low-level to high-level, wherein the size of the convolution kernel is 3×3, the stride is 1, the ReLU activation function is used, and each convolution layer is followed by a batch normalization layer; The encoder-decoder structure includes an encoder part and a decoder part, wherein the encoder part is used to gradually reduce the spatial dimension and extract features; The decoder part restores the image size through deconvolution and upsampling operations to restore the spatial resolution of the image, wherein the deconvolution kernel size is 4×4 and the stride is 2; The output layer is a single-channel speckle image, and a 1×1 convolution layer and a Sigmoid activation function are used to map the output to the range of [0, 1] to represent the generated infrared speckle image.

[0010] Optionally, after the step of inputting the preprocessed fused image into a generation network to generate an infrared speckle image, the method includes: Inputting the generated infrared speckle image into a discriminant network, wherein the discriminant network is another set of network architectures in the pre-trained generative adversarial network; Based on the external input real infrared speckle image, the similarity of the generated infrared speckle image is judged.

[0011] Optionally, the discriminant network includes an input layer, a convolutional layer, a fully connected layer and an output layer; The input layer is used to receive the infrared speckle image generated by the generation network and the real infrared speckle image input externally, and the size is H×W×1; The convolution layer includes multiple layers, each of which is followed by a Leaky ReLU activation function and a batch normalization layer, wherein the convolution kernel size is 3×3 and the stride is 2; The fully connected layer is used to flatten the feature map obtained by the convolution layer into a one-dimensional vector, and output a corresponding scalar value based on the flattened one-dimensional vector; The output layer is used to judge the similarity of the generated infrared speckle images according to the scalar value and output the corresponding judgment result, and the range of the scalar value is [0, 1], wherein the closer the scalar value is to 1, the higher the similarity is.

[0012] Optionally, the step of analyzing the fused image to obtain spectral characteristics and color channel information of the fused image, and separating the infrared speckle image and the visible image according to the spectral characteristics and color channel information of the fused image includes: Performing Fourier transform on the fused image to obtain a frequency domain image of the fused image; Acquiring a frequency range corresponding to the speckle image, wherein the frequency range is obtained by analyzing the frequency spectrum of the speckle image; performing high pass filtering on the frequency domain image based on the frequency range; The frequency domain image after high-pass filtering is inversely Fourier transformed to obtain the separated infrared speckle image.

[0013] Optionally, the step of performing high-pass filtering on the frequency domain image based on the frequency range includes: The frequency domain image is processed using a high-pass filter, and the transfer function used by the high-pass filter is: ; in, is the distance to the frequency origin, is the cut-off frequency, is the order of the filter.

[0014] Optionally, the step of analyzing the fused image to obtain spectral characteristics and color channel information of the fused image, and separating the infrared speckle image and the visible image according to the spectral characteristics and color channel information of the fused image includes: Performing wavelet transform on the fused image to obtain a wavelet domain image, and decomposing the wavelet domain image to obtain a wavelet coefficient group; The coefficients in the wavelet coefficient group that are lower than a preset threshold are suppressed, and the suppressed wavelet coefficient group is reconstructed into a spatial domain image to separate and obtain an infrared speckle image.

[0015] Optionally, the wavelet transform formula used in the step of performing wavelet transform on the fused image is: ; in, is the wavelet coefficient, is the image data to be analyzed, is the mother wavelet function, is the scale parameter, is the translation parameter; The formula used in the step of suppressing the coefficients in the wavelet coefficient group that are lower than a preset threshold is: ; in, is the threshold value set; The wavelet coefficient group after the reconstruction and suppression is a spatial domain image, and the formula used in the step of separating and obtaining the infrared speckle image is: ; in, is the wavelet coefficient after processing.

[0016] In a second aspect, the present application provides a dual-modal image capture device based on an RGB camera and an infrared laser speckle projector, using an RGB camera with IR-CUT removed, comprising: A light spot distribution generating module is used to control the infrared laser speckle projector to emit a highly random infrared speckle image to the surface of the target to be measured, so as to form a light spot distribution on the surface of the target to be measured; A fused image acquisition module is used to drive the RGB camera to collect images of the target surface to be measured that forms a light spot distribution, so as to obtain a fused image; The speckle image separation module is used to analyze the fused image to obtain the spectral characteristics and color channel information of the fused image, and separate the spectral characteristics and color channel information of the fused image to obtain the infrared speckle image and the visible image.

[0017] In a third aspect, the present application provides an electronic device, comprising: at least one processor and at least one memory, wherein the memory stores computer-readable instructions; The computer-readable instructions are executed by one or more of the processors to enable the electronic device to implement the dual-modal image capturing method.

[0018] In a fourth aspect, the present application provides a storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions are executed by a processor to implement the above-mentioned dual-modal image capture method.

[0019] The beneficial effects of the present application are as follows: the relevant control module first issues a command to stimulate the infrared laser speckle projector to start working. After being stimulated, the infrared laser speckle projector emits highly random infrared speckle images, which are projected onto the surface of the target to be measured and interact with the texture features of the surface of the target to be measured to form a corresponding spot distribution. This step is the startup stage of the entire system and lays the foundation for subsequent image acquisition; Then, the control module drives the RGB camera to acquire images and captures a fused image including complete speckle information and visible image information at the current frame rate. The corresponding processing module performs image processing. The image processing applies an infrared speckle image separation algorithm to separate infrared speckle and visible light information from the fused image by analyzing the spectral characteristics and color channel information of the fused image, thereby ensuring the independence and integrity of the two images. Since the above two images can be obtained with only one shot, the infrared speckle image and the visible image are completely matched in space and time, thereby avoiding the image alignment problem caused by multiple shots or device asynchrony in the traditional system, and achieving the beneficial effect of overcoming the defect that the existing monocular laser speckle projection system cannot simultaneously obtain visible images and infrared speckle images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of a system structure corresponding to a dual-modal image capturing method provided in an embodiment of the present application; Figure 2 is a schematic flow chart of a dual-modal image capturing method provided in an embodiment of the present application; Figure 3 It is a virtual structural schematic diagram of a dual-modal image capture device based on an RGB camera and an infrared laser speckle projector provided by the present application; Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The present application is further described below in conjunction with the accompanying drawings and embodiments.

[0022] The following will clearly and completely describe the concept, specific structure and technical effects of the present application in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments of the present application, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present application. In addition, all the connection / connection relationships involved in the patent do not simply refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features created in this application can be combined interactively without conflicting with each other.

[0023] Reference Figure 1 , Figure 1 is a schematic diagram of the system structure of the dual-modal image capturing method provided in the embodiment of the present application. Figure 1 To explain in detail: The reconstructed object is the surface of the target to be measured. The processing module is the core of the system and is responsible for the logical control and data processing of the equipment to ensure the coordinated work between the RGB camera and the laser speckle projector. It issues instructions through the control module to adjust the camera's shutter speed, exposure time and the activation timing of the laser speckle projector to achieve the best shooting effect. The power supply module provides a stable power supply for the camera, projector and processing module to ensure the efficient operation of the system under different working conditions.

[0024] Specifically, in terms of system layout, the RGB camera and the infrared laser speckle projector can be fixed in a variety of ways to optimize system performance, including but not limited to: 1. Vertical arrangement. Make the RGB camera and the laser speckle projector form a right-angle relationship. This arrangement ensures that the RGB camera can clearly capture the interaction between the speckle image and the target to be measured, thereby improving the measurement accuracy. In this configuration, the projection direction of the laser speckle is consistent with the field of view of the camera, reducing the measurement error caused by the change of viewing angle. In addition, the vertical arrangement can effectively prevent potential reflection interference and ensure that the image information obtained is more reliable.

[0025] 2. Parallel arrangement. Fix the RGB camera and the laser speckle projector in parallel on the same horizontal plane to optimize the projection angle. This arrangement is suitable for more regular measurement scenes and can effectively reduce the measurement error caused by angle deviation, thereby improving the overall stability and measurement consistency of the system. The parallel arrangement can also avoid the occlusion of the speckle projection area to a certain extent, making the measurement coverage area larger.

[0026] 3. Angled arrangement. Arrange the RGB camera and the laser speckle projector at a certain angle (e.g. 45 degrees). This layout enhances the interaction between the speckle and the target surface and is suitable for targets with complex surface structures. The angled configuration can better adapt to measurement objects of different shapes and materials, increase the coverage of the speckle on the target surface, and improve the accuracy and reliability of depth measurement. The angled arrangement can also improve the incident angle of light, thereby improving the clarity of the image and the ability to capture details.

[0027] In summary, relevant personnel can adopt corresponding arrangements according to the actual application scenario requirements to ensure that the relative position and angle between the RGB camera and the laser speckle projector are always consistent, further improving the stability and measurement accuracy of the system.

[0028] Please refer to the following Figure 2 , Figure 2 : is a flow chart of a dual-modal image capturing method provided in an embodiment of the present application, which specifically includes the following steps: In step S1 , an infrared laser speckle projector is controlled to emit a highly random infrared speckle image to the surface of a target to be measured, so as to form a light spot distribution on the surface of the target to be measured.

[0029] Specifically, the dual-modal image capture method provided in the present application is applied to a dual-modal image capture system based on an RGB camera and an infrared laser speckle projector, wherein the RGB camera is an RGB camera with IR-CUT removed, wherein IR-CUT is a filter, also known as an infrared cut-off filter, which is an optical element specially designed for a camera for filtering out infrared light. In order to obtain a fused image including an infrared speckle image and a visible image, the RGB camera used should remove IR-CUT.

[0030] More specifically, the relevant control module first issues a command to stimulate the infrared laser speckle projector to start working. After being stimulated, the infrared laser speckle projector emits highly random infrared speckle images, which are projected onto the surface of the target to be measured and interact with the texture features of the surface of the target to be measured to form a corresponding spot distribution. This step is the startup stage of the entire system and lays the foundation for subsequent image acquisition.

[0031] In step S2, the RGB camera is driven to capture images of the target surface to be measured that forms the light spot distribution, so as to obtain a fused image.

[0032] Specifically, the control module drives the RGB camera to acquire images, and captures a fused image including complete speckle information and visible image information at the current frame number.

[0033] More specifically, according to specific needs, the RGB camera can take single shots or continuous shots. When using the single-shot mode, the system can obtain the fused image of the current scene at the fastest speed, which is suitable for depth measurement of static scenes. When using the continuous shooting mode, the RGB camera continues to work for a period of time to capture multiple frames of fused images, which is suitable for real-time depth information acquisition of dynamic scenes. In this mode, the control module needs to accurately synchronize the shooting cycle of the RGB camera with the projection cycle of the projector to ensure that each frame of the image can capture complete speckle information.

[0034] In step S3, the fused image is analyzed to obtain the spectral characteristics and color channel information of the fused image, and the infrared speckle image and the visible image are separated according to the spectral characteristics and color channel information of the fused image.

[0035] Specifically, after receiving the fused image taken by the RGB camera, the processing module starts to perform image processing. The image processing applies the infrared speckle image separation algorithm. By analyzing the spectral characteristics and color channel information of the fused image, the infrared speckle and visible light information are separated from the fused image to ensure the independence and integrity of the two images. Since only one shot is required to obtain the above two images, the infrared speckle image and the visible image are completely matched in space and time, thereby avoiding the image alignment problem caused by multiple shots or device asynchrony in the traditional system, and achieving the beneficial effect of overcoming the defect that the existing monocular laser speckle projection system cannot simultaneously obtain visible images and infrared speckle images.

[0036] More specifically, the specific feature of the infrared speckle image separation algorithm is to input a fusion image of a visible image and an infrared speckle image, and output an infrared speckle image. In addition, the output may also include a visible image and a noise image. The algorithm for implementing speckle separation proposed in the embodiment of the present application includes but is not limited to: One of the methods is to use an infrared speckle image separation method based on a generative adversarial network. This method realizes the fusion of the input visible image and the infrared speckle image by designing and training a deep convolutional generative adversarial network, and outputs the infrared speckle image. The following is a detailed description of this method: The step of analyzing the fused image to obtain the spectral characteristics and color channel information of the fused image, and separating the infrared speckle image and the visible image according to the spectral characteristics and color channel information of the fused image comprises: preprocessing the fused image; The preprocessed fused image is input into a generation network to generate an infrared speckle image. The generation network is a set of network architectures in a pre-trained generation adversarial network.

[0037] Optionally, the generation network includes an input layer, a convolutional layer, an encoder-decoder structure and an output layer; The input layer is used to receive the preprocessed fused image with a size of H×W×C, where H is the image height, W is the image width, and C is the number of channels: The convolution layer uses multi-layer convolution operations to gradually extract image features from low-level to high-level, wherein the size of the convolution kernel is 3×3, the stride is 1, the ReLU activation function is used, and each convolution layer is followed by a batch normalization layer; The encoder-decoder structure includes an encoder part and a decoder part, wherein the encoder part is used to gradually reduce the spatial dimension and extract features; The decoder part restores the image size through deconvolution and upsampling operations to restore the spatial resolution of the image, wherein the deconvolution kernel size is 4×4 and the stride is 2; The output layer is a single-channel speckle image, and a 1×1 convolution layer and a Sigmoid activation function are used to map the output to the range of [0, 1] to represent the generated infrared speckle image.

[0038] Specifically, an infrared speckle image separation method based on a generative adversarial network is adopted. By designing and training a deep convolutional generative adversarial network, a fusion image of an input visible image and an infrared speckle image is realized, and an infrared speckle image is output. The deep convolutional generative adversarial network includes a generative network architecture. The main task of the generative network is to generate a pure infrared speckle image from the input fusion image. A multi-layer deep convolutional neural network structure is adopted. The specific architecture of the generative network is described below: About the input layer: The input layer receives the preprocessed fused image with a size of H×W×C, where H and W are the height and width of the image, respectively, and C is the number of channels. A typical input size may be 256×256×3. This is only used as an example in this application to improve the solution, and the scope of protection of this application is not limited by this example.

[0039] About convolutional layers: Multi-layer convolution operations are used to gradually extract low-level and high-level features of the image. The convolution kernel size is usually set to 3×3, the stride is 1, and the ReLU activation function is used to increase the nonlinear representation ability of the network. Each convolutional layer is followed by a batch normalization layer to stabilize the network training process and speed up the convergence.

[0040] Regarding the encoder-decoder structure: a U-Net-style encoder-decoder structure is adopted, in which the encoder part gradually reduces the spatial dimension and extracts features, and the decoder part restores the image size through upsampling operations. It is worth noting that during the decoding process, a jump connection is used to combine the feature map extracted in the encoder with the feature map of the corresponding layer in the decoder, preserving image details and improving separation accuracy.

[0041] More specifically, in the decoder stage, the spatial resolution of the image is gradually restored through deconvolution and upsampling operations, where the deconvolution kernel size can be set to 4×4 with a stride of 2 to ensure that the output image has the same size as the input image.

[0042] About the output layer: The output layer is a single-channel speckle image. A 1×1 convolutional layer and a Sigmoid activation function are used to map the network output to the range of [0, 1] to represent the separated infrared speckle image.

[0043] Furthermore, the network architecture of the generative adversarial network provided in the present application also includes a discriminative network architecture, and the discriminative network architecture is specifically described below: After the step of inputting the preprocessed fused image into the generation network to generate the infrared speckle image, the method comprises: Inputting the generated infrared speckle image into a discriminant network, wherein the discriminant network is another set of network architectures in the pre-trained generative adversarial network; Based on the external input real infrared speckle image, the similarity of the generated infrared speckle image is judged.

[0044] Optionally, the discriminant network includes an input layer, a convolutional layer, a fully connected layer and an output layer; The input layer is used to receive the infrared speckle image generated by the generation network and the real infrared speckle image input externally, and the size is H×W×1; The convolution layer includes multiple layers, each of which is followed by a Leaky ReLU activation function and a batch normalization layer, wherein the convolution kernel size is 3×3 and the stride is 2; The fully connected layer is used to flatten the feature map obtained by the convolution layer into a one-dimensional vector, and output a corresponding scalar value based on the flattened one-dimensional vector; The output layer is used to judge the similarity of the generated infrared speckle images according to the scalar value and output the corresponding judgment result, and the range of the scalar value is [0, 1], wherein the closer the scalar value is to 1, the higher the similarity is.

[0045] Specifically, the discriminant network is used to determine whether the input image is a speckle image generated by the generative network or an actual real image. The discriminant network also uses a convolutional neural network architecture, but focuses more on extracting local features of the image to achieve true and false image discrimination. The specific architecture is as follows: About the input layer: The input layer receives the output of the generation network (i.e., the separated infrared speckle image) and the real infrared speckle image, with a size of H×W×1.

[0046] About convolutional layers: The discriminative network consists of multiple convolutional layers, each of which is followed by a Leaky ReLU activation function and a batch normalization layer. The convolution kernel size is usually 3×3, with a stride of 2. The size of the feature map is gradually reduced by increasing the number of convolutional layers.

[0047] About the fully connected layer: After the convolutional layer, the discriminant network flattens the feature map into a one-dimensional vector through the fully connected layer, and outputs a scalar value through the Sigmoid activation function, which represents the probability of the image being true or false (equivalent to judging the similarity).

[0048] About the output layer: The output layer is a scalar in the range of [0, 1], where the closer the value is to 1, it indicates that it is a real image, and the closer the value is to 0, it indicates that it is a generated image.

[0049] Furthermore, the embodiment of the present application further discloses a training and optimization scheme for the generator network and the discriminator network, which is specifically described below: About loss function: The loss function of the generative network mainly includes adversarial loss and image reconstruction loss. The adversarial loss is used to measure the similarity between the generated image and the real image, and the image reconstruction loss is used to ensure that the generated speckle image matches the real speckle image at the pixel level. The loss function of the discriminative network uses binary cross entropy loss to measure its ability to correctly classify the generated image and the real image.

[0050] Regarding the optimization algorithm: Stochastic Gradient Descent (SGD) or Adam optimizer is used during training to minimize the loss function of the generator and discriminator networks. The learning rate is usually set to 0.0002 and is dynamically adjusted according to the training process.

[0051] Specifically, in each iteration, the generative network is first fixed and the discriminative network is trained to improve its ability to distinguish between true and false images. Then the discriminative network is fixed and the generative network is trained to enable it to generate infrared speckle images that are closer to the real thing. Through cyclic iterations, the generative network gradually generates high-quality speckle images, and the discriminative network also continuously improves its discrimination ability in the process.

[0052] The second method is to use a speckle image separation method based on Fourier transform. The core steps of this method include Fourier transform, frequency domain filtering, inverse Fourier transform and image reconstruction, which are described in detail below: Performing Fourier transform on the fused image to obtain a frequency domain image of the fused image; Acquiring a frequency range corresponding to the speckle image, wherein the frequency range is obtained by analyzing the frequency spectrum of the speckle image; performing high pass filtering on the frequency domain image based on the frequency range; The frequency domain image after high-pass filtering is inversely Fourier transformed to obtain the separated infrared speckle image.

[0053] Specifically, a speckle image separation method based on Fourier transform is used to efficiently extract infrared speckle images from fused images. The core steps include Fourier transform, frequency domain filtering, inverse Fourier transform and image reconstruction, which are described in detail below: The input fusion image is subjected to Fourier transform, which converts the image from the spatial domain to the frequency domain to obtain the frequency components of the image. The Fourier transform result F(u,v) is a complex matrix, that is, it contains information of different frequency components. The formula used for Fourier transform is: ; Then, the frequency range related to the infrared speckle image is identified through spectrum analysis. Generally speaking, the speckle pattern presents a higher frequency feature, while other pattern information such as the background is mainly concentrated in a lower frequency range. Therefore, an embodiment of the present application proposes that a high-pass filter (such as a Butterworth or Gaussian filter) can be used to perform high-pass filtering on the frequency domain representation of the fused image (i.e., the frequency domain image), so as to remove the low-frequency background information and retain the high-frequency speckle feature by selecting a suitable cutoff frequency. The step of performing high-pass filtering on the frequency domain image based on the frequency range includes: The frequency domain image is processed using a high-pass filter, and the transfer function used by the high-pass filter is: ; in, is the distance to the frequency origin, is the cut-off frequency, is the order of the filter.

[0054] The frequency domain image obtained after filtering is subjected to inverse Fourier transform, and the frequency domain result is converted back to the spatial domain to obtain the separated speckle image. The inverse Fourier transform process ensures that only the filtered frequency components are restored to the spatial domain. Among them, the formula used for inverse Fourier transform is: .

[0055] More specifically, in order to further improve the separation effect, threshold processing can be applied after image reconstruction to remove noise and enhance speckle features. Specifically, an adaptive threshold algorithm can be used to perform binarization processing on the reconstructed image, thereby highlighting the speckle part and eliminating background noise.

[0056] The third method is to use a speckle image separation method based on wavelet transform. The core steps of this method include wavelet transform, multi-level decomposition, threshold processing and image reconstruction, which are described in detail below: Performing wavelet transform on the fused image to obtain a wavelet domain image, and decomposing the wavelet domain image to obtain a wavelet coefficient group; The coefficients in the wavelet coefficient group that are lower than a preset threshold are suppressed, and the suppressed wavelet coefficient group is reconstructed into a spatial domain image to separate and obtain an infrared speckle image.

[0057] Specifically, the input fused image is subjected to wavelet transform to realize multi-resolution analysis. By selecting a suitable wavelet basis (such as Haar wavelet, Daubechies wavelet, etc.), the wavelet transform formula is used to convert the fused image into the wavelet domain, and a set of wavelet coefficients, namely the wavelet coefficient group, is obtained. The wavelet coefficient group contains information of different frequencies and spatial positions. Then, through multi-level decomposition, the wavelet coefficient group corresponding to the fused image is decomposed layer by layer into multiple sub-bands, including low-frequency approximate coefficients and high-frequency detail coefficients, where the low-frequency approximate coefficients represent the overall structure of the fused image, and the high-frequency detail coefficients represent the detailed features of the captured infrared speckle image.

[0058] Furthermore, a threshold processing technique can be used to enhance the speckle feature and suppress background noise. Specifically, a suitable threshold is set, and the high-frequency coefficients are processed using a soft threshold or hard threshold method so that the coefficients below the threshold are suppressed, thereby effectively removing the background information.

[0059] Furthermore, the processed coefficients are reconstructed into a spatial domain image through inverse wavelet transform, and the separated speckle image is obtained using the inverse wavelet transform formula. This method significantly improves the image quality and the accuracy of subsequent processing by retaining the detailed features of the speckle image.

[0060] The wavelet transform formula used in the step of performing wavelet transform on the fused image is: ; Specifically, is the wavelet coefficient, is the image data to be analyzed, is the mother wavelet function, is the scale parameter (controls scaling), is the translation parameter (controls position).

[0061] The formula used in the step of suppressing the coefficients in the wavelet coefficient group that are lower than a preset threshold is: ; in, is the threshold value set; The wavelet coefficient group after the reconstruction and suppression is a spatial domain image, and the formula used in the step of separating and obtaining the infrared speckle image is: .

[0062] Specifically, is the wavelet coefficient after processing.

[0063] Furthermore, after the step of separating the infrared speckle image and the visible image, that is, after the infrared speckle image is obtained, the obtained image mainly includes the following application scenarios: 1. Three-dimensional reconstruction: By analyzing the deformation and displacement of the speckle pattern, the depth information of the object surface can be obtained and a high-precision three-dimensional model can be generated.

[0064] 2. Surface morphology detection: using infrared speckle imaging technology, it is possible to detect tiny defects, cracks and unevenness on the surface of materials, and is widely used in industrial quality control and material science research.

[0065] 3. Autonomous driving and robot vision: Infrared speckle images can be used for environmental perception, helping robots identify obstacles and navigation paths, and making intelligent decisions.

[0066] In summary, the dual-modal image capture method provided in the present application realizes the function of synchronously capturing color visible images and infrared speckle images on a single camera through the combined design of an RGB camera without an IR-CUT filter and an infrared laser speckle projector. In addition, due to the use of a monocular laser speckle projection system, the filter is removed, thereby achieving the beneficial effects of reducing hardware costs and simplifying system complexity.

[0067] In addition, since the system can achieve simultaneous dual-modal image acquisition, it can further meet users' needs for high-precision and high-efficiency measurement systems in fields such as smart homes, virtual reality, and autonomous driving.

[0068] Reference Figure 3 , Figure 3This is a virtual structural diagram of a dual-modal image capture device based on an RGB camera and an infrared laser speckle projector provided by the present application, which uses an RGB camera without IR-CUT, including: The light spot distribution generating module 100 is used to control the infrared laser speckle projector to emit a highly random infrared speckle image to the surface of the target to be measured, so as to form a light spot distribution on the surface of the target to be measured; The fused image acquisition module 200 is used to drive the RGB camera to collect images of the target surface to be measured that forms the light spot distribution, so as to obtain a fused image; The speckle image separation module 300 is used to analyze the fused image to obtain the spectral characteristics and color channel information of the fused image, and to separate the spectral characteristics and color channel information of the fused image to obtain an infrared speckle image and a visible image.

[0069] The dual-modal image capture device described in the embodiment of the present application can execute the dual-modal image capture method provided in the above embodiment. The dual-modal image capture device has the corresponding functional steps and beneficial effects of the dual-modal image capture method described in the above embodiment. Please refer to the embodiment of the above-mentioned dual-modal image capture method for details. The embodiment of the present application will not be repeated here.

[0070] The present application also provides an electronic device. Figure 4 , Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include a processor and a memory, wherein the processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various types of chips. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the dual-modal image capture method in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the dual-modal image capture method in the above-mentioned method embodiment is realized.

[0071] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. The one or more modules are stored in the memory, and when executed by the processor, the dual-modal image capture method in the above-mentioned method embodiment is executed. The specific details of the above-mentioned electronic device can be understood by corresponding to the corresponding descriptions and effects in the above-mentioned method embodiment, and will not be repeated here. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0072] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0073] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims, the inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present application.

[0074] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims.

Claims

1. A dual-modal image capturing method, characterized in that: Applied to a dual-modal image capture system based on an RGB camera and an infrared laser speckle projector, wherein the RGB camera is an RGB camera without IR-CUT, the method comprises: Controlling the infrared laser speckle projector to emit a highly random infrared speckle image to the surface of the target to be measured, so as to form a spot distribution on the surface of the target to be measured; The RGB camera is driven to collect images of the target surface to be measured where the light spot distribution is formed, so as to obtain a fused image; The fused image is analyzed to obtain the spectral characteristics and color channel information of the fused image, and the infrared speckle image and the visible image are separated according to the spectral characteristics and color channel information of the fused image.

2. The dual-modal image capturing method according to claim 1, characterized in that: The step of analyzing the fused image to obtain the spectral characteristics and color channel information of the fused image, and separating the infrared speckle image and the visible image according to the spectral characteristics and color channel information of the fused image comprises: preprocessing the fused image; The preprocessed fused image is input into a generation network to generate an infrared speckle image. The generation network is a set of network architectures in a pre-trained generation adversarial network.

3. The dual-modal image capturing method according to claim 2, characterized in that: The generation network includes an input layer, a convolutional layer, an encoder-decoder structure and an output layer; The input layer is used to receive the preprocessed fused image with a size of H×W×C, where H is the image height, W is the image width, and C is the number of channels: The convolution layer uses multi-layer convolution operations to gradually extract image features from low-level to high-level, wherein the size of the convolution kernel is 3×3, the stride is 1, the ReLU activation function is used, and each convolution layer is followed by a batch normalization layer; The encoder-decoder structure includes an encoder part and a decoder part, wherein the encoder part is used to gradually reduce the spatial dimension and extract features; The decoder part restores the image size through deconvolution and upsampling operations to restore the spatial resolution of the image, wherein the deconvolution kernel size is 4×4 and the stride is 2; The output layer is a single-channel speckle image, and a 1×1 convolution layer and a Sigmoid activation function are used to map the output to the range of [0, 1] to represent the generated infrared speckle image.

4. The dual-modal image capturing method according to claim 3, characterized in that: After the step of inputting the preprocessed fused image into the generation network to generate the infrared speckle image, the method comprises: Inputting the generated infrared speckle image into a discriminant network, wherein the discriminant network is another set of network architectures in the pre-trained generative adversarial network; Based on the external input real infrared speckle image, the similarity of the generated infrared speckle image is judged.

5. The dual-modal image capturing method according to claim 4, characterized in that: The discriminant network includes an input layer, a convolutional layer, a fully connected layer and an output layer; The input layer is used to receive the infrared speckle image generated by the generation network and the real infrared speckle image input externally, and the size is H×W×1; The convolution layer includes multiple layers, each of which is followed by a Leaky ReLU activation function and a batch normalization layer, wherein the convolution kernel size is 3×3 and the stride is 2; The fully connected layer is used to flatten the feature map obtained by the convolution layer into a one-dimensional vector, and output a corresponding scalar value based on the flattened one-dimensional vector; The output layer is used to judge the similarity of the generated infrared speckle images according to the scalar value and output the corresponding judgment result, and the range of the scalar value is [0, 1], wherein the closer the scalar value is to 1, the higher the similarity is.

6. The dual-modal image capturing method according to claim 1, characterized in that: The step of analyzing the fused image to obtain the spectral characteristics and color channel information of the fused image, and separating the infrared speckle image and the visible image according to the spectral characteristics and color channel information of the fused image comprises: Performing Fourier transform on the fused image to obtain a frequency domain image of the fused image; Acquiring a frequency range corresponding to the speckle image, wherein the frequency range is obtained by analyzing the frequency spectrum of the speckle image; performing high pass filtering on the frequency domain image based on the frequency range; The frequency domain image after high-pass filtering is inversely Fourier transformed to obtain the separated infrared speckle image.

7. The dual-modal image capturing method according to claim 6, characterized in that: The step of performing high-pass filtering on the frequency domain image based on the frequency range comprises: The frequency domain image is processed using a high-pass filter, and the transfer function used by the high-pass filter is: ； in, is the distance to the frequency origin, is the cut-off frequency, is the order of the filter.

8. The dual-modal image capturing method according to claim 1, characterized in that: The step of analyzing the fused image to obtain the spectral characteristics and color channel information of the fused image, and separating the infrared speckle image and the visible image according to the spectral characteristics and color channel information of the fused image comprises: Performing wavelet transform on the fused image to obtain a wavelet domain image, and decomposing the wavelet domain image to obtain a wavelet coefficient group; The coefficients in the wavelet coefficient group that are lower than a preset threshold are suppressed, and the suppressed wavelet coefficient group is reconstructed into a spatial domain image to separate and obtain an infrared speckle image.

9. The dual-modal image capturing method according to claim 8, characterized in that: The wavelet transform formula used in the step of performing wavelet transform on the fused image is: ； in, is the wavelet coefficient, is the image data to be analyzed, is the mother wavelet function, is the scale parameter, is the translation parameter; The formula used in the step of suppressing the coefficients in the wavelet coefficient group that are lower than a preset threshold is: ； in, is the threshold value set; The wavelet coefficient group after the reconstruction and suppression is a spatial domain image, and the formula used in the step of separating and obtaining the infrared speckle image is: ； in, is the processed wavelet coefficient.

10. A dual-modal image capture device based on an RGB camera and an infrared laser speckle projector, characterized in that: Use an RGB camera without IR-CUT, including: A light spot distribution generating module is used to control the infrared laser speckle projector to emit a highly random infrared speckle image to the surface of the target to be measured, so as to form a light spot distribution on the surface of the target to be measured; The fused image acquisition module is used to drive the RGB camera to collect images of the target surface to be measured that forms the light spot distribution, so as to obtain a fused image; The speckle image separation module is used to analyze the fused image to obtain the spectral characteristics and color channel information of the fused image, and separate the spectral characteristics and color channel information of the fused image to obtain the infrared speckle image and the visible image.

11. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein the memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors so that the electronic device implements the dual-modal image capturing method according to any one of claims 1 to 9.

12. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer readable instructions are executed by a processor to implement the dual-modality image capturing method according to any one of claims 1 to 9.

Images