Data acquisition system for infrared image separation and method using same

Abstract

The invention relates to the technical field of data acquisition, in particular to a data acquisition system for infrared image separation and a method using the same. According to the method, the visual image of the target object is acquired through the first RGB camera added with the first optical filter, the terminal controller controls the infrared speckle projector to send the infrared speckle pattern to the target object, and then the infrared speckle fusion image of the target object after the infrared speckle pattern is projected is acquired through the second RGB camera removed from the first optical filter. A third RGB camera added with a second optical filter is used for collecting an infrared speckle image of the target object after the infrared speckle pattern is projected, so that a visual image, an infrared speckle fusion image and an infrared speckle image are efficiently obtained, and multi-modal image data are provided; therefore, subsequent separation processing of the visual image and the infrared speckle image based on the infrared speckle fusion image is supported.

Description

Technical Field

[0001] The present application relates to the technical field of data acquisition, and in particular to a data acquisition system for infrared image separation and a method using the same. Background Art

[0002] With the rapid progress of artificial intelligence and computer vision technology, infrared imaging technology is widely used in biometrics, security monitoring, industrial inspection and medical imaging. Infrared imaging technology can obtain image information of target objects in low-light or visible light-limited environments by capturing infrared light emitted or reflected by objects. However, since traditional single infrared imaging technology cannot provide enough detailed information in practical applications, infrared speckle imaging uses a speckle projector to project specific infrared spots on the surface of an object. By analyzing the shape and distribution of infrared speckles, the surface morphology and depth information of the object can be obtained. It is widely used in fields such as three-dimensional imaging and high-precision object recognition.

[0003] With the development of computer vision technology based on deep learning and the enhancement of model modeling capabilities, existing technologies can achieve related tasks through feature encoding of data sets and complex model fitting. For the image separation model, its training data set consists of a ternary image group, including visible images, infrared speckle images, and infrared speckle fusion images that fuse visible image information and infrared speckle information. The infrared speckle fusion image is signal separated by a deep learning method to achieve visible image restoration and infrared speckle image generation. However, model training requires a large number of training samples. In this case, the RGB camera with the IR-CUT filter removed can be used to collect training samples, and the photosensitive chip receives RGB signals and infrared signal interference at the same time. For example, when obtaining RGB information in a normal indoor environment, even if there is no infrared signal, the sunlight and lights that penetrate outdoors may also contain infrared light interference information in the same band as the infrared speckle signal. Therefore, it is necessary to construct an environment containing only infrared speckle signals to collect infrared speckle images, and at the same time collect RGB images in a visible light band environment, and obtain infrared speckle fusion images of RGB information and infrared speckle information in a normal environment. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a data acquisition system for infrared image separation and a method using the same, which efficiently acquires multimodal image data that fuses infrared speckle images and visible images, so as to support subsequent separation processing of visible images and infrared speckle images based on infrared speckle fusion images.

[0005] A first aspect of the present application provides a data acquisition system for infrared image separation, the system comprising: A first RGB camera with a first filter added, a second RGB camera with the first filter removed, a third RGB camera with a second filter added, an infrared speckle projector, and a terminal controller, wherein the first RGB camera, the second RGB camera, the third RGB camera, and the infrared speckle projector are all communicatively connected to the terminal controller; The first RGB camera is used to capture a visible image of a target object upon receiving a first capture instruction sent by the terminal controller; The infrared speckle projector is used to project an infrared speckle pattern onto the target object upon receiving an opening instruction sent by the terminal controller; The second RGB camera is further configured to acquire the infrared speckle fusion image of the target object after the infrared speckle pattern is projected when receiving the second acquisition instruction sent by the terminal controller; The third RGB camera is used to collect the infrared speckle image of the target object after the infrared speckle pattern is projected upon receiving the third collection instruction sent by the terminal controller; The terminal controller is configured to send the first acquisition instruction to the first RGB camera so that the first RGB camera acquires the visible image, send an on instruction to the infrared speckle projector so that the infrared speckle projector emits an infrared speckle pattern to the target object, send the second acquisition instruction to the second RGB camera so that the second RGB camera acquires the infrared speckle fusion image, and send the third acquisition instruction to the third RGB camera so that the RGB camera acquires the infrared speckle image.

[0006] In an optional implementation, the terminal controller is further configured to: Controlling the second RGB camera to collect an infrared fused image of the target object before the infrared speckle pattern is projected; The infrared speckle image is determined based on the infrared speckle fusion image and the infrared fusion image.

[0007] In an optional embodiment, the system further includes: a switch, configured to remove the first filter to obtain the second RGB camera through the first RGB camera upon receiving a first movement instruction sent by the terminal controller; The switch is further configured to move the second filter to the second RGB camera when receiving a second movement instruction sent by the terminal controller, so as to obtain the third RGB camera through the second RGB camera.

[0008] A second aspect of the present application provides a data acquisition method for infrared image separation, the method comprising: Controlling a first RGB camera with a first filter to collect a visible image of a target object; Control the first RGB camera to remove the first filter to obtain a second RGB camera; Controlling an infrared speckle emitter to turn on so that the infrared speckle emitter projects an infrared speckle pattern toward the target object; Controlling the second RGB camera to collect an infrared speckle fusion image of the target object after the infrared speckle pattern is projected; Controlling the second RGB camera to add a second filter to obtain a third RGB camera; The third RGB camera is controlled to collect an infrared speckle image of the target object after the infrared speckle pattern is projected.

[0009] In an optional embodiment, the method further comprises: Controlling the second RGB camera to collect an infrared fused image of the target object before the infrared speckle pattern is projected; The infrared speckle image is determined based on the infrared speckle fusion image and the infrared fusion image.

[0010] In an optional implementation, the determining the infrared speckle image based on the infrared speckle fusion image and the infrared fusion image includes: The infrared speckle image is determined by the following formula: ; in, S is the infrared speckle image, VIRS is the infrared speckle fusion image, VIR is the infrared fusion image.

[0011] In an optional embodiment, the method further comprises: Sending a first movement instruction to the switch, so that the switch removes the first filter of the first RGB camera to obtain the second RGB camera; A second movement instruction is sent to the switch, so that the switch moves the second filter to the second RGB camera to obtain the third RGB camera.

[0012] In summary, the data acquisition system for infrared image separation provided by the present application and the method using the same acquire visible images of the target object by means of a first RGB camera with a first filter added thereto, a terminal controller controls the infrared speckle projector to send an infrared speckle pattern to the target object, and then acquires an infrared speckle fused image of the target object after the infrared speckle pattern is projected by means of a second RGB camera with the first filter removed therefrom, and acquires an infrared speckle image of the target object after the infrared speckle pattern is projected by means of a third RGB camera with a second filter added thereto, thereby efficiently acquiring visible images, infrared speckle fused images and infrared speckle images, and providing multimodal image data to support subsequent separation processing of visible images and infrared speckle images based on infrared speckle fused images. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a structural schematic diagram of a data acquisition system for infrared image separation shown in an embodiment of the present application; Figure 2 It is a second structural schematic diagram of a data acquisition system for infrared image separation shown in an embodiment of the present application; Figure 3 is a third structural schematic diagram of a data acquisition system for infrared image separation shown in an embodiment of the present application; Figure 4 is a fourth structural schematic diagram of a data acquisition system for infrared image separation shown in an embodiment of the present application; Figure 5 It is a flow chart of a data acquisition method for infrared image separation shown in an embodiment of the present application; Figure 6 It is a second flow chart of a data acquisition method for infrared image separation shown in an embodiment of the present application; Figure 7 It is a third flow chart of a data acquisition method for infrared image separation shown in an embodiment of the present application; Figure 8 It is a fourth flow chart of a data acquisition method for infrared image separation shown in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0015] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0016] Reference Figure 1 , which is a structural schematic diagram of a data acquisition system for infrared image separation shown in an embodiment of the present application.

[0017] In some embodiments, the data acquisition system 1 includes an RGB camera 10 and two filters, namely a first filter and a second filter. The first filter may be an IR-CUT filter that allows visible light to enter; the second filter may be an infrared filter that allows infrared light of a specific band to pass through, especially a band for infrared speckle. The RGB camera 10 includes a first RGB camera 101 with a first filter added, a second RGB camera 102 with the first filter removed, and a third RGB camera 103 with a second filter added. The first RGB camera 101 is used to collect a visible image V of a target object, the second RGB camera 102 is used to collect an infrared speckle fusion image VIRS of a target object, and the third RGB camera 103 is used to collect an infrared speckle image S of a target object.

[0018] The data acquisition system 1 further includes an infrared speckle projector 20 and a terminal controller 30 . The first RGB camera 101 , the second RGB camera 102 , the third RGB camera 103 and the infrared speckle projector 20 are all in communication connection with the terminal controller 30 .

[0019] In some embodiments, the infrared speckle projector 20 may be a vertical cavity surface emitting laser (VCSEL), which emits specially modulated invisible infrared light to the target object through the VCSEL laser, and then captures the infrared light information reflected back by the target object through the RGB camera 10. The infrared speckle projector 20 is used to emit an infrared speckle pattern to the target object when receiving a start instruction sent by the terminal controller 30.

[0020] In some embodiments, the terminal controller 30 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The terminal controller 30 may also include user equipment, which includes but is not limited to any electronic product that can interact with a user through a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smart phone, digital camera, etc. In other embodiments, the terminal controller 30 may also be built into the RGB camera 10.

[0021] The terminal controller 30 is used to send a first acquisition instruction to the first RGB camera 101, so that the first RGB camera 101 acquires a visible image V of the target object after receiving the first acquisition instruction.

[0022] In some embodiments, reference is made to Figure 3 When it is necessary to collect a visible image V, the first filter is moved in front of the lens of the RGB camera 10 to obtain the first RGB camera 101, ensuring that only visible light is allowed to enter the sensor of the first RGB camera 101. The first RGB camera 101 captures an image of the target scene including the target object to obtain a visible image V. The visible image V contains only visible light information and can accurately provide the real color and brightness of the target scene under natural light.

[0023] The terminal controller 30 is further configured to send a second acquisition instruction to the second RGB camera 102, so that the second RGB camera 102 acquires the infrared speckle fusion image VIRS of the target object after receiving the second acquisition instruction.

[0024] In some embodiments, after obtaining the visible image V, the terminal controller 30 can control the first filter to be removed from the lens of the first RGB camera 101 to obtain the second RGB camera 102. At this time, there is no filter blocking the lens of the second RGB camera 102. The infrared speckle projector 20 is started by the terminal controller 30, and the infrared speckle projector 20 can project a specific infrared speckle pattern to the target object in the target scene. Then, the second RGB camera 102 captures an image of the target scene including the target object to obtain an infrared speckle fusion image VIRS. The infrared speckle fusion image VIRS not only contains mixed information of visible light and infrared light naturally existing in the environment, but also superimposes the infrared speckle pattern generated by the infrared speckle projector 20.

[0025] The terminal controller 30 is further configured to send a third acquisition instruction to the third RGB camera 103, so that the third RGB camera 103 acquires the infrared speckle image S of the target object after receiving the third acquisition instruction.

[0026] In some embodiments, reference is made to Figure 4 After the infrared speckle fusion image VIRS is acquired, the terminal controller 30 can control the second filter to move in front of the lens of the second RGB camera 102 to ensure that only infrared light in the infrared speckle band can be effectively captured, thereby obtaining the third RGB camera 103. The third RGB camera 103 captures an image of the target scene including the target object to obtain an infrared speckle image S. The infrared speckle image S only contains infrared speckle information.

[0027] In some embodiments, the data acquisition system 1 further includes a switch 40, which is installed in front of the lens of the RGB camera 10. The switch 40 is used to remove the first filter when receiving the first movement instruction sent by the terminal controller 30, so as to obtain the second RGB camera 102 through the first RGB camera 101. In addition, the switch 40 is also used to move the second filter to the second RGB camera 102 when receiving the second movement instruction sent by the terminal controller 30, so as to obtain the third RGB camera 103 through the second RGB camera 102. Specifically, the switch 40 controls the translation of the filter through a precise electric drive device. Specifically, the first filter and the second filter are both installed on a linear track, and the motor drive device drives the first filter and the second filter to move smoothly on the linear track to achieve the switching of the filter in front of the lens of the RGB camera 10. When it is necessary to shoot a pure visible image V, the first filter is moved to the front of the lens of the RGB camera 10 to block the infrared light, and the first RGB camera 101 is obtained; when shooting the infrared speckle fusion image VIRS, the first filter is moved aside to obtain the second RGB camera 102, so that the infrared light can enter the camera lens of the second RGB camera 102, and the infrared speckle projector 20 is turned on. When shooting the infrared speckle image S containing infrared information, the second filter is moved to the front of the lens of the second RGB camera 102 to block the infrared light, and the third RGB camera 103 is obtained, so that the infrared light can enter the camera lens of the third RGB camera 103. The switch 40 ensures that the switching speed of the filter is fast and stable, and does not affect the quality of the image. In other embodiments, the switching mode of the filter can be changed to a rotation mode. In the rotation mode, the first filter and the second filter are both installed on a rotating disk, and the rotation of the rotating disk is driven by a small motor. The rotating disk can accurately control the in and out position of the filter. When it is necessary to shoot a visible light image V, the first filter will rotate to the front of the lens. When shooting an infrared speckle fusion image, the first filter will rotate to the side. When shooting an infrared speckle image, the second filter will rotate to the front of the lens.

[0028] It should be noted that the acquisition order of the visible image V, the infrared speckle fusion image VIRS and the infrared speckle image S can be exchanged.

[0029] By using two different filters through the above optional implementation, the filtering of visible light and infrared light can be precisely controlled at the hardware level, reducing the spectral interference introduced by a single filter, which helps to obtain higher quality visible images and infrared speckle images. Secondly, since the dual filters can pre-separate specific spectral information at the hardware level, the subsequent image processing and analysis tasks become relatively simple and efficient. By directly shooting pure infrared speckle images, the complex requirements for the algorithm to separate infrared speckle images can be reduced, thereby improving the stability and reliability of the system.

[0030] Reference Figure 2 As shown, a second structural schematic diagram of a data acquisition system for infrared image separation shown in an embodiment of the present application.

[0031] The data acquisition system 1 includes an RGB camera 10 and a filter. The filter is an IR-CUT filter that allows visible light to enter. The RGB camera 10 includes a first RGB camera 101 with an IR-CUT filter added and a second RGB camera 102 with the IR-CUT filter removed. The first RGB camera 101 is used to acquire a visible image V of a target object, and the second RGB camera 102 is used to acquire an infrared fusion image VIR and an infrared speckle fusion image VIRS of the target object.

[0032] The data acquisition system 1 further includes an infrared speckle projector 20 and a terminal controller 30 . The first RGB camera 101 , the second RGB camera 102 , and the infrared speckle projector 20 are all in communication connection with the terminal controller 30 .

[0033] The terminal controller 30 is used to send a first acquisition instruction to the first RGB camera 101, so that the first RGB camera 101 acquires a visible image V of the target object after receiving the first acquisition instruction.

[0034] In some embodiments, reference is made to Figure 3 When it is necessary to collect a visible image V, the IR-CUT filter is moved in front of the lens of the RGB camera 10 to obtain the first RGB camera 101, ensuring that only visible light is allowed to enter the sensor of the first RGB camera 101. The first RGB camera 101 captures an image of the target scene including the target object to obtain a visible image V. The visible image V contains only visible light information and can accurately provide the real color and brightness of the target scene under natural light.

[0035] The terminal controller 30 is further configured to send a second acquisition instruction to the second RGB camera 102 , so that the second RGB camera 102 acquires the infrared fusion image VIR of the target object after receiving the second acquisition instruction.

[0036] In some embodiments, after obtaining the visible image V, the terminal controller 30 can control the IR-CUT filter to be removed from the lens of the first RGB camera 101 to obtain the second RGB camera 102. At this time, there is no filter blocking the lens of the second RGB camera 102, and the second RGB camera 102 captures the image of the target scene including the target object to obtain the infrared fusion image VIR. The infrared fusion image VIR contains not only visible light information, but also infrared light information naturally existing in the environment.

[0037] The terminal controller 30 is further configured to send a third acquisition instruction to the second RGB camera 102, so that the second RGB camera 102 acquires the infrared speckle fusion image VIRS of the target object after receiving the third acquisition instruction.

[0038] In some embodiments, after acquiring the infrared speckle fusion image VIRS, the terminal controller 30 may start the infrared speckle projector 20, and the infrared speckle projector 20 may project a specific infrared speckle pattern to the target object in the target scene. Then, the second RGB camera 102 captures an image of the target scene including the target object to obtain the infrared speckle fusion image VIRS. The infrared speckle fusion image VIRS not only contains mixed information of visible light and infrared light naturally existing in the environment, but also superimposes the infrared speckle pattern generated by the infrared speckle projector 20.

[0039] In some embodiments, the terminal controller 30 is further configured to determine the infrared speckle image S of the target object based on the infrared speckle fusion image VIRS and the infrared fusion image VIR.

[0040] Specifically, since the signals of the infrared part of visible light and natural light are consistent in the infrared speckle fusion image VIRS and the infrared fusion image VIR, the influence of the visible light and the infrared light in natural light can be eliminated after subtraction, and the remaining part is the infrared speckle image S projected by the infrared speckle projector. That is, the terminal controller 30 can use the pixel-by-pixel subtraction method to directly linearly subtract the infrared speckle fusion image VIRS from the infrared fusion image VIR to obtain the infrared speckle image S.

[0041] The infrared speckle image S is determined by the following formula: .

[0042] In some embodiments, the IR-CUT filter is installed on a linear track, and the motor drive device drives the IR-CUT filter to move smoothly on the linear track to achieve the switching of the filter in front of the lens of the RGB camera 10. When it is necessary to shoot a pure visible image V, the IR-CUT filter is moved in front of the lens to block infrared light; when shooting the infrared fusion image VIR, the IR-CUT filter is moved aside; and when shooting the infrared speckle fusion image VIRS, it is also ensured that the IR-CUT filter is moved aside. And the infrared speckle projector 20 is turned on. It should be noted that the acquisition order of the visible image V, the infrared fusion image VIR and the infrared speckle fusion image VIRS can be interchanged.

[0043] Through the above optional implementation, by using only one IR-CUT filter, the complexity of the mechanical structure and the difficulty of maintenance are reduced. The acquisition of the infrared speckle image S relies on algorithm processing, which reduces the need for hardware switching, thereby reducing the risk of system failure. Secondly, since there is no need for a switch and a double filter, the hardware cost is usually low, so that an effective infrared speckle image separation function can still be provided under a limited budget. In addition, since there is no need to frequently adjust or replace the filter, the operation process is easy to control, which is particularly important for application scenarios that require fast and efficient data acquisition.

[0044] In some embodiments, since different shooting objects may have different surface materials, shapes and optical properties, they all directly affect the effects of infrared speckle pattern projection and image capture. Before using the RGB camera 10 to capture relevant images of the target object, it is necessary to clarify what the target object to be photographed is and its corresponding specific position. After the target object is determined, the acquisition device (including the RGB camera 10, the infrared speckle projector 20 and the switch 40) is placed in a suitable position as required, for example, the acquisition device is placed on a bracket that can be stably supported, and the RGB camera 10 and the infrared speckle projector 20 should be installed side by side to ensure that the optical axes of the two are parallel and aligned with the key area of ​​the target object, and the selection of the installation position should take into account the size and shape of the target object, as well as the working distance of the infrared speckle projector, so as to obtain sufficient detailed information during the shooting process. Among them, the target object can include, but is not limited to: tables, chairs, computers, toys, plants, etc.

[0045] In some embodiments, after the acquisition device is placed, the shooting parameters of the RGB camera 10 need to be adjusted to ensure that the captured image has good brightness, contrast and color reproduction. First, adjust the exposure parameters of the RGB camera 10 to adapt to the lighting conditions of the shooting environment where the target object is located. The exposure time should be short enough to prevent the image from being overexposed, but not too low to avoid loss of image details. Next, adjust the white balance parameters of the RGB camera 10 to ensure accurate restoration of image colors, especially when shooting images containing visible light information. The setting of the white balance parameters should take into account the color temperature of the ambient light source to avoid color deviation. In addition, other parameters of the RGB camera 10, such as ISO sensitivity, focus mode, shutter speed, etc., can also be adjusted according to specific shooting requirements to optimize image quality. For the shooting of infrared speckle images, it may be necessary to appropriately adjust the exposure and sensitivity according to the brightness of the infrared speckle projector 20 and the dark light conditions of the shooting environment to ensure that the infrared speckle is clearly presented in the image.

[0046] Compared with the prior art, in order to train an infrared speckle image separation model that can separate visible images and infrared speckle images based on infrared speckle fusion images, it is necessary to construct a multimodal dataset that can reflect the characteristics of various scenes and target objects, including infrared speckle fusion images, visible images and infrared speckle images. This application proposes a data acquisition method for infrared image separation, which combines the automatic control of filters, the coordinated work of RGB cameras and infrared speckle projectors, and can efficiently acquire visible images V, infrared fusion images VIR, fusion images VIRS and infrared speckle images (S). By automatically switching the shooting mode, this application can realize the separate acquisition of visible light images V and infrared speckle images S without affecting the position of the equipment and the imaging quality, providing high-quality data support for the subsequent training of infrared speckle image separation models.

[0047] In some embodiments, in order to separate the visible image and the speckle image from the infrared speckle fusion image, the terminal controller needs to train an infrared speckle image separation model. Therefore, the terminal controller needs to collect three types of images in advance, namely, the infrared speckle fusion image VIRS, the visible image V and the infrared speckle image S, so that the terminal controller can train and verify the infrared speckle image separation model based on the multimodal data set composed of the infrared speckle fusion image VIRS, the visible image V and the infrared speckle image as the training sample set and the test sample set.

[0048] Reference Figure 5 As shown, it is a flow chart of a data acquisition method for infrared image separation shown in an embodiment of the present application. The data acquisition method for infrared image separation includes the following steps.

[0049] S51, controlling a first RGB camera with a first filter added thereto to collect a visible image of a target object.

[0050] The target object refers to any object, which may include, but not limited to: tables, chairs, computers, toys, plants, etc.

[0051] In some embodiments, reference is made to Figure 6 First, the RGB camera used to collect relevant images of the target object needs to be placed in a suitable position, for example, the RGB camera is placed on a bracket that can be stably supported. The RGB camera may include a first RGB camera with a first filter added, a second RGB camera with the first filter removed, and a third RGB camera with a second filter added, and the first filter is an IR-CUT filter that allows visible light to enter, and the second filter is an infrared filter that allows infrared light of a specific band to pass, especially a band for infrared speckles.

[0052] When the RGB camera is placed, the terminal controller sends a first acquisition instruction for acquiring a visible image to the RGB camera, so that the RGB camera acquires a visible image of the target object. Before acquiring a visible image of the target object through the RGB camera, the terminal controller can send a filter movement instruction to the switch, and drive the first filter to move in front of the lens of the RGB camera through the motor drive device on the switch to obtain a first RGB camera. The first RGB camera can then acquire a visible image V of the target object according to the first acquisition instruction.

[0053] S52: Control the first RGB camera to remove the first filter to obtain a second RGB camera.

[0054] S53, controlling the infrared speckle emitter to turn on, so that the infrared speckle emitter projects an infrared speckle pattern toward the target object.

[0055] S54, controlling the second RGB camera to collect an infrared speckle fused image of the target object after the infrared speckle pattern is projected.

[0056] In some embodiments, it is also necessary to pre-place an infrared speckle projector, wherein the infrared speckle projector should be installed in parallel with the RGB camera to ensure that the optical axes of the two are parallel and aligned with the key areas of the target object, and the selection of its installation position should take into account factors such as the size and shape of the target object, and the working distance of the infrared speckle projector, so as to obtain sufficient detail information during the shooting process.

[0057] Refer to Figure 6, when the visible image V is acquired, the terminal controller can continue to send a second acquisition instruction to the first RGB camera for acquiring an infrared speckle fusion image, so that the first RGB camera acquires an infrared speckle fusion image of the target object. Before the infrared speckle fusion image of the target object needs to be acquired by the first RGB camera, the terminal controller can send a filter movement instruction (also referred to as a first movement instruction) to the switcher, so that the switcher drives the first filter to be removed through the motor drive device, that is, the first filter is moved away from the lens of the first RGB camera to obtain a second RGB camera. At the same time, the terminal controller can also send an opening instruction to the infrared speckle projector, so that the infrared speckle projector emits an infrared speckle pattern to the target object after receiving the opening instruction. At this time, the second RGB camera can acquire the infrared speckle fusion image VIRS of the target object after emitting the infrared speckle pattern based on the second acquisition instruction.

[0058] S55, controlling the second RGB camera to add a second filter to obtain a third RGB camera.

[0059] S56, controlling the third RGB camera to collect an infrared speckle image of the target object after the infrared speckle pattern is projected.

[0060] In some embodiments, reference is made to Figure 6 After the infrared speckle fusion image VIRS is acquired, the terminal controller may continue to send a third acquisition instruction to the second RGB camera, so that the second RGB camera acquires the infrared speckle image of the target object. Before the infrared speckle image of the target object needs to be acquired by the second RGB camera, the terminal controller may send a filter movement instruction (also referred to as a second movement instruction) to the switcher, so that the switcher drives the second filter to move in front of the lens of the second RGB camera through the motor drive device to obtain a third RGB camera, and the third RGB camera can acquire the infrared speckle image S of the target object after the infrared speckle pattern is emitted.

[0061] It should be noted that the acquisition order of the visible image V, the infrared speckle fusion image VIRS and the infrared speckle image S can be exchanged.

[0062] Through the above optional implementation, firstly, a visible image is collected by using the first RGB camera with an IR-CUT filter, then the IR-CUT filter is removed and the infrared speckle emitter is turned on to collect the infrared speckle fusion image, and finally an infrared filter is added to collect the infrared speckle image, which improves the efficiency and accuracy of data collection, ensures the construction of a multimodal dataset, and provides a basis for the subsequent training and verification of the infrared speckle image separation model.

[0063] In an optional embodiment, reference is made to Figure 6Before collecting the visible image, infrared speckle fusion image and infrared speckle image of the target object, it is necessary to first clarify the shooting object (that is, the target object) and its specific location, and determine the installation position of the RGB camera according to the size and shape of the target object to ensure that the RGB camera can capture a clear image. In addition, it is also necessary to determine the working distance of the infrared speckle projector and determine the installation position of the infrared speckle projector based on the working distance. Among them, the installation position of the RGB camera and the infrared speckle projector can be adjusted according to the different shooting angles and specific positions of the target object, allowing shooting in more complex scenes and adapting to different shooting needs and environmental conditions.

[0064] In some embodiments, reference is made to Figure 6 , before collecting the visible image, infrared speckle fusion image and infrared speckle image of the target object, after the RGB camera is installed, the terminal controller needs to fine-tune the camera's shooting parameters to ensure that the captured image has good brightness, contrast and color reproduction. First, adjust the exposure parameters of the RGB camera to adapt to the lighting conditions of the shooting environment where the target object is located. The exposure time should be short enough to prevent the image from being overexposed, but not too low to avoid loss of image details. Next, adjust the white balance parameters of the RGB camera to ensure accurate restoration of image colors, especially when shooting images containing visible light information. The setting of the white balance parameters should take into account the color temperature of the ambient light source to avoid color deviation. In addition, other parameters of the RGB camera, such as ISO sensitivity, focus mode, shutter speed, etc., can also be adjusted according to specific shooting requirements to optimize image quality. For the shooting of infrared speckle images, it may be necessary to appropriately adjust the exposure and sensitivity according to the brightness of the infrared speckle projector and the dark light conditions of the shooting environment to ensure that the infrared speckle is clearly presented in the image.

[0065] Through the above optional implementation mode, by predetermining the shooting object and its position, reasonably setting the installation positions of the RGB camera and the infrared speckle projector, and finely adjusting the shooting parameters of the RGB camera (such as exposure, white balance, ISO sensitivity, focus mode, shutter speed, etc.), it is ensured that the collected visible image, infrared speckle fusion image and infrared speckle image have high definition, good color reproduction and accurate infrared speckle presentation, thereby improving the quality and efficiency of image acquisition and providing a reliable data basis for subsequent infrared image separation and processing.

[0066] Refer to Figure 7 As shown, it is a third flow chart of a data acquisition method for infrared image separation shown in an embodiment of the present application. The data acquisition method for infrared image separation includes the following steps.

[0067] S71, controlling a first RGB camera with a first filter added thereto to collect a visible image of a target object.

[0068] Refer to Figure 8 According to the same implementation as the above step S51, a visible image V of the target object is captured by the first RGB camera.

[0069] S72: Control the first RGB camera to remove the first filter to obtain a second RGB camera.

[0070] S73, controlling the second RGB camera to capture an infrared fused image of the target object before the infrared speckle pattern is projected.

[0071] In some embodiments, reference is made to Figure 8 , when the visible image V is acquired, the terminal controller can continue to send a second acquisition instruction to the first RGB camera to acquire an infrared fusion image, so that the first RGB camera acquires an infrared fusion image of the target object. Before the infrared fusion image of the target object needs to be acquired by the first RGB camera, the terminal controller can send a filter movement instruction (also referred to as a first movement instruction) to the switch, so that the switch drives the first filter to be removed through the motor drive device, that is, the first filter is moved away from the lens of the first RGB camera to obtain a second RGB camera. At this time, the second RGB camera can acquire the infrared fusion image VIR of the target object based on the second acquisition instruction.

[0072] The infrared signal and the visible signal satisfy the linear superposition. After removing the IR-CUT filter and when the infrared speckle projector is not excited, the infrared fusion image VIR of the infrared signal IR and the visible light signal V in the daylight is obtained. The infrared fusion image VIR is determined by the following formula: .

[0073] S74, controlling the infrared speckle emitter to turn on, so that the infrared speckle emitter projects an infrared speckle pattern toward the target object.

[0074] S75, controlling the second RGB camera to collect an infrared speckle fused image of the target object after the infrared speckle pattern is projected.

[0075] In some embodiments, it is also necessary to pre-place an infrared speckle projector, wherein the infrared speckle projector should be installed in parallel with the RGB camera to ensure that the optical axes of the two are parallel and aligned with the key areas of the target object, and the selection of its installation position should take into account factors such as the size and shape of the target object, and the working distance of the infrared speckle projector, so as to obtain sufficient detail information during the shooting process.

[0076] After the infrared fusion image VIR is acquired, the terminal controller may continue to send a third acquisition instruction to the second RGB camera, so that the second RGB camera acquires the infrared speckle fusion image of the target object. Before the infrared speckle fusion image of the target object needs to be acquired by the second RGB camera, the terminal controller may send an opening instruction to the infrared speckle projector, so that the infrared speckle projector emits an infrared speckle pattern to the target object after receiving the opening instruction. At this time, the second RGB camera can acquire the infrared speckle fusion image VIRS of the target object after the infrared speckle pattern is emitted based on the third acquisition instruction.

[0077] When the infrared laser speckle emitter is started, the infrared speckle signal S is captured. The infrared speckle fusion image VIRS is determined by the following formula: .

[0078] S76: Determine the infrared speckle image based on the infrared speckle fusion image and the infrared fusion image.

[0079] In some embodiments, reference is made to Figure 8 When the second RGB camera collects the infrared speckle fusion image VIRS and the infrared fusion image VIR, the second RGB camera can transmit the infrared speckle fusion image VIRS and the infrared fusion image VIR to the terminal controller in an active or passive form, so that when the terminal controller receives the infrared speckle fusion image VIRS and the infrared fusion image VIR, it directly linearly subtracts the infrared speckle fusion image VIRS from the infrared fusion image VIR through a pixel subtraction method to obtain the infrared speckle image S.

[0080] The infrared speckle image S is determined by the following formula: .

[0081] It should be noted that the acquisition order of the visible image V, the infrared fusion image VIR and the infrared speckle fusion image VIRS can be exchanged.

[0082] In some embodiments, based on the pixel-by-pixel subtraction algorithm, the terminal controller can introduce an adaptive adjustment mechanism to automatically adjust the subtraction parameters according to different regions or different lighting conditions in the image, thereby improving the flexibility and accuracy of image separation. Specifically, the terminal controller can divide the image into multiple regions based on the brightness information, extract features such as color, texture, and brightness in each region, assign different weights to each region, and control the impact of factors such as lighting on the image, so as to better preserve image details in the pixel-by-pixel subtraction operation. Apply morphological operations such as dilation and erosion to the calculated image to further remove noise and artifacts in the separated image.

[0083] Through the above optional implementation, the visible image, infrared fusion image and infrared speckle fusion image are effectively collected through the collaborative work of the RGB camera and the infrared speckle emitter, and then the infrared speckle image is determined by pixel subtraction, which not only ensures the diversity and accuracy of the data, but also provides a solid data foundation for the subsequent infrared image separation model training, effectively improving the efficiency and accuracy of image processing.

[0084] In order to facilitate understanding of the inventive concept of the present application, two examples are listed below to describe the practical application of the embodiments of the present application.

[0085] Example 1: In this embodiment, a photographic object with a standard test texture, such as a test board with different materials, is first selected as the target object, and the target object is placed in a laboratory environment. The RGB camera and infrared speckle projector in the acquisition device are mounted on a high-stability bracket, and the bracket can adjust the height and angle to ensure the stability of the device. The distance between the RGB camera and the target object is set to 1.5 meters, which ensures that the shooting details are clear enough while avoiding image distortion caused by being too close. The positions of the RGB camera and the infrared speckle projector are relatively fixed, ensuring that the optical axes of the two are parallel and accurately aligned with the key areas of the target object, ensuring that the infrared speckle pattern projected by the infrared speckle projector can accurately cover the entire shooting area.

[0086] After the acquisition equipment is installed, the parameters of the RGB camera are adjusted in detail. The model of the RGB camera is Canon EOS 5D Mark IV, equipped with a standard lens of 50mm f / 1.8. The exposure time is set to 1 / 125 second to prevent the image from being overexposed. The ISO sensitivity is set to 200 to balance the brightness and noise level of the image. The white balance parameter is adjusted to automatic mode to adapt to the current lighting environment. For the specific needs of shooting infrared speckle images, the shutter speed may need to be slightly adjusted, especially in low-light environments, to ensure that the infrared speckle pattern can be clearly captured.

[0087] First, the target scene is photographed by an RGB camera equipped with an IR-CUT filter. The IR-CUT filter can effectively block infrared light and only allow visible light to pass through. The RGB camera is set to shoot in JPEG format with a resolution of 6000x4000 pixels. The visible image V contains only visible light information, providing the true color and brightness of the target scene under natural light conditions, which is a key reference in the subsequent separation process.

[0088] Then, remove the IR-CUT filter and re-photograph the target scene using the same RGB camera. At this point, the RGB camera is able to capture an infrared fusion image VIR that includes mixed information of visible light and natural infrared light in the environment. The infrared fusion image VIR contains both the details of visible light and the components of infrared light. The resolution of the infrared fusion image is consistent with that of the visible image V to ensure the consistency and accuracy of subsequent processing.

[0089] Next, the infrared speckle projector is started to project a specific infrared speckle pattern in the target scene. The speckle pattern is composed of infrared light generated by the infrared speckle projector, with a wavelength range of 850nm to 950nm. After removing the IR-CUT filter, the target scene is photographed again with the same RGB camera to obtain an infrared speckle fusion image VIRS containing infrared speckles. The shooting parameters remain the same, but due to the different light intensity and speckle density of the infrared speckle projector, the infrared speckle fusion image appears as a mixture of visible light and natural infrared light plus additional infrared speckle features.

[0090] In the image processing stage, the infrared speckle fusion image VIRS containing infrared speckle is first subtracted pixel by pixel from the infrared fusion image VIR containing mixed information of visible light and natural infrared light. This image processing process uses an efficient image subtraction algorithm in the image processing software to ensure that the infrared speckle intensity of each pixel can be accurately calculated. The pixel-by-pixel subtraction process can eliminate the background signal generated by visible light and natural infrared light in the image, thereby extracting the pure infrared speckle image S generated by the infrared speckle projector.

[0091] Example 2: The RGB camera and infrared speckle projector in the acquisition device are installed on a stable bracket to ensure that the equipment will not move or vibrate during the shooting process. The distance between the RGB camera and the target object is set to 1.5 meters. This distance is calculated and measured to ensure that the image details are clear enough while avoiding image distortion caused by too close distance. The position of the RGB camera and the infrared speckle projector remains fixed to ensure that the optical axes of the two are parallel and accurately aligned with the key areas of the object.

[0092] The RGB camera model is Nikon D850, equipped with a standard zoom lens of 24-120mm f / 4. Before data collection, the parameters of the RGB camera were finely adjusted. The exposure time was set to 1 / 125 second to adapt to the lighting conditions on site, and the ISO sensitivity was set to 200 to balance the brightness and noise level of the image. The white balance was adjusted to automatic mode to ensure accurate restoration of image colors. Under this parameter configuration, the shutter speed remained unchanged to adapt to the actual shooting environment.

[0093] After the acquisition equipment is installed and the parameters are adjusted, the application of dual filters begins. First, the first filter, the IR-CUT filter, is moved in front of the RGB camera lens to ensure that only visible light is allowed to enter the camera sensor. Then, the RGB camera is used to capture the image of the target scene. The resulting visible image V contains only visible light information, providing the true color and brightness of the target scene under natural light conditions.

[0094] Next, the IR-CUT filter is controlled to be removed. At this time, there is no filter in front of the RGB camera lens. The infrared speckle projector is started to project a specific infrared speckle pattern in the target scene. The RGB camera is controlled to capture the infrared speckle fusion image VIRS.

[0095] Finally, the second filter, the infrared filter, is controlled to move in front of the RGB camera lens, where the infrared filter is designed with an adjustable spectral range so that it can be switched to allow infrared light in the infrared speckle band to pass through. Then the target scene is photographed again using the RGB camera to obtain the infrared speckle image S.

[0096] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0097] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0098] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A data acquisition system for infrared image separation, characterized in that: The system comprises: A first RGB camera with a first filter added, a second RGB camera with the first filter removed, a third RGB camera with a second filter added, an infrared speckle projector, and a terminal controller, wherein the first RGB camera, the second RGB camera, the third RGB camera, and the infrared speckle projector are all communicatively connected to the terminal controller; The first RGB camera is used to capture a visible image of a target object upon receiving a first capture instruction sent by the terminal controller; The infrared speckle projector is used to project an infrared speckle pattern onto the target object upon receiving an opening instruction sent by the terminal controller; The second RGB camera is further configured to acquire the infrared speckle fusion image of the target object after the infrared speckle pattern is projected when receiving the second acquisition instruction sent by the terminal controller; The third RGB camera is used to collect the infrared speckle image of the target object after the infrared speckle pattern is projected upon receiving the third collection instruction sent by the terminal controller; The terminal controller is configured to send the first acquisition instruction to the first RGB camera so that the first RGB camera acquires the visible image, send an on instruction to the infrared speckle projector so that the infrared speckle projector emits an infrared speckle pattern to the target object, send the second acquisition instruction to the second RGB camera so that the second RGB camera acquires the infrared speckle fusion image, and send the third acquisition instruction to the third RGB camera so that the RGB camera acquires the infrared speckle image.

2. The infrared image separation data acquisition system according to claim 1, characterized in that: The terminal controller is also used for: Controlling the second RGB camera to collect an infrared fused image of the target object before the infrared speckle pattern is projected; The infrared speckle image is determined based on the infrared speckle fusion image and the infrared fusion image.

3. The infrared image separation data acquisition system according to claim 1, characterized in that: The system further comprises: a switch, configured to remove the first filter to obtain the second RGB camera through the first RGB camera upon receiving a first movement instruction sent by the terminal controller; The switch is further configured to move the second filter to the second RGB camera when receiving a second movement instruction sent by the terminal controller, so as to obtain the third RGB camera through the second RGB camera.

4. A data acquisition method for infrared image separation, characterized in that: The data acquisition system for infrared image separation according to any one of claims 1 to 3, wherein the method comprises: Controlling a first RGB camera with a first filter to collect a visible image of a target object; Control the first RGB camera to remove the first filter to obtain a second RGB camera; Controlling an infrared speckle emitter to turn on so that the infrared speckle emitter projects an infrared speckle pattern toward the target object; Controlling the second RGB camera to collect an infrared speckle fusion image of the target object after the infrared speckle pattern is projected; Controlling the second RGB camera to add a second filter to obtain a third RGB camera; The third RGB camera is controlled to collect an infrared speckle image of the target object after the infrared speckle pattern is projected.

5. The data acquisition method for infrared image separation according to claim 4 is characterized in that: The method further comprises: Controlling the second RGB camera to collect an infrared fused image of the target object before the infrared speckle pattern is projected; The infrared speckle image is determined based on the infrared speckle fusion image and the infrared fusion image.

6. The data acquisition method for infrared image separation according to claim 5 is characterized in that: The determining the infrared speckle image based on the infrared speckle fusion image and the infrared fusion image comprises: The infrared speckle image is determined by the following formula: ； in, S is the infrared speckle image, VIRS is the infrared speckle fusion image, VIR is the infrared fusion image.

7. The data acquisition method for infrared image separation according to claim 4 is characterized in that: The method further comprises: Sending a first movement instruction to the switch, so that the switch removes the first filter of the first RGB camera to obtain the second RGB camera; A second movement instruction is sent to the switch, so that the switch moves the second filter to the second RGB camera to obtain the third RGB camera.

Images