A depth information replay method and system based on infrared dot matrix

Abstract

This invention discloses a method and system for replaying depth information based on an infrared dot matrix. The method comprises: obtaining a planar infrared reference image from a structured light depth camera; obtaining an infrared scatter plot projected onto a replay target; extracting infrared projection points on the replay target using a local high-pass filtering algorithm; screening, comparing, and cleaning the projection points; and projecting the target infrared scatter plot using an infrared dot matrix projection device, and having a depth camera acquire the replayed depth information. This invention can effectively replay the target depth and forge non-existent depth information, providing a simulated safety drill platform for intelligent systems that make decisions based on depth information.

Description

Technical Field

[0001] The present invention relates to the field of information security, and in particular to a depth information replay method and system based on infrared dot matrix. Background Art

[0002] With the rapid development of information technology and artificial intelligence, computer vision has become one of the primary methods for receiving physical signals in intelligent systems. Traditional RGB images are easily affected by factors such as lighting and posture. Therefore, image processing and recognition systems based on depth information have gradually become important research and application directions in computer vision. Structured light depth cameras, due to their advantages such as low cost, low power consumption, and high image quality, have been widely used in smartphones, smart door locks, autonomous driving, and other fields.

[0003] Many attack and defense methods exist for RGB image recognition systems, effectively improving their security. However, no attack and defense methods exist for depth information, which is often used for privacy protection and security authentication. Therefore, how to conduct security testing on image recognition systems based on depth information is an urgent problem that needs to be solved. Summary of the Invention

[0004] The purpose of the present invention is to provide a depth information replay method and system based on infrared dot matrix, which can replay the depth information of the target and forge non-existent depth information, and can be used as a security testing method for image recognition systems for depth information.

[0005] To achieve the above object, the present invention provides a depth information replay method based on infrared dot matrix, comprising:

[0006] S1. Obtain an infrared scattergram projected onto a plane by a scattered structured light depth camera as a reference scattergram;

[0007] S2. Obtain an infrared scatter image projected onto the playback target by a scattered structured light depth camera;

[0008] S3. Extract the infrared projection points on the replay target through a local high-pass filtering algorithm to obtain clearer infrared projection scattered points;

[0009] S4, screening, comparing, and cleaning the extracted infrared scattered points and the reference scattered points, and deleting interference points and noise;

[0010] S5. Projecting the infrared dot matrix of the playback target using an infrared dot matrix projection device so that the depth camera obtains the playback depth information;

[0011] S6. Use the depth camera to complete the replay of the depth information.

[0012] A depth information replay system based on infrared dot matrix, comprising:

[0013] An image acquisition module is used to acquire a reference infrared scatter image and a target infrared scatter image;

[0014] Image processing module, used to filter and clean the target infrared scatter image to obtain a clear and effective scatter image;

[0015] Dot projection module, used to project the replayed scatter plot;

[0016] The depth generation module is used to receive the infrared scatter map and generate the corresponding depth.

[0017] The beneficial effects of the present invention are:

[0018] 1. The present invention performs local high-pass filtering on the infrared scattergram of the replay target collected, and screens, compares and cleans each projection point, which can effectively remove background and noise. Compared with the infrared scattergram of the replay target obtained by directly replaying, the degree of restoration of the replay depth can be significantly improved.

[0019] 2. The present invention establishes a complete depth replay system, including four processes: image acquisition, image processing, projection replay, and depth generation. It can effectively replay the target depth when the target does not exist. It can be used as a security test for image recognition systems based on depth information to improve the detection robustness of the image recognition system.

[0020] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is an architecture diagram of a depth information playback system based on infrared dot matrix according to an embodiment of the present invention;

[0022] Figure 2 This is a flow chart of a depth information replay method based on infrared dot matrix according to an embodiment of the present invention;

[0023] Figure 3 Schematic diagram of a method for obtaining a reference scatter plot according to an embodiment of the present invention;

[0024] Figure 4 Schematic diagram of a method for obtaining a target scatter plot according to an embodiment of the present invention;

[0025] Figure 5 Schematic diagram of a projection scatter plot replay method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0027] Example

[0028] As shown in the figures, the following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for example only, and those skilled in the art will appreciate other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0029] The accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0030] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0031] The present invention provides a depth information replay method and system based on infrared dot matrix. Figure 1 The system architecture diagram of the embodiment is shown in FIG. Figure 1 As shown, the system architecture may include an image acquisition module, an image processing module, a dot projection module, and a depth generation module. The image acquisition module can be a device capable of receiving and imaging infrared light, including but not limited to wavelengths between 780nm and 2526nm, such as a mobile phone, camera, or night vision device. The image acquisition module and the dot projection module can communicate with the image processing module via wired or wireless means. The image acquisition module can transmit the captured infrared image to the image processing module, and the processed infrared image can also be transmitted to the dot projection module for projection. The image processing module can be another terminal connected to the image acquisition module and the dot projection module, or a backend server that provides image analysis and processing. The dot projection module can be a device capable of projecting infrared light, such as a laser array, a projector, or an infrared light source equipped with a diffractive optical element. The depth imaging module generates depth information by receiving an infrared scattered dot pattern. It can be a device capable of structured light depth generation, such as a structured light depth camera or an infrared camera equipped with a depth generation algorithm.

[0032] In one embodiment, the depth generation module is comprised of a structured light-based depth camera, which itself is equipped with an infrared camera capable of acquiring an infrared scatter pattern. The received infrared scatter pattern is then compared with a reference pattern stored in internal memory to generate a corresponding depth map. The image acquisition module described herein can be comprised of the infrared camera of the depth camera or an external infrared light receiving device.

[0033] The following describes the depth information replay method based on an infrared dot matrix of this embodiment. Application scenarios of this method include, but are not limited to: a user first collects an infrared scattergram containing target depth information, uses the image processing method proposed in this embodiment to obtain a clear and effective projected scattergram, then removes the target and projects the scattergram using a scatter projection device, causing the depth generation and detection system to generate non-existent depth information, completing a safety test of the depth generation and detection system.

[0034] Figure 2 An exemplary process of a depth information replay method is shown, including:

[0035] Step 1: The image acquisition module obtains the infrared scattergram projected onto the plane by the scattered structured light depth camera as a reference scattergram and transmits it to the image processing module;

[0036] Step 2: The image acquisition module obtains the infrared scatter image projected by the scattered structured light depth camera onto the playback target and transmits it to the image processing module;

[0037] Step 3: The image processing module extracts the infrared projection points on the replay target through a local high-pass filtering algorithm to obtain clearer infrared projection scattered points;

[0038] Step 4: The image processing module screens, compares, and cleans the extracted infrared scattered points and the reference scattered points, deletes interference points and noise, and transmits them to the dot matrix projection module;

[0039] Step 5: The dot matrix projection module uses an infrared dot matrix projection device to project the infrared dot matrix of the replay target, so that the depth generation module obtains the replayed depth information and completes the replay of the depth information.

[0040] Based on the above method, the depth information of the target is collected and replayed when the target is not within the acquisition range of the depth generation and detection system, which can effectively test the security of the depth generation and detection system.

[0041] Below Figure 2 Each step is described in detail.

[0042] 1. Get a plane reference map. Figure 3As shown in the figure, a reflector is placed 1 meter in front of the scattered structured light depth camera to cover the shooting angle of the scattered structured light camera. At the same time, an infrared camera is placed above the infrared lens of the scattered structured light depth camera. The scattered structured light depth camera is turned on. After observing the infrared speckle image on the display of the infrared camera, an infrared scattergram is obtained as a reference scattergram for subsequent comparison and cleaning of the replayed target infrared scattergram.

[0043] 2. Get the target scatter plot. Figure 4 As shown, a replay target object is placed in front of a scattered structured light depth camera, and an infrared camera is placed above the infrared lens of the scattered structured light depth camera. The scattered structured light depth camera is turned on, and after observing the infrared speckle image on the display of the infrared camera, an infrared scatter map is obtained as the infrared scatter map of the replay target.

[0044] 3. Infrared scatter point extraction. The infrared scatter point map of the replayed target obtained above contains a lot of noise, and directly using it to replay the depth map will result in a lot of incompleteness. This noise mainly includes: infrared light reflected by the target object itself; infrared light reflected by the target object's surrounding environment; noise generated by the infrared camera shooting in low brightness; and noise caused by the infrared camera's resolution limitations.

[0045] Taking into account the characteristic that the scattered points projected by scattered structured light must be the brightest points in a certain neighborhood to be captured and recognized as depth information by the depth camera, this paper proposes a local high-pass filtering algorithm to extract the infrared scattered points of the playback target. The specific method is as follows:

[0046] 1) Improve the contrast of image details. Each infrared spot in the target image occupies a very small pixel area, typically 1 to 3 pixels, so the contrast of the image details needs to be improved. This example uses a histogram equalization algorithm to make the grayscale distribution of the entire image approximately uniform, giving the entire image a larger grayscale dynamic range and higher contrast, making the image more detailed and better able to extract infrared spots. The histogram equalization algorithm, whose grayscale range is between [0, 255], is as follows:

[0047]

[0048] In the above formula, HW is the total number of pixels in the image, is the number of pixels with gray value k, s k is the grayscale value of the corresponding pixel after histogram equalization.

[0049] 2) Local high-pass filtering. Each infrared scattered point in the target image is sparsely distributed on the image. That is, within a small area, only one brightest point appears as the scattered point to be extracted. The present invention proposes a filtering algorithm based on local high-pass filtering. The main algorithm is as follows:

[0050] The local high-pass filter performs a single computation using a small neighborhood. Each computational process consists of four stages: convolution, sorting, threshold calculation, and filtering. This example uses a 3×3 neighborhood as the computational area. You can also use a different neighborhood size based on the pixel range occupied by the scattered points.

[0051] In the convolution stage of this example, 9 convolution kernels are used to perform convolution operations on the neighborhood range, and a 1×9 vector can be obtained. The convolution kernel obtains the weighted sum of the brightness values ​​of each pixel in the area and its surrounding points. The representation of the convolution kernel should include but is not limited to the following form provided in this example:

[0052]

[0053]

[0054]

[0055] The generated vectors will be sorted in the sorting stage. If the subscript corresponding to the maximum grayscale value is 4, that is, the maximum grayscale value is located at the center point within the range, then the point is recorded as an infrared scatter point and its grayscale value is retained.

[0056] During the threshold calculation phase, considering the low resolution of infrared cameras, the pixel area occupied by each infrared scattered point is expanded to 1 to 3 pixels. This example uses a threshold control method to control the grayscale values ​​of pixels within the area except the center point. That is, after determining that the center point of a certain area is an infrared scattered point, instead of setting all remaining grayscale values ​​within the neighborhood to 0, a threshold is determined using the following function:

[0057] y=-0.001x 2 +0.01x

[0058] Among them, x is the gray value of the center point within the range of the area, and y is the threshold.

[0059] After the threshold is determined, the filtering stage sets the grayscale values ​​of pixels within the area that are less than the threshold to 0, while retaining the grayscale values ​​of pixels that are greater than the threshold.

[0060] This example uses a sliding window method to perform local high-pass filtering on the input image. That is, a 3×3 local high-pass filter is used to filter from left to right and from top to bottom. This can effectively obtain a clearer infrared scattered projection image of the playback target.

[0061] 4. Infrared scatter point screening, comparison, and cleaning. As a further improvement to the above-mentioned local high-pass filtering algorithm for extracting infrared scatter points, this example uses a reference scatter point map to further correct the re-entered target infrared scatter point map, taking into account the interference of the infrared camera and background infrared reflected light on local extreme values. This is mainly divided into two parts: image alignment and scatter point matching and cleaning.

[0062] The image alignment part refers to matching the target object's area with the reference scatter plot, which is the basis for the next step of scatter matching. The speckle pattern of the scattered structured light projector is determined by the diffractive optical element of the projector. The speckle pattern projected by the same scattered structured light projector is constant. The position of the replayed target in its infrared scatter plot can be directly mapped to the reference scatter plot. At the same time, the replayed target can be marked in its infrared scatter plot by a rectangular box, that is, it can be marked with {x center ,y center The quadruple {\mathcal {\mathcal { _{ ... { _{\math { _{\math { _{

[0063] The scatter point matching and cleaning part refers to matching the target reference scatter points with the corresponding scatter points in the target actual scatter point map, and using them as matching points. Non-matching points are treated as noise and the grayscale value of the pixel point is directly set to 0. The principle is that although the scattered point structured light depth imaging also uses the offset of the scattered points to calculate the depth, it only performs horizontal offset and not vertical offset. At the same time, the pixel value of the movement is also carried out within a smaller range and does not cross-overlap with the other scattered points. Therefore, the present invention matches the scattered points in a row scanning matching manner, and the specific method is as follows:

[0064] In units of rows, record the set of scattered horizontal coordinates of the pixels in row i of the reference image whose grayscale values ​​are non-zero:

[0065]

[0066] Record the set of scatter points with non-zero grayscale values ​​in the pixel values ​​of the i-th row of the target scatter plot:

[0067]

[0068] Each row is calculated as follows: First calculate and T iThe distance between all points in the target scatter plot is calculated, and the nearest point (if it is the kth point) is selected as the matching point. If the grayscale value of the two points adjacent to the kth point is less than 50 compared with the kth point, it is also retained as a matching point. At the same time, all points before the kth point in the i-th row of the target scatter plot are deleted. That is, the second calculation is in and The grayscale values ​​of all matching points in the row are retained, while the grayscale values ​​of all non-matching points are directly set to 0.

[0069] 5. Infrared scatter replay: Using the above image processing algorithm, a target scatter diagram that can be used for replay can be obtained.

[0070] 6. Depth camera generates depth. Figure 5 As shown in the figure, this example uses an infrared dot matrix projector and a projection reflector to project the scatter diagram of the playback target, so that the depth camera can obtain the replayed depth information and complete the replay of the depth information. Specifically, first turn off the scatter projector of the structured light depth camera so that it cannot project the structured light speckle pattern normally. Then place the infrared dot matrix projector under the infrared lens of the structured light depth camera. The scatter diagram of the playback target is transmitted from the data processing module to the infrared dot matrix projector through the communication interface. A projection reflector is placed 1 meter in front of the infrared dot matrix projector. The scatter diagram of the playback target is reflected by the projection reflector to the structured light depth camera. When the infrared lens of the depth camera receives the scatter diagram, it combines it with the built-in reference speckle pattern to perform depth calculation, that is, the depth information is completely replayed.

[0071] It should be noted that although the image acquisition and processing and depth generation modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0072] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented as the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system". Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the claims.

[0073] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings and that various modifications and variations can be made without departing from the scope thereof, which is limited only by the appended claims.

Claims

1. A depth information replay method based on infrared dot matrix, characterized in that: include: S1. Obtain an infrared scattergram projected onto a plane by a scattered structured light depth camera as a reference scattergram; S2. Obtain an infrared scatter image projected onto the playback target by a scattered structured light depth camera; S3. Extract the infrared projection points on the replayed target through a local high-pass filtering algorithm to obtain relatively clear infrared projection scattered points. The local high-pass filtering algorithm is a local high-pass filter that performs a calculation process with a small neighborhood range as a calculation unit. Each calculation process is divided into four stages: convolution, sorting, threshold calculation, and filtering. S4, screening, comparing, and cleaning the extracted infrared scattered points and the reference scattered points, and deleting interference points and noise; The reference scatter plot is used to further correct the replayed target infrared scatter plot, which is mainly divided into two parts: image alignment and scatter plot matching and cleaning; S5. Projecting the infrared dot matrix of the replay target using an infrared dot matrix projector so that the depth camera obtains the replay depth information. Specifically, projecting the scatter diagram of the replay target using an infrared dot matrix projector and a projection reflector. The steps for projecting the scatter plot of the replayed target are as follows: turning off the scatter projector of the structured light depth camera so that it cannot project the structured light speckle pattern normally, placing an infrared dot matrix projector under the infrared lens of the structured light depth camera, transmitting the scatter plot of the replayed target from the data processing module to the infrared dot matrix projector through the communication interface, placing a projection reflector 1 meter in front of the infrared dot matrix projector, and reflecting the scatter plot of the replayed target to the structured light depth camera through the projection reflector; S6. Use the depth camera to complete the replay of the depth information. Specifically, after the infrared lens of the depth camera receives the scatter pattern, it performs depth calculation in combination with the built-in reference speckle pattern, so that the depth information is completely replayed.

2. A depth information replay system based on an infrared dot matrix, using the depth information replay method based on an infrared dot matrix as claimed in claim 1, characterized in that: include: An image acquisition module is used to acquire a reference infrared scatter image and a target infrared scatter image; Image processing module, used to filter and clean the target infrared scatter image to obtain a clear and effective scatter image; Dot projection module, used to project the replayed scatter plot; The depth generation module is used to receive the infrared scatter map and generate the corresponding depth.

Images