A non-line-of-sight object recognition method, system and device

Abstract

The present invention relates to the field of computer vision technology, and specifically to a method, system and device for non-line-of-sight object recognition, the method comprising: obtaining a local speckle image of a target object, the local speckle image being obtained by irradiating the target object with coherent light or partially coherent light and reflecting it to a scattering surface and then reflecting it to a camera for shooting, and obstacles existing on one or two of the paths of the reflection; inputting the local speckle image into a trained object information prediction neural network to obtain information of the target object and complete object recognition. The present invention can realize the recognition of non-line-of-sight target objects through local speckles when the field of view is limited due to factors such as partial occlusion in the optical path formed by the coherent light source, the target object and the camera, the limitation of the camera position angle, the object not fully entering the preset position, the coherent light source being disturbed, etc. Compared with the prior art, the present invention is suitable for engineering practice, has a wider range of application scenarios and adaptability to practical applications.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and more specifically, relates to a non-line-of-sight object recognition method, system and device. Background Art

[0002] In the field of computer vision technology, non-line-of-sight (NLOS) object recognition technology has made great progress in recent years. The typical solution is the NLOS object imaging method based on the photon time-of-flight (Time-of-Flight), such as Figure 1 As shown in the figure, this method uses ultrafast lasers and single-photon detectors. By measuring the flight time of photons to the surface of hidden objects and then returning, multiplying it by the speed of light, the flight distance of the photons can be obtained. Then, a set of mirrors is used to change the direction of the laser and scan the object, and finally the shape of the hidden object can be scanned out. The disadvantage of this method is that it requires expensive experimental instruments (ultrafast lasers and single-photon detectors), which is very costly. Another disadvantage is that the object needs to be mechanically scanned during the recognition process, so the technical solution takes a long time to identify non-line-of-sight objects.

[0003] At present, some researchers have proposed a new non-line-of-sight object recognition technology, which can realize the recognition of non-line-of-sight objects only by relying on coherent light sources and cameras. The cost is relatively low and the recognition process is efficient and fast. However, its implementation process is in a very ideal laboratory environment. In actual application, it may encounter interference from some adverse external factors, such as partial obstruction in the light path formed by the coherent light source, target object and camera, limitation of the camera position angle, and the object not fully entering the preset position, etc., resulting in a limited field of view, and the camera can only capture a local speckle image; for example, the actual lighting will be interfered by natural light, and the coherent light source will actually partially decoherently transform into partially coherent light illumination. This makes it questionable whether the above technology can accurately identify non-line-of-sight objects in actual scenes. Summary of the invention

[0004] The present invention provides a non-line-of-sight object recognition method, system and device, which are used to solve the technical problem that the existing non-line-of-sight object recognition method cannot be applied to engineering practice.

[0005] The technical solution of the present invention to solve the above technical problem is as follows: A non-line-of-sight object recognition method comprises:

[0006] S1, controlling coherent light or partially coherent light to illuminate a target object and reflect the light to a scattering surface and then to a camera, so that the camera captures an image; obtaining a local speckle image of the target object from the image captured by the camera, wherein the local speckle image is an image of a portion of the overall speckle of the target object;

[0007] S2. Input the local speckle image into a trained object information prediction neural network to obtain information of the target object and complete object recognition.

[0008] The beneficial effects of the present invention are as follows: the present invention collects a local speckle image of a target object obtained by irradiating the target object with coherent light or partially coherent light, and inputs the local speckle image into a pre-trained prediction neural network for object recognition. In the case where the field of view is limited due to factors such as partial occlusion in the optical path formed by the coherent light source, the target object and the camera, the limitation of the camera position angle, and the object not fully entering the preset position, the present method can realize the recognition of non-line-of-sight target objects through local speckle. At the same time, in the case where the coherent light source is disturbed and the good coherent light decoheres into partially coherent light, the recognition of non-line-of-sight target objects can also be realized to a certain extent. Therefore, compared with the prior art, it is suitable for engineering practice, has a wider range of application scenarios and adaptability to practical applications.

[0009] Based on the above technical solution, the present invention can also be improved as follows.

[0010] Furthermore, before S1, the method further includes:

[0011] Constructing a recognition accuracy curve of the object information prediction neural network at different local speckle sizes, and determining a critical point of the local speckle size at an actually required recognition accuracy level based on a recognition accuracy variation trend on the recognition accuracy curve;

[0012] Then, when S1 is executed, a local speckle image of the target object whose size is larger than the local speckle of the critical point is collected.

[0013] Furthermore, the field angle corresponding to the local speckle image is greater than or equal to 1°.

[0014] A further beneficial effect of the present invention is that when the field angle corresponding to the local speckle image is greater than or equal to 1°, a higher prediction accuracy can be guaranteed.

[0015] Furthermore, the local speckle image is a grayscale image.

[0016] A further beneficial effect of the present invention is that when the local speckle image is a grayscale image, the training efficiency and prediction efficiency of the neural network can be improved.

[0017] The present invention also provides a non-line-of-sight object recognition system, comprising a coherent light source, a camera and a processor;

[0018] The coherent light source is used to illuminate the target object; the camera is used to photograph the speckles formed when the coherent light source illuminates the target object and is reflected on the scattering surface; and the processor is used to execute the non-line-of-sight object recognition method as described above.

[0019] The beneficial effect of the present invention is that the recognition system constructed by the present invention using the above-mentioned non-line-of-sight object recognition method is suitable for engineering practice, has a wider range of application scenarios and adaptability to practical applications.

[0020] The present invention also provides a non-line-of-sight object recognition device, comprising a coherent light source, a camera, a processor, and a mechanism for partially blocking the camera lens;

[0021] The coherent light source is used to illuminate the target object; the camera is used to photograph the speckles formed by the coherent light source irradiating the target object and reflecting to the scattering surface; the processor is used to execute the non-line-of-sight object recognition method as described above.

[0022] The beneficial effect of the present invention is as follows: the present invention adopts the above-mentioned non-line-of-sight object recognition method to construct a non-line-of-sight object recognition device, in which a mechanism for partially blocking a camera lens is provided. When performing special tasks, the setting of the mechanism can be used to hide the presence of the camera.

[0023] The present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute a non-line-of-sight object recognition method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the prior art of non-line-of-sight object recognition method based on photon flight time;

[0025] Figure 2 A flowchart of a non-line-of-sight object recognition method provided by an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of a prior art method for performing non-line-of-sight object recognition using speckles formed when a target object is illuminated by a coherent light source and then reflected to a scattering surface;

[0027] Figure 4 A schematic diagram of a system for identifying obstacles on the path of light reflected to a scattering surface provided by an embodiment of the present invention;

[0028] Figure 5A schematic diagram of a system for identifying obstacles on the path reflected from a scattering surface to a camera provided by an embodiment of the present invention;

[0029] Figure 6 A schematic diagram of a non-line-of-sight object recognition method provided by an embodiment of the present invention in which coherent light illuminates a target object and is reflected to a scattering surface to form speckles;

[0030] Figure 7 A schematic diagram of different sizes of MNIST data images and corresponding speckle images of a non-line-of-sight object recognition method provided by an embodiment of the present invention;

[0031] Figure 8 A schematic diagram of an image capture scheme of a non-line-of-sight object recognition method provided by an embodiment of the present invention;

[0032] Fig. 9 A schematic diagram of a neural network structure of a non-line-of-sight object recognition method provided by an embodiment of the present invention;

[0033] Fig.10 A relationship diagram between recognition accuracy and image size of a non-line-of-sight object recognition method provided by an embodiment of the present invention;

[0034] Fig.11 A schematic diagram of different overlapping rate interception schemes for a non-line-of-sight object recognition method provided by an embodiment of the present invention;

[0035] Fig.12 A relationship diagram of recognition accuracy under different overlap rates for a non-line-of-sight object recognition method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0037] Embodiment 1

[0038] A non-line-of-sight object recognition method 100, such as Figure 2 As shown, including:

[0039] S110, controlling coherent light or partially coherent light to illuminate a target object and reflect the light to a scattering surface and then to a camera, so that the camera captures an image; obtaining a local speckle image of the target object from the image captured by the camera, where the local speckle image is an image of a portion of the overall speckle of the target object;

[0040] S120 , inputting the local speckle image into the trained object information prediction neural network to obtain the information of the target object and complete the object recognition.

[0041] It should be noted that the training method of the object information prediction neural network can be: taking multiple local speckle images of the target object, the multiple local speckle images constitute a data set; inputting the data set into the neural network, and the neural network learns the mapping function between the local speckle image and the target object information. In addition, the method uses the local speckle image to identify the object, wherein the reason for the formation of the local speckle image may be the existence of obstacles on the path of one or two reflections in the above-mentioned optical path, but is not limited to the existence of obstacles in the optical path, and at least may also include: the complete speckle image taken is partially missing due to various reasons, thereby obtaining a local speckle image.

[0042] In addition, after the coherent light or partially coherent light illuminates the target object, one path is for the light to be reflected to the scattering surface, and the other path is for the light to be reflected to the camera by the scattering surface, for example Figure 3 In the identification system 200 shown in the figure, a coherent light source illuminates a target object (the target used in this embodiment is an LCD screen that can display images); a camera captures the speckle formed by the coherent light source irradiating the target object and reflecting to a scattering surface (i.e., a wall). However, in actual scenes, when collecting speckle images, it is inevitable that obstacles will affect the acquisition of the speckle image, which makes it difficult for the camera to collect the entire speckle image of the target object. The existence of obstacles can be roughly divided into three situations: when there are obstacles on the path of light reflected to the scattering surface, such as Figure 4 As shown in Figure 5 As shown in the figure (there is a baffle with a pinhole blocking the shooting path of the camera when it is shooting the speckle image), or there are obstacles on both paths. In these three cases, the speckle images shot by the camera are regarded as local speckle images of the target object.

[0043] in, Figure 3 The coherent light source in the image is a HeNe laser (632.8mm, Thorlabs HNL150R) with a beam expander. The model of the camera 420 is the Thorlabs CS2100M CMOS. The target object is an LCD screen that can display handwritten numbers, that is, the LCD screen will play the MNIST data set. Due to the obstruction of the wall, the camera cannot directly shoot the LCD, but the LCD screen is illuminated by a coherent light source. The LCD screen reflects the light to the wall in front of the LCD screen. The wall is a scattering surface, and the light forms speckles on the wall. The camera can directly capture the speckles on the wall and obtain a speckle image. Figure 6As shown, a coherent light source illuminates the LCD display surface that displays the MNIST data set, and the LCD reflects light to the imaging surface. When the LCD displays the numbers "2", "4", and "6", corresponding speckles appear on the imaging surface.

[0044] In the case of coherent light illumination, the light intensity at a point r on the image plane is given by the following formula:

[0045]

[0046] Among them, i is the imaginary unit; the exp() function is the e exponential function; Re[] means taking only the real part of a complex number; the physical quantity r represents the position vector of a point on the image plane, and R represents the position vector of a point on the object. represents the light intensity at the image plane r; Represents the light field amplitude of the light emitted from a certain point R on the object at the image plane r, Represents the phase function of a point R on the object in the image plane r, represents the integration variable, and the integration region is the entire object.

[0047] The distribution of the light intensity (grayscale value) of the speckle pattern is caused by the interference of coherent light. From the above formula, it can be seen that the integral area corresponding to the integral of the light intensity of each point in the speckle pattern covers the entire object to be identified, so the local part of the speckle pattern can reflect the overall information of the object to be identified to a certain extent.

[0048] The method of this embodiment can realize the recognition of non-line-of-sight target objects through local speckle when the field of view is limited due to partial occlusion in the optical path formed by the coherent light source, the target object and the camera, the limitation of the camera position angle, and the object not fully entering the preset position. At the same time, when the coherent light source is disturbed and the good coherent light decoheres to partially coherent light, the recognition of non-line-of-sight target objects can also be realized to a certain extent. Therefore, compared with the existing technology, it is suitable for engineering practice and has a wider range of application scenarios and adaptability to practical applications.

[0049] Preferably, before S110, the method 100 further includes:

[0050] Constructing a recognition accuracy curve of the object information prediction neural network at different local speckle sizes, and determining a critical point of the local speckle size at an actually required recognition accuracy level based on a recognition accuracy change trend on the recognition accuracy curve; when executing S110, a local speckle image of the local speckle of the target object whose size is larger than the critical point is collected.

[0051] It should be noted that the recognition accuracy level can be the accuracy level obtained by dividing the accuracy interval. The recognition accuracy variation trend that the recognition accuracy varies with the size of the local speckle can be determined in advance. In actual measurement, the size range of the local speckle can be determined according to the actual recognition accuracy requirements, and the image of the local speckle in the size range can be obtained to meet the actual recognition accuracy requirements.

[0052] The following uses the manual interception method to obtain the local speckle pattern, such as Figure 7 and Figure 8 As shown in the figure, the length and width of the image are used to represent the image size, and a comparison is made to verify the recognition effectiveness of this method.

[0053] Figure 7 In the paper, the MNIST digital image is cut from a complete image of 100*100 to 90*90, 80*80, 70*70, 60*60, 50*50, 40*40, 30*30, 20*20..., and the speckle image is also cut according to the size change. Local speckle images of different sizes correspond to local speckles of different sizes. Figure 8 The five sub-images in the figure show the image of an MNIST digit from its lower left corner, lower right corner, upper right corner, upper left corner, center, respectively. Figure 7 The local MNIST digital images of different sizes are captured in the manner of (cropped according to the frame in the figure). Correspondingly, in this example, the local speckle patterns of different sizes corresponding to the aforementioned local MNIST digital images are also captured from the lower left corner, lower right corner, upper right corner, upper left corner, and center of the captured speckle pattern. Among them, the above different orientations (lower left corner, lower right corner, upper right corner, upper left corner, and center) correspond to different positions in the entire speckle.

[0054] The neural network framework used in this example is ResNet-18, a neural network architecture widely used for classification problems. Fig. 9 The figure shows the structure of the ResNet-18 neural network. Conv refers to the convolution layer, 7*7 represents the convolution kernel size of 7*7, 3*3 represents the convolution kernel size of 3*3, max pooling represents the maximum pooling layer, and finally fc is the fully connected layer. The solid line in the middle represents direct addition, and the dotted line represents dimensionality reduction (1*1 convolution kernel, stride=2) before addition.

[0055] The neural network is trained and tested before identifying the target object. Specifically, the training and testing methods are as follows:

[0056] Data collection: 10,000 handwritten digital photos were randomly selected from the MNIST dataset and played. The corresponding 10,000 speckle images were captured and local speckle images were cut from the 10,000 speckle images to form a dataset. 90% of the random data in the dataset was used as the training set, and the remaining 10% was used as the test set.

[0057] Training: Adjust the local speckle image to 224*224*3, where the first two values ​​are the size, and the third value represents the RGB three channels. It is preferred to use a grayscale image, so the RGB three channel values ​​are the same. The neural network establishes a function mapping between the local speckle image and the specific MNIST data.

[0058] In the process of training a neural network, it is necessary to constantly adjust the parameters to make the prediction results of the neural network more accurate. This example uses the cross entropy loss function to calculate the deviation between the predicted value of the neural network and the true value:

[0059]

[0060] Where M refers to the number of categories (for the MNIST handwritten digit dataset, this number is 10), y ic represents a variable (0 or 1), if the category is the same as the sample i category, it is 1, otherwise it is 0; p ic Represents the predicted probability that the observed sample i belongs to category c.

[0061] The local speckle image (X) is input, and a series of calculations are performed on the internal parameters of the neural network to give a predicted value (Y^hat). The loss function (Loss Function) is calculated using the predicted value (Y^hat) and the true value (Y). The parameters are then updated through the back propagation algorithm (automatically implemented in modern neural network frameworks) and the optimizer (Optimize) to reduce the loss function, that is, the predicted value (Y^hat) is closer to the true value (Y).

[0062] By inputting the local speckle image into the trained neural network model, objects can be recognized.

[0063] like Fig.10 As shown, Figure 7 and Figure 8 The result of the object recognition verification process is based on the speckle image acquisition scheme shown in , where Fig.10 The five subgraphs in correspond to the above five relative position relationships (i.e. Figure 8 Each sub-image shows the prediction accuracy of different image sizes under a certain relative position relationship. Fig.10The results shown show that for speckle images, when no cropping is performed, the recognition accuracy can reach 97%. When cropped to a 5*5 pixel image (approximately 0.5° field of view), the recognition accuracy can still reach about 60%. As the cropping window becomes larger, the recognition accuracy gradually increases until it stabilizes. Fig.10 It can also be seen in each sub-figure in that for the speckle image, under any of the above relative position relationships (i.e. Figure 8 In any of the cropping methods, the size of the speckle image will have a critical point, above which the recognition accuracy remains almost unchanged, and below which the recognition accuracy begins to decrease (but even if it decreases, the accuracy is still not low, sufficient for non-line-of-sight object recognition). In addition, it can be seen that for speckle images, different cropping methods will not affect the recognition accuracy, that is, whether cropping from the periphery or from the middle, the effect achieved is almost the same. Fig.10 It can also be seen that for the original MNIST data set, when cropped to a size of 5*5, the recognition accuracy of cropping from the four sides is only about 10% to 11%, while the accuracy of cropping from the middle can reach 50%. It can be seen that the object recognition method of this embodiment can effectively recognize the acquired speckle image of the target object even when the size is very small and regardless of the relative position relationship between the acquired local speckle image and the camera lens, eliminating the obstacle of only being able to obtain local speckles caused by various factors in the actual application process, and is highly practical.

[0064] In addition, when the present technical solution is actually applied, problems such as camera disturbance may occur, resulting in deviation between the local area of ​​the object corresponding to the local speckle image used to identify the target object and the local area of ​​the object corresponding to the local speckle image sample set used to train the neural network, that is, they cannot be completely aligned. Taking this into account, the following simulation experiments are conducted to verify the robustness of the present technical solution:

[0065] like Fig.11 As shown in the figure, the image with a window size of 60*60 in the upper left corner is cropped as the training set, and then the window is translated horizontally and vertically respectively, and the translation distances are 0, 5, 10, 15, 20, 25, 30, 35, and 40. When the translation is 0, it is a complete overlap, and the rest are partial overlaps. The more the translation is, the less the overlap is.

[0066] like Fig.12 As shown, Fig.11 The experimental results were performed according to the scheme shown in . Fig.12The results shown in show that for partial overlap, when the translation is within 10 pixels, the technical solution can still achieve a recognition accuracy of nearly 60%. The results verify the robustness of the technical solution.

[0067] In summary, the method of this embodiment can realize the recognition of non-line-of-sight objects using local speckles. This enables the technical solution of combining coherent light source with camera to be better applied in actual scenes, eliminating the obstacle of only obtaining local speckles caused by various factors in actual application.

[0068] Embodiment 2

[0069] A non-line-of-sight object recognition system includes a coherent light source, a camera, and a processor. The coherent light source is used to illuminate a target object; the camera is used to capture speckles formed by the coherent light source irradiating the target object and reflecting to a scattering surface; and the processor is used to execute a non-line-of-sight object recognition method as described in the first embodiment above. A neural network model is set in the processor, and the processor receives the speckles and completes the process of recognition and outputting the results.

[0070] The relevant technical solution is the same as that in Embodiment 1 and will not be described in detail here.

[0071] Embodiment 3

[0072] A non-line-of-sight object recognition device comprises a coherent light source, a camera, a processor, and a mechanism for partially blocking the camera lens;

[0073] The coherent light source is used to illuminate the target object; the camera is used to capture the speckle formed by the coherent light source irradiating the target object and reflecting to the scattering surface; the processor is used to execute the non-line-of-sight object recognition method described in the above embodiment 1. The relevant technical solution is the same as that in embodiment 1 and will not be repeated here.

[0074] Embodiment 4

[0075] A computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute a non-line-of-sight object recognition method as described in the first embodiment above. The related technical solution is the same as the first embodiment and will not be described in detail here.

[0076] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A non-line-of-sight object recognition method, characterized in that: include: S1, controlling coherent light or partially coherent light to illuminate a target object and reflect the light to a scattering surface and then to a camera, so that the camera captures an image; obtaining a local speckle image of the target object from the image captured by the camera, wherein the local speckle image is an image of a portion of the overall speckle of the target object; S2. Input the local speckle image into a trained object information prediction neural network to obtain information of the target object and complete object recognition.

2. A non-line-of-sight object recognition method according to claim 1, characterized in that: Before S1, the method further includes: Constructing a recognition accuracy curve of the object information prediction neural network at different local speckle sizes, and determining a critical point of the local speckle size at an actually required recognition accuracy level based on a recognition accuracy variation trend on the recognition accuracy curve; Then the size of the local speckle image acquired when executing S1 is not smaller than the critical point.

3. A non-line-of-sight object recognition method according to claim 1 or 2, characterized in that: The field angle corresponding to the local speckle image is greater than or equal to 1°.

4. A non-line-of-sight object recognition method according to claim 1 or 2, characterized in that: The local speckle image is a grayscale image.

5. A non-line-of-sight object recognition system, characterized in that: Includes a coherent light source, camera, and processor; The coherent light source is used to illuminate the target object; the camera is used to photograph the speckle formed by the coherent light source irradiating the target object and reflecting to the scattering surface; the processor is used to execute a non-line-of-sight object recognition method as described in any one of claims 1 to 4.

6. A non-line-of-sight object recognition device, characterized in that: It includes a coherent light source, a camera, a processor, and a mechanism for partially blocking the camera lens; The coherent light source is used to illuminate the target object; the camera is used to capture the speckle formed by the coherent light source illuminating the target object and reflecting onto the scattering surface; the processor is used to execute a non-line-of-sight object recognition method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute a non-line-of-sight object recognition method according to any one of claims 1 to 4.

Images