Method and device for detecting and identifying non-vision-field moving target of area-array time-of-flight camera

Abstract

The invention relates to the field of non-vision-field three-dimensional point cloud data processing, and provides a non-vision-field moving target detection and recognition method and device for an area-array time-of-flight camera, and the method comprises the steps: obtaining the imaging three-dimensional point cloud data of a non-vision-field moving target based on the area-array time-of-flight camera; projecting the imaging three-dimensional point cloud data to a two-dimensional plane to obtain a two-dimensional projection drawing; performing target detection on the two-dimensional projection drawing by using a trained Yolov5 model to obtain a two-dimensional detection recognition result of the moving target, wherein the two-dimensional detection recognition result comprises the position coordinates of the two-dimensional detection frame, the target category and the confidence coefficient; determining the depth direction position of the moving target according to the depth direction distribution characteristics of the three-dimensional point cloud in the corresponding range of the two-dimensional detection frame; and the three-dimensional detection and recognition result of the moving target is determined by combining the two-dimensional detection and recognition result and the depth direction position, so that the detection and recognition of the non-vision-field moving target are realized by depending on the area-array time-of-flight camera with the cost advantage, and the non-vision-field application capability of the area-array time-of-flight camera is effectively expanded.

Description

Technical Field

[0001] The present invention relates to the technical field of non-line-of-sight three-dimensional point cloud data processing, and in particular to a method and device for detecting and identifying non-line-of-sight moving targets with a time-of-flight camera. Background Art

[0002] Non-line-of-sight imaging technology can achieve "viewing objects through walls", which is of great significance in the fields of medical imaging, disaster relief, etc. Time-of-flight (ToF) technology can determine the distance of the target object by measuring the time it takes for the light signal to be reflected from the transmitter to the target object and then back to the receiver. Compared with streak tube time-of-flight cameras and single photon avalanche diode time-of-flight cameras, area array time-of-flight cameras with price advantages have attracted the attention of many scholars. However, existing technical research focuses on the development of non-line-of-sight imaging systems based on area array time-of-flight cameras and the imaging processing and detection and recognition of static non-line-of-sight targets, and the detection and recognition capabilities of moving targets are insufficient.

[0003] In recent years, with the rapid development of deep learning technology, three-dimensional object recognition networks have emerged. Existing projection-based target object detection methods usually first project a single view or multiple views of a three-dimensional point cloud to generate a two-dimensional grid, and then process the two-dimensional grid to achieve detection and recognition of the target object. This method achieves a good balance between time complexity and detection performance at the expense of spatial information loss. However, the above-mentioned existing projection-based detection methods are difficult to directly adapt to non-line-of-sight imaging three-dimensional point cloud data obtained based on a planar array time-of-flight camera, which limits the application scope of non-line-of-sight imaging technology. Summary of the invention

[0004] The present disclosure aims to solve at least one of the problems existing in the prior art and provides a method and device for detecting and identifying non-line-of-sight moving targets using an array time-of-flight camera.

[0005] In one aspect of the present disclosure, a method for detecting and identifying non-line-of-sight moving targets using an array time-of-flight camera is provided, the method comprising: Acquire imaging 3D point cloud data of non-visual moving targets based on area array time-of-flight cameras; Projecting the imaging three-dimensional point cloud data onto a two-dimensional plane to obtain a corresponding two-dimensional projection image; Using the trained Yolov5 model to perform target detection on the two-dimensional projection image, and obtaining a two-dimensional detection and recognition result corresponding to the moving target, wherein the two-dimensional detection and recognition result includes the position coordinates of the two-dimensional detection box, the target category and the confidence level; Determining the depth position of the moving target according to the depth distribution characteristics of the three-dimensional point cloud within the corresponding range of the two-dimensional detection frame; The three-dimensional detection and recognition result of the moving target is determined by combining the two-dimensional detection and recognition result and the depth position.

[0006] Optionally, projecting the imaging three-dimensional point cloud data onto a two-dimensional plane to obtain a corresponding two-dimensional projection image includes: Taking a two-dimensional plane perpendicular to the depth direction of the area array time-of-flight camera as a projection plane; The imaging three-dimensional point cloud data is projected onto the projection plane for pixel coordinate system representation to obtain the two-dimensional projection image.

[0007] Optionally, projecting the imaging three-dimensional point cloud data onto the projection plane to perform pixel coordinate system representation to obtain the two-dimensional projection image includes: The first n The 3D point cloud is represented as ,in, Respectively represent n The length of a 3D point cloud in the camera coordinate system x Dimension, width y Dimension, depth z The corresponding coordinates of the dimension, ; N Indicates the total number of three-dimensional point clouds included in the imaging three-dimensional point cloud data; Determine respectively the length direction of each three-dimensional point cloud in the imaging three-dimensional point cloud data x Minimum coordinate of dimension and the maximum coordinates , and the width of each three-dimensional point cloud in the imaging three-dimensional point cloud data y Minimum coordinate of dimension and the maximum coordinates ; Will , , , The defined quadrilateral is used as the image range boundary, based on the imaging 3D point cloud data in the camera coordinate system x - y A scatter plot is drawn in a plane to obtain the two-dimensional projection graph.

[0008] Optionally, determining the depth position of the moving target according to the depth distribution characteristics of the three-dimensional point cloud within the corresponding range of the two-dimensional detection frame includes: The position coordinates of the two-dimensional detection frame are converted into the camera coordinate system, and all target point clouds within the range corresponding to the two-dimensional detection frame in the camera coordinate system are cut out from the imaging three-dimensional point cloud data as two-dimensional detection frame point clouds; Using a histogram to count the depth distribution of the two-dimensional detection frame point cloud, to obtain a depth distribution histogram corresponding to the two-dimensional detection frame point cloud; According to the following formula, the minimum depth value of the target three-dimensional frame corresponding to the moving target is determined: and the maximum depth value : ; ; in, and are threshold coefficients, represents the minimum depth value corresponding to the two-dimensional detection frame in the camera coordinate system, represents the maximum depth value corresponding to the two-dimensional detection frame in the camera coordinate system, Indicates the depth value corresponding to the peak value of the depth distribution histogram.

[0009] Optionally, converting the position coordinates of the two-dimensional detection frame into a camera coordinate system includes: The position coordinates of the two diagonal corners of the two-dimensional detection frame in the pixel coordinate system are respectively recorded as , ,in, They are the two diagonals of the two-dimensional detection frame in the pixel coordinate system x dimensional coordinates and , They are the two diagonals of the two-dimensional detection frame in the pixel coordinate system y dimensional coordinates and ; According to the following formula, the position coordinates of the two diagonal corners of the two-dimensional detection frame in the pixel coordinate system are , Convert to the position coordinates in the camera coordinate system respectively: ; ; in, Respectively represent the two-dimensional detection frame in the pixel coordinate system x Dimensional coordinates The corresponding camera coordinate system x dimensional coordinates, Respectively represent the two-dimensional detection frame in the pixel coordinate system y Dimensional coordinates The corresponding camera coordinate system y dimensional coordinates, Respectively represent the two-dimensional projection image in the pixel coordinate system x dimension, y The number of pixels in the dimension.

[0010] Optionally, combining the two-dimensional detection and recognition result and the depth position to determine the three-dimensional detection and recognition result of the moving target includes: Combining the position coordinates of the two-dimensional detection frame in the camera coordinate system with the depth position of the moving target to obtain a corresponding three-dimensional target frame position; According to the position of the three-dimensional target frame and the target category and confidence included in the two-dimensional detection and recognition result, the three-dimensional target frame, target category and confidence are marked in the imaging three-dimensional point cloud data in the camera coordinate system as the three-dimensional detection and recognition result of the moving target.

[0011] Optionally, the Yolov5 model is trained according to the following steps: Based on the area array time-of-flight camera, a plurality of different posture data and speed data of the non-viewing area moving target are collected respectively, and the plurality of different posture data and the speed data are imaged and processed to obtain the non-viewing area point cloud data of the moving target in each different posture; Annotating the non-viewing point cloud data in different postures respectively, determining the position information and category information of the moving target respectively corresponding to the non-viewing point cloud data in different postures, and constructing annotated data sets; The Yolov5 model is trained and verified using the labeled dataset.

[0012] Another aspect of the present disclosure provides a non-line-of-sight moving target detection and recognition device using an array time-of-flight camera, the device comprising: An acquisition module is used to acquire imaging three-dimensional point cloud data of non-viewing moving targets based on a plane array time-of-flight camera; A projection module, used to project the imaging three-dimensional point cloud data onto a two-dimensional plane to obtain a corresponding two-dimensional projection image; A two-dimensional detection module is used to perform target detection on the two-dimensional projection image using a trained Yolov5 model to obtain a two-dimensional detection and recognition result corresponding to the moving target, wherein the two-dimensional detection and recognition result includes the position coordinates of the two-dimensional detection frame, the target category and the confidence level; A depth determination module, used to determine the depth position of the moving target according to the depth distribution characteristics of the three-dimensional point cloud within the corresponding range of the two-dimensional detection frame; The result determination module is used to determine the three-dimensional detection and recognition result of the moving target by combining the two-dimensional detection and recognition result and the depth position.

[0013] Optionally, the device further comprises: A training module is used to train the Yolov5 model according to the following steps: Based on the area array time-of-flight camera, a plurality of different posture data and speed data of the non-viewing area moving target are collected respectively, and the plurality of different posture data and the speed data are imaged and processed to obtain the non-viewing area point cloud data of the moving target in each different posture; Annotating the non-viewing point cloud data in different postures respectively, determining the position information and category information of the moving target respectively corresponding to the non-viewing point cloud data in different postures, and constructing annotated data sets; The Yolov5 model is trained and verified using the labeled dataset.

[0014] Another aspect of the present disclosure provides an electronic device, including: at least one processor; and a memory in communication with the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the non-line-of-sight moving target detection and recognition method of the area array time-of-flight camera described above.

[0015] Another aspect of the present disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for detecting and identifying non-line-of-sight moving targets using a time-of-flight camera as described above.

[0016] Compared with the prior art, the present invention relies on the cost-effective area array time-of-flight camera, which not only retains the speed advantage of the single-stage detection of the Yolov5 model, but also completes the three-dimensional position of the moving target, and realizes the intelligent detection and recognition of the non-line-of-sight moving target imaging three-dimensional point cloud data obtained based on the area array time-of-flight camera, effectively broadening the non-line-of-sight application capabilities of the area array time-of-flight camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0018] Figure 1 A flowchart of a method for detecting and identifying non-line-of-sight moving targets using an area array time-of-flight camera according to an embodiment of the present disclosure; Figure 2 A flowchart of a method for detecting and identifying non-line-of-sight moving targets using an area array time-of-flight camera provided in another embodiment of the present disclosure; Figure 3 A schematic diagram of a non-field of view imaging optical path provided by another embodiment of the present disclosure; Figure 4 A schematic diagram of a two-dimensional target detection result of a moving target person provided by another embodiment of the present disclosure; Figure 5 A schematic diagram of two-dimensional target detection results of a moving target unmanned vehicle provided by another embodiment of the present disclosure; Figure 6 A schematic diagram of the position of a two-dimensional detection frame in a pixel coordinate system provided by another embodiment of the present disclosure; Figure 7 A schematic diagram of a three-dimensional detection and recognition result of a moving target person provided by another embodiment of the present disclosure; Figure 8 A schematic diagram of three-dimensional detection and recognition results of a moving target unmanned vehicle provided by another embodiment of the present disclosure; Fig. 9 A schematic diagram of the structure of a non-line-of-sight moving target detection and recognition device for an area array time-of-flight camera provided by another embodiment of the present disclosure; Fig.10 A schematic structural diagram of an electronic device provided in another embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. However, it can be understood by those skilled in the art that in each embodiment of the present disclosure, many technical details are proposed in order to enable readers to better understand the present disclosure. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed for protection in the present disclosure can also be implemented. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific implementation of the present disclosure. The various embodiments can be combined and referenced with each other without contradiction.

[0020] One embodiment of the present disclosure relates to a method for detecting and identifying non-line-of-sight moving targets using a time-of-flight camera. The process is as follows: Figure 1 As shown, it includes steps S110 to S150. Figure 2 Steps S110 to S150 are described in detail.

[0021] Step S110, acquiring imaging three-dimensional point cloud data of a non-viewing moving target based on an area array time-of-flight camera.

[0022] Specifically, an area array time-of-flight camera is an imaging device that can obtain scene depth information in real time. It calculates the distance between the object and the camera by measuring the time difference between the light signal emitted from the camera to the surface of the object and then reflected back to the camera, thereby generating a three-dimensional depth image and converting it into three-dimensional point cloud data.

[0023] Non-line-of-sight moving targets refer to moving targets that are not within the linear field of view of the area array time-of-flight camera but can be detected by the area array time-of-flight camera through other means such as reflection. In other words, there are obstacles between the non-line-of-sight moving targets and the area array time-of-flight camera, which causes the signal between the moving target and the area array time-of-flight camera to be unable to propagate in a straight line, but can only propagate through reflection and other means.

[0024] For example, step S110 may utilize an area array time-of-flight camera of model OPT8241, based on Figure 3 The non-line-of-sight imaging optical path shown in the figure is used to obtain the imaging 3D point cloud data of the moving target. Figure 3 In the non-line-of-sight imaging optical path shown, the light signal emitted by the light source cannot reach the moving target through straight-line propagation due to the obstruction of the obstructing surface, but can only reach the moving target through reflection on the reflective wall. At the same time, the light signal reaching the moving target cannot be reflected back to the area array time-of-flight camera through straight-line propagation due to the obstruction of the obstructing surface, but can only reach the area array time-of-flight camera through reflection on the reflective wall. At this time, the area array time-of-flight camera can receive the light signal reflected back by the moving target to complete the acquisition of the non-line-of-sight moving target reflection data. By filtering, enhancing and other processing of the non-line-of-sight moving target reflection data, the imaging three-dimensional point cloud data of the moving target can be obtained.

[0025] Step S120, projecting the imaging three-dimensional point cloud data onto a two-dimensional plane to obtain a corresponding two-dimensional projection image.

[0026] Exemplarily, step S120 may include: taking a two-dimensional plane perpendicular to the depth direction of the area array time-of-flight camera as a projection plane; projecting the imaging three-dimensional point cloud data onto the projection plane for pixel coordinate system representation to obtain a two-dimensional projection image.

[0027] Specifically, step S120 may use a two-dimensional plane composed of a length direction and a width direction perpendicular to the depth direction of the area array time-of-flight camera as a projection plane to retain the length direction information and the width direction information of the imaged three-dimensional point cloud data.

[0028] Exemplarily, in step S120, the imaging three-dimensional point cloud data is projected onto a projection plane for pixel coordinate system representation to obtain a two-dimensional projection map, including: n The 3D point cloud is represented as ,in, Respectively represent n The length of a 3D point cloud in the camera coordinate system x Dimension, width y Dimension, depth z The corresponding coordinates of the dimension, .N Represents the total number of 3D point clouds included in the imaging 3D point cloud data. Determine the length of each 3D point cloud in the imaging 3D point cloud data. x Minimum coordinate of dimension and the maximum coordinates , and the width of each 3D point cloud in the imaging 3D point cloud data y Minimum coordinate of dimension and the maximum coordinates .Will , , , The defined quadrilateral is used as the image range boundary, based on the imaging 3D point cloud data in the camera coordinate system. x - y Draw a scatter plot on the plane and get a two-dimensional projection.

[0029] Specifically, the dataset corresponding to the imaging 3D point cloud data P It can be expressed as . , That is, the imaging 3D point cloud data corresponds to x The boundary point coordinates of the dimension, , That is, the imaging 3D point cloud data corresponds to y The boundary point coordinates of the dimension , , , The defined quadrilateral is the image range boundary.

[0030] For example, when drawing a scatter plot, you can call the relevant drawing function to use the drawing function based on the imaging 3D point cloud data and the image range boundary in the camera coordinate system. x - y Draw a scatter plot on the plane, get the corresponding two-dimensional projection map, and store the two-dimensional projection map in a specified folder for use in subsequent steps.

[0031] Step S130, using the trained Yolov5 model to perform target detection on the two-dimensional projection image, to obtain a two-dimensional detection and recognition result corresponding to the moving target, the two-dimensional detection and recognition result including the position coordinates of the two-dimensional detection box, the target category and the confidence level.

[0032] Specifically, the Yolov5 model has a high detection accuracy as a target detection model. The network structure of the Yolov5 model generally includes an input layer, a backbone network, a neck network and a head network. Among them, the input layer is used to receive image data. The backbone network is used to extract the features of the image. The neck network is used to further process the features extracted by the backbone network to enhance the expressiveness of the features. The head network is used to generate the final detection results, including the bounding box position, target category and confidence corresponding to the target. Therefore, step S130 can input the two-dimensional projection image into the Yolov5 model, and perform two-dimensional detection and recognition based on the Yolov5 model, so as to obtain the two-dimensional detection and recognition results including the position coordinates of the two-dimensional detection box of the moving target, the target category and the confidence output by the Yolov5 model.

[0033] For example, step S130 can first set the hyperparameters of the Yolov5 model, then read the two-dimensional projection image stored in the specified folder, load the Yolov5 model, input the read two-dimensional projection image into the Yolov5 model for target detection, and obtain the position coordinates, target category and confidence of the two-dimensional detection box of the moving target output by the Yolov5 model. When the moving target is a person, the target detection result of the Yolov5 model, that is, the two-dimensional detection recognition result, is as follows: Figure 4 shown. Figure 4 In the example, the target category is a person, and the person opens his arms and is detected and recognized with a confidence level of 0.69. When the moving target is an unmanned vehicle, the target detection result of the Yolov5 model, i.e., the two-dimensional detection and recognition result, is as follows: Figure 5 shown. Figure 5 In the figure, the target type is an unmanned vehicle, and the outline and wheel features of the unmanned vehicle are significant and detected and identified with a confidence level of 0.85.

[0034] Step S140, determining the depth position of the moving target according to the depth distribution characteristics of the three-dimensional point cloud within the corresponding range of the two-dimensional detection frame.

[0035] Exemplarily, step S140 includes: converting the position coordinates of the two-dimensional detection frame to the camera coordinate system, and cutting out all target point clouds within the range corresponding to the two-dimensional detection frame in the camera coordinate system from the imaging three-dimensional point cloud data as the two-dimensional detection frame point cloud. Using a histogram to count the depth distribution of the two-dimensional detection frame point cloud, a depth distribution histogram corresponding to the two-dimensional detection frame point cloud is obtained. According to the following formula, the minimum depth value of the target three-dimensional frame corresponding to the moving target is determined: and the maximum depth value : ; .

[0036] in, and are threshold coefficients, Indicates the minimum depth value corresponding to the two-dimensional detection box in the camera coordinate system. Indicates the maximum depth value corresponding to the two-dimensional detection box in the camera coordinate system. Indicates the depth value corresponding to the peak of the depth distribution histogram.

[0037] Step S140 mainly converts the position coordinates of the two-dimensional detection frame in the pixel coordinate system to the camera coordinate system through coordinate conversion, so as to crop all target point clouds within the two-dimensional detection frame according to the position coordinates of the two-dimensional detection frame in the camera coordinate system, and determine the depth position of the moving target according to these target point clouds.

[0038] By determining the depth position of the moving target, the depth information of the moving target in the imaging three-dimensional point cloud data can be retained, the loss of the moving target's spatial information can be avoided, and the moving target can be identified more accurately.

[0039] Exemplarily, in step S140, the position coordinates of the two-dimensional detection frame are converted to the camera coordinate system, including: recording the position coordinates of two diagonal corners of the two-dimensional detection frame in the pixel coordinate system as , ,in, They are the two diagonals of the two-dimensional detection box in the pixel coordinate system x dimensional coordinates and , They are the two diagonals of the two-dimensional detection box in the pixel coordinate system y dimensional coordinates and According to the following formula, the position coordinates of the two diagonal corners of the two-dimensional detection frame in the pixel coordinate system are , Convert to the position coordinates in the camera coordinate system respectively: ; .

[0040] in, Respectively represent the two-dimensional detection box in the pixel coordinate system x Dimensional coordinates The corresponding camera coordinate system x dimensional coordinates, Respectively represent the two-dimensional detection box in the pixel coordinate system y Dimensional coordinates The corresponding camera coordinate system y dimensional coordinates, Respectively represent the two-dimensional projection image in the pixel coordinate system x dimension, y The number of pixels in the dimension.

[0041] For example, Figure 6 As shown, in the pixel coordinate system, let x The direction of the dimension is to the right. y The direction of the dimension is downward. , are the upper left corner and lower right corner of the two-dimensional detection frame in the pixel coordinate system, respectively. The two-dimensional detection frame is located between the origin O and point The defined rectangle.

[0042] Step S150, combining the two-dimensional detection and recognition result and the depth position to determine the three-dimensional detection and recognition result of the moving target.

[0043] Exemplarily, step S150 may include: combining the position coordinates of the two-dimensional detection frame in the camera coordinate system with the depth position of the moving target to obtain the corresponding three-dimensional target frame position. According to the three-dimensional target frame position and the target category and confidence included in the two-dimensional detection and recognition result, the three-dimensional detection frame, category and confidence corresponding to the target are marked in the imaging three-dimensional point cloud data in the camera coordinate system as the three-dimensional detection and recognition result of the moving target.

[0044] For example, in and hour, Figure 4 The three-dimensional detection and recognition results of the moving target person in Figure 7 As shown, Figure 5 The three-dimensional detection and recognition results of the moving target unmanned vehicle are as follows Figure 8 As shown. Among them, the X-axis, Y-axis, and Z-axis represent the dimensions of length, width, and height respectively, and the units of the X-axis, Y-axis, and Z-axis are all m (meter).

[0045] Compared with the prior art, the method for detecting and identifying non-line-of-sight moving targets using an area array time-of-flight camera provided in the embodiments of the present disclosure relies on the cost-effective area array time-of-flight camera, retains the speed advantage of the single-stage detection of the Yolov5 model, and completes the three-dimensional position of the moving target, thereby realizing intelligent detection and identification of the three-dimensional point cloud data of non-line-of-sight moving target imaging obtained based on the area array time-of-flight camera, and effectively broadens the non-line-of-sight application capabilities of the area array time-of-flight camera.

[0046] Exemplarily, the Yolov5 model is trained according to the following steps: Based on the array time-of-flight camera, a variety of different posture data and speed data of the non-line-of-sight moving target are collected respectively, and the various different posture data and speed data are imaged and processed to obtain the non-line-of-sight point cloud data of the moving target in different postures. The non-line-of-sight point cloud data in different postures are labeled respectively, and the position information and category information of the moving target corresponding to the non-line-of-sight point cloud data in different postures are determined, and a labeled data set is constructed. The Yolov5 model is trained and verified using the labeled data set.

[0047] For example, this embodiment can first build Figure 3 The optical path shown in FIG. 1 is used to collect data of moving target personnel in different postures such as swinging arms, walking, and lying prone, and at different speeds using a planar array time-of-flight camera, or collect data of moving target unmanned vehicles in 360° rotation, forward, backward, partial body, and full body, and at different speeds. At the same time, non-viewing background data without moving targets is collected, and further processed by filtering, enhancement, etc. to obtain corresponding non-viewing point cloud data. Labelimg is then used to annotate the non-viewing point cloud data, determine the position information and category information of the moving targets corresponding to the non-viewing point cloud data in different postures, and construct an annotated data set. Finally, the annotated data set is used to train and verify the Yolov5 model, so that the Yolov5 model can learn the characteristics of specific targets such as two-dimensional position information and target categories, and improve detection performance.

[0048] Another embodiment of the present disclosure relates to a non-line-of-sight moving target detection and recognition device for an area array time-of-flight camera, such as Fig. 9 As shown, it includes an acquisition module 910, a projection module 920, a two-dimensional detection module 930, a depth determination module 940, and a result determination module 950.

[0049] The acquisition module 910 is used to acquire imaging three-dimensional point cloud data of non-viewing moving targets based on the area array time-of-flight camera.

[0050] The projection module 920 is used to project the imaging three-dimensional point cloud data onto a two-dimensional plane to obtain a corresponding two-dimensional projection image.

[0051] The two-dimensional detection module 930 is used to perform target detection on the two-dimensional projection image using the trained Yolov5 model to obtain the two-dimensional detection and recognition results corresponding to the moving target. The two-dimensional detection and recognition results include the position coordinates of the two-dimensional detection box, the target category and the confidence level.

[0052] The depth determination module 940 is used to determine the depth position of the moving target according to the depth distribution characteristics of the three-dimensional point cloud within the corresponding range of the two-dimensional detection frame.

[0053] The result determination module 950 is used to determine the three-dimensional detection and recognition result of the moving target by combining the two-dimensional detection and recognition result and the depth position.

[0054] Exemplarily, the area array time-of-flight camera non-line-of-sight moving target detection and recognition device also includes a training module.

[0055] The training module is used to train the Yolov5 model according to the following steps: based on the area array time-of-flight camera, a variety of different posture data and speed data of the non-line-of-sight moving target are collected respectively, and the various different posture data and speed data are imaged and processed to obtain the non-line-of-sight point cloud data of the moving target in different postures; the non-line-of-sight point cloud data in different postures are labeled respectively, the position information and category information of the moving target corresponding to the non-line-of-sight point cloud data in different postures are determined, and a labeled data set is constructed; the Yolov5 model is trained and verified using the labeled data set.

[0056] The specific implementation method of the non-line-of-sight moving target detection and identification device for the area array time-of-flight camera provided in the embodiment of the present disclosure can be referred to the non-line-of-sight moving target detection and identification method for the area array time-of-flight camera provided in the embodiment of the present disclosure, and will not be repeated here.

[0057] Compared with the prior art, the non-line-of-sight moving target detection and recognition device for the area array time-of-flight camera provided in the embodiment of the present disclosure relies on the area array time-of-flight camera with cost advantage, retains the speed advantage of the single-stage detection of the Yolov5 model, and completes the three-dimensional position of the moving target, and realizes the intelligent detection and recognition of the non-line-of-sight moving target imaging three-dimensional point cloud data obtained based on the area array time-of-flight camera, which effectively broadens the non-line-of-sight application capability of the area array time-of-flight camera.

[0058] Another embodiment of the present disclosure relates to an electronic device, such as Fig.10 As shown, including: At least one processor 1001; and, a memory 1002 communicatively connected to the at least one processor 1001; wherein the memory 1002 stores instructions executable by the at least one processor 1001, and the instructions are executed by the at least one processor 1001 so that the at least one processor 1001 can execute the non-line-of-sight moving target detection and recognition method of the area array time-of-flight camera described in the above embodiment.

[0059] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0060] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0061] Another embodiment of the present disclosure relates to a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for detecting and identifying non-line-of-sight moving targets using a time-of-flight camera of an area array as described in the above embodiment.

[0062] That is, those skilled in the art can understand that all or part of the steps in the method described in the above embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0063] Those skilled in the art will appreciate that the above-mentioned embodiments are specific embodiments for implementing the present disclosure, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present disclosure.

Claims

1. A method for detecting and identifying non-line-of-sight moving targets using a time-of-flight camera, characterized in that: The method comprises: Acquire imaging 3D point cloud data of non-visual moving targets based on area array time-of-flight cameras; Projecting the imaging three-dimensional point cloud data onto a two-dimensional plane to obtain a corresponding two-dimensional projection image; Using the trained Yolov5 model to perform target detection on the two-dimensional projection image, and obtaining a two-dimensional detection and recognition result corresponding to the moving target, wherein the two-dimensional detection and recognition result includes the position coordinates of the two-dimensional detection box, the target category and the confidence level; Determining the depth position of the moving target according to the depth distribution characteristics of the three-dimensional point cloud within the corresponding range of the two-dimensional detection frame; The three-dimensional detection and recognition result of the moving target is determined by combining the two-dimensional detection and recognition result and the depth position.

2. The method according to claim 1, characterized in that The step of projecting the imaging three-dimensional point cloud data onto a two-dimensional plane to obtain a corresponding two-dimensional projection image includes: Taking a two-dimensional plane perpendicular to the depth direction of the area array time-of-flight camera as a projection plane; The imaging three-dimensional point cloud data is projected onto the projection plane for pixel coordinate system representation to obtain the two-dimensional projection image.

3. The method according to claim 2, characterized in that The step of projecting the imaging three-dimensional point cloud data onto the projection plane to characterize the pixel coordinate system to obtain the two-dimensional projection image includes: The first n The 3D point cloud is represented as ,in, Respectively represent n The length of a 3D point cloud in the camera coordinate system x Dimension, width y Dimension, depth z The corresponding coordinates of the dimension, ; N Indicates the total number of three-dimensional point clouds included in the imaging three-dimensional point cloud data; Determine respectively the length direction of each three-dimensional point cloud in the imaging three-dimensional point cloud data x Minimum coordinate of dimension and the maximum coordinates , and the width of each three-dimensional point cloud in the imaging three-dimensional point cloud data y Minimum coordinate of dimension and the maximum coordinates ; Will , , , The defined quadrilateral is used as the image range boundary, based on the imaging 3D point cloud data in the camera coordinate system x - y A scatter plot is drawn in a plane to obtain the two-dimensional projection graph.

4. The method according to claim 3, characterized in that The step of determining the depth position of the moving target according to the depth distribution characteristics of the three-dimensional point cloud within the corresponding range of the two-dimensional detection frame includes: The position coordinates of the two-dimensional detection frame are converted into the camera coordinate system, and all target point clouds within the corresponding range of the two-dimensional detection frame in the camera coordinate system are cut out from the imaging three-dimensional point cloud data as two-dimensional detection frame point clouds; Using a histogram to count the depth distribution of the two-dimensional detection frame point cloud, to obtain a depth distribution histogram corresponding to the two-dimensional detection frame point cloud; According to the following formula, the minimum depth value of the target three-dimensional frame corresponding to the moving target is determined: and the maximum depth value : ； ； in, and are threshold coefficients, represents the minimum depth value corresponding to the two-dimensional detection frame in the camera coordinate system, represents the maximum depth value corresponding to the two-dimensional detection frame in the camera coordinate system, Indicates the depth value corresponding to the peak value of the depth distribution histogram.

5. The method according to claim 4, characterized in that The converting the position coordinates of the two-dimensional detection frame into a camera coordinate system includes: The position coordinates of the two diagonal corners of the two-dimensional detection frame in the pixel coordinate system are respectively recorded as , ,in, They are the two diagonals of the two-dimensional detection frame in the pixel coordinate system x dimensional coordinates and , They are the two diagonals of the two-dimensional detection frame in the pixel coordinate system y dimensional coordinates and ; According to the following formula, the position coordinates of the two diagonal corners of the two-dimensional detection frame in the pixel coordinate system are , Convert to the position coordinates in the camera coordinate system respectively: ； ； in, Respectively represent the two-dimensional detection frame in the pixel coordinate system x Dimensional coordinates The corresponding camera coordinate system x dimensional coordinates, Respectively represent the two-dimensional detection frame in the pixel coordinate system y Dimensional coordinates The corresponding camera coordinate system y dimensional coordinates, Respectively represent the two-dimensional projection image in the pixel coordinate system x dimension, y The number of pixels in the dimension.

6. The method according to claim 5, characterized in that The combining the two-dimensional detection and recognition result and the depth position to determine the three-dimensional detection and recognition result of the moving target includes: Combining the position coordinates of the two-dimensional detection frame in the camera coordinate system with the depth position of the moving target to obtain a corresponding three-dimensional target frame position; According to the position of the three-dimensional target frame and the target category and confidence included in the two-dimensional detection and recognition result, the three-dimensional target frame, target category and confidence are marked in the imaging three-dimensional point cloud data in the camera coordinate system as the three-dimensional detection and recognition result of the moving target.

7. The method according to any one of claims 1 to 6, characterized in that: The Yolov5 model is trained according to the following steps: Based on the area array time-of-flight camera, a plurality of different posture data and speed data of the non-viewing area moving target are collected respectively, and the plurality of different posture data and the speed data are imaged and processed to obtain the non-viewing area point cloud data of the moving target in each different posture; Annotating the non-viewing point cloud data in different postures respectively, determining the position information and category information of the moving target respectively corresponding to the non-viewing point cloud data in different postures, and constructing annotated data sets; The Yolov5 model is trained and verified using the labeled dataset.

8. A non-line-of-sight moving target detection and recognition device using an array time-of-flight camera, characterized in that: The device comprises: An acquisition module is used to acquire imaging three-dimensional point cloud data of non-viewing moving targets based on a plane array time-of-flight camera; A projection module, used to project the imaging three-dimensional point cloud data onto a two-dimensional plane to obtain a corresponding two-dimensional projection image; A two-dimensional detection module is used to perform target detection on the two-dimensional projection image using a trained Yolov5 model to obtain a two-dimensional detection and recognition result corresponding to the moving target, wherein the two-dimensional detection and recognition result includes the position coordinates of the two-dimensional detection frame, the target category and the confidence level; A depth determination module, used to determine the depth position of the moving target according to the depth distribution characteristics of the three-dimensional point cloud within the corresponding range of the two-dimensional detection frame; The result determination module is used to determine the three-dimensional detection and recognition result of the moving target by combining the two-dimensional detection and recognition result and the depth position.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the non-line-of-sight moving target detection and recognition method for an area array time-of-flight camera according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for detecting and identifying non-line-of-sight moving targets with an area array time-of-flight camera according to any one of claims 1 to 7 is implemented.

Images