Motion detection method, device and equipment based on event camera and lidar

Abstract

The present invention provides a motion detection method, device and equipment based on event cameras and laser radars. The method clearly combines the advantages of event cameras and laser radars, can not only obtain the spatial position of an object, but also determine whether the object is moving, and realizes the judgment of whether each point is moving while outputting point cloud spatial information. Compared with other motion detection algorithms that only use laser radars or RGB cameras, the present invention only requires one scanning cycle of the laser radar, such as one frame of data, to complete motion state detection. In addition, the present invention can also use point cloud data containing motion information to simplify the implementation of many tasks. The motion detection method of the present invention can filter out all moving point clouds and simplify the mapping process. In some target detection tasks, all point clouds can be screened once based on whether they are moving or not, which shortens the recognition time and simplifies the working process.

Description

Technical Field

[0001] The present invention relates to the technical fields of sensor fusion, computer vision and motion detection algorithms, and in particular to a motion detection method, apparatus and device based on an event camera and a laser radar. Background Art

[0002] With the development of technology, the application of LiDAR (LiDAR) in our daily lives has become increasingly common. It can improve the focus accuracy of smartphones when taking photos or videos, and can also help cars achieve different levels of autonomous driving. It can not only allow sweeping robots to locate and clean designated areas, but also improve mapping accuracy and build clearer maps. The principle of LiDAR is to emit a laser beam at a target and then compare the received signal reflected from the target with the transmitted signal. After relevant processing, a point cloud containing the spatial position information of the object can be obtained. However, LiDAR cannot obtain dynamic information about the object in a single scan, that is, it cannot determine whether the object is moving. In many application scenarios, such as simultaneous localization and mapping (SLAM), if the motion of the point cloud can be determined, the accuracy and speed of mapping and positioning can be greatly improved.

[0003] An event camera is a relatively new sensor, invented by researchers inspired by the workings of the human retina. It can only capture objects that are in motion or whose surface illumination changes. If an object remains stationary and its surface illumination does not change, it will not be recorded by the event camera. Furthermore, the images produced by an event camera are not frame-by-frame images, but rather point clouds similar to lidar. Unlike lidar, while an event camera can capture dynamic information about an object, it does not include its spatial depth; it only captures the point cloud's position on a plane.

[0004] Currently, there are several technologies that utilize event cameras for dynamic monitoring. For example, patent ZL202110811885.3 uses a neural network to process signals collected by an event camera to determine whether an object is in motion. Patent ZL202011088240.3 uses event stream integration and filtering to obtain the velocity of each event camera data point, thereby determining whether it is in motion. Regarding the fusion of LiDAR and event cameras, patent ZL202111502007.X uses event cameras to increase the density of LiDAR point clouds.

[0005] However, an inherent flaw in existing technologies for dynamic recognition based on event cameras is that, due to the lack of fusion with LiDAR, even if it can determine whether an object is moving, it cannot obtain the spatial information of the scanned point cloud. The drawback of dynamic recognition technology based on LiDAR is that at least two frames of LiDAR information are required to obtain results. Both the calculation of the neural network and the integration of the event stream require a large amount of computing resources and training time, making real-time recognition difficult. Fusion algorithms based on LiDAR and event cameras, while able to increase the density of LiDAR point clouds, cannot determine the dynamic information of objects. Summary of the Invention

[0006] In view of the above problems, the present invention provides a motion detection method, device and equipment based on event camera and lidar, which outputs point cloud spatial information while determining whether each point is moving.

[0007] To achieve the above-mentioned object, the first aspect of the present invention provides a motion detection method based on an event camera and a laser radar, comprising: step S1, configuring and calibrating the event camera and the laser radar; step S2, acquiring a moving target to be detected, scanning the moving target, and collecting multiple two-dimensional imaging points of the event camera and multiple three-dimensional point clouds of the laser radar in one scanning cycle; step S3, projecting the multiple three-dimensional point clouds onto the plane where the event camera is located to obtain multiple two-dimensional projection points of the laser radar; step S4, extracting distance data from the multiple two-dimensional projection points, and calculating the distance data according to the maximum effective distance z of the laser radar. max And the preset distance separation threshold m, cluster the distance data to get z max / m distance range categories; step S5, for each two-dimensional projection point in each distance range category, traverse and calculate the Euler distance between the two-dimensional projection point and all the two-dimensional imaging points of the event camera and all the two-dimensional projection points of the laser radar in the same distance range category, and when the Euler distance is less than the preset distance, accumulate the target points of the two-dimensional imaging point and the target points of the two-dimensional projection point; step S6, compare the target points of the two-dimensional imaging point and the target points of the two-dimensional projection point with the preset first point number threshold and second point number threshold respectively, obtain the motion state detection result of the two-dimensional projection point, and generate the corresponding motion state identification; step S7, repeat the above steps S5 to S6 until the motion state identification of all the two-dimensional projection points of the laser radar are generated, and merge the three-dimensional point cloud of the laser radar and the corresponding motion state identification for output.

[0008] The second aspect of the present invention provides a motion detection device based on an event camera and a laser radar, comprising: a configuration and calibration module for configuring and calibrating the event camera and the laser radar; a data acquisition module for acquiring a moving target to be detected, scanning the moving target, and acquiring multiple two-dimensional imaging points of the event camera and multiple three-dimensional point clouds of the laser radar within a scanning cycle; a data projection module for projecting the multiple three-dimensional point clouds onto the plane where the event camera is located to obtain multiple two-dimensional projection points of the laser radar; a distance clustering module for extracting distance data from the multiple two-dimensional projection points, and clustering the distance data according to the farthest effective distance z of the laser radar. max And the preset distance separation threshold m, cluster the distance data to get z max / m distance range categories; a point accumulation module, for traversing and calculating the Euler distance between each two-dimensional projection point in each distance range category and all two-dimensional imaging points of the event camera and all two-dimensional projection points of the laser radar in the same distance range category, and when the Euler distance is less than the preset distance, accumulating the target points of the two-dimensional imaging point and the target points of the two-dimensional projection point; a point comparison module, for comparing the target points of the two-dimensional imaging point and the target points of the two-dimensional projection point with the preset first point threshold and second point threshold respectively, obtaining the motion state detection result of the two-dimensional projection point, and generating the corresponding motion state identification; a result output module, for repeating the steps from the above-mentioned point accumulation module to the point comparison module until the motion state identification of all two-dimensional projection points of the laser radar are generated, and merging the three-dimensional point cloud of the laser radar and the corresponding motion state identification for output.

[0009] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0010] Compared with the prior art, the motion detection method, device and apparatus based on event camera and lidar provided by the present invention have at least the following beneficial effects:

[0011] (1) The present invention combines the advantages of event cameras and lidar, which can not only obtain the spatial position of an object, but also determine whether the object is moving. It can output point cloud spatial information while determining whether each point is moving.

[0012] (2) Compared with other motion detection algorithms that only use lidar or RGB camera, the present invention only needs data from one scanning cycle (e.g., one frame) of the lidar to complete motion state detection;

[0013] (3) The present invention can also simplify the implementation of many tasks by using point cloud data containing motion information. For example, in the process of map surveying and mapping, moving vehicles and pedestrians will interfere with the surveying and mapping. The motion detection method of the present invention can filter out all moving point clouds and simplify the mapping process. In addition, in certain target detection tasks, all point clouds can be screened by whether they are moving or not. For example, to identify trees or road signs, they can only be identified in the static point cloud, which can shorten the recognition time and simplify the work process. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0015] FIG1( a ), FIG1( b ), and FIG1( c ) schematically illustrate a front view, a top view, and a side view, respectively, of an installation method of an event camera and a laser radar according to an embodiment of the present invention;

[0016] Figure 2 Schematically shows a flow chart of a motion detection method based on an event camera and a laser radar according to an embodiment of the present invention;

[0017] Figure 3 A diagram schematically illustrates the horizontal viewing angle geometric relationship between an event camera and a laser radar according to an embodiment of the present invention;

[0018] Figure 4 Schematically shows a distribution diagram of multiple two-dimensional imaging points of an event camera and multiple two-dimensional projection points of a laser radar according to an embodiment of the present invention;

[0019] Figure 5 Schematically shows a sampling timing diagram of an event camera and a lidar according to an embodiment of the present invention;

[0020] Figure 6 Schematically shows a block diagram of a motion detection device based on an event camera and a laser radar according to an embodiment of the present invention;

[0021] Figure 7 A block diagram of an electronic device suitable for implementing an access control method according to an embodiment of the present disclosure is schematically shown.

[0022] [Description of Reference Numerals]

[0023] 1-LiDAR; 2-Event camera; 21-Lens center of the event camera; 3-Protective case and other supporting circuits. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific embodiments and the accompanying drawings. It is apparent that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] FIG1( a ), FIG1( b ) and FIG1( c ) schematically illustrate a front view, a top view and a side view of an installation method of an event camera and a laser radar according to an embodiment of the present invention, respectively.

[0028] Before performing the following steps, you first need to install the event camera 2 and the lidar 1 according to the following principles: the event camera 2 and its supporting circuits are fixed in the protective shell, and the lidar 1 is fixed on the protective shell 3; the central axis of the lidar 1 coincides with the lens center 21 of the event camera.

[0029] Specifically, from the front and top view perspectives, the center axis of the LiDAR 1 and the lens center 21 of the event camera should coincide, as shown by the dashed lines in Figures 1(a) and 1(b). As shown in Figure 1(c), from the side view perspective, the LiDAR 1 cannot be installed too far back, that is, the LiDAR 1 should be as close to the z-axis origin as possible so that the protective shell 3 does not block the downward scanning beam of the LiDAR 1. The distance from the center of the LiDAR 1 to the event camera 2 is recorded as L.

[0030] Then, after the event camera 2 and lidar 1 are installed, Figure 2 The flowchart of the motion detection method based on event camera and laser radar according to an embodiment of the present invention is schematically shown.

[0031] like Figure 2 As shown, the motion detection method based on the event camera and the laser radar according to this embodiment may include operations S1 to S7.

[0032] Step S1: configure and calibrate the event camera and lidar.

[0033] In an embodiment of the present invention, the configuration and calibration specifically include: controlling the horizontal viewing angle of the laser radar 1 to be smaller than the horizontal viewing angle of the event camera 2; controlling the vertical viewing angle of the laser radar 1 to be 60% to 80% of the vertical viewing angle of the event camera 2, and the line beam is evenly distributed; ensuring that the parallax of the event camera 2 and the laser radar 1 when detecting objects at different distances is smaller than a preset parallax threshold.

[0034] See also Figure 3 , which represents the event camera 2 and the lidar 1 from a top-down perspective. The horizontal perspective of the lidar 1 is shown by the dotted line and can be recorded as θ Lh The horizontal viewing angle of event camera 2 is shown as a solid line and can be denoted as θ Eh In the vertical direction, the vertical viewing angle of the laser radar 1 is generally not adjustable. When purchasing the equipment, it should be ensured that its vertical viewing angle is 60%-80% of the vertical viewing angle of the event camera 2, and the beam is evenly distributed. The vertical viewing angle of the laser radar 1 is denoted as θ Lv , the vertical viewing angle of event camera 2 is denoted as θ Ev During the calibration process, the parallax threshold is set and the relevant formula is used to ensure that the parallax between the event camera 2 and the lidar 1 when detecting objects at different distances is as small as possible. The parallax threshold can be set according to the needs of the actual application and is not limited in the present invention.

[0035] Step S2: Acquire the moving target to be detected, scan the moving target, and collect multiple two-dimensional imaging points of the event camera and multiple three-dimensional point clouds of the laser radar within one scanning cycle.

[0036] In step S3, the multiple three-dimensional point clouds are projected onto the plane where the event camera is located to obtain multiple two-dimensional projection points of the laser radar.

[0037] In the embodiment of the present invention, within one scanning cycle, multiple two-dimensional projection points of the laser radar are obtained according to the following formula:

[0038] x′ li =Lx li / z li

[0039] y′ li =Ly li / z li

[0040] Among them, (x′ li , y′ li ) are the coordinates of multiple two-dimensional projection points of the laser radar; (x li ,y li , z li) are the coordinates of multiple three-dimensional point clouds of the lidar; L is the distance from the center of the lidar to the event camera.

[0041] See also Figure 4 The solid black boxes represent the multiple 2D imaging points captured by the event camera, and the hollow circles represent the multiple 2D projection points of the lidar. Because the lidar's horizontal and vertical viewing angles are smaller than those of the event camera during configuration and calibration, the lidar's projection range does not extend beyond the plane of the event camera. After projection, the lidar's range data (also known as the z-axis data) is not displayed on the graph but is saved for use in subsequent tasks.

[0042] It should be noted that when scanning a moving target, the sampling time of the event camera is longer than that of the lidar. Figure 5 , the black square represents the timestamp of the event camera sampling, and the hollow circle represents the timestamp of the lidar sampling. After collecting the lidar data once, the coordinates of each event camera point in its plane are recorded as p ei (x ei ,y ei ). Before the next laser radar data arrives, the projected laser radar point p′ can be calculated. li (x′ li , y′ li ) and event camera point p ei (x ei ,y ei ) is calculated. Here, the multiple projection points of the lidar and the multiple two-dimensional imaging points of the event camera satisfy the following relationship:

[0043]

[0044] Among them, p ei (x ei ,y ei ) are the two-dimensional coordinates of multiple two-dimensional imaging points of the event camera; p′ li (x′ li , y′ li ) are multiple two-dimensional projection points of the lidar; set A represents the plane range of the event camera imaging; set B represents the plane range after lidar projection. It can be understood that set B is included in set A because the lidar's field of view is smaller than the event camera's field of view.

[0045] Step S4, extract the distance data of multiple two-dimensional projection points, according to the farthest effective distance z of the laser radar max And the preset distance separation threshold m, cluster the distance data to get z max / m distance range categories.

[0046] For each two-dimensional projection point of the laser radar, a cluster is performed based on its distance data (i.e., z-axis data). The farthest effective distance z max It is set by the laser radar itself. After clustering, all the two-dimensional projection points of the laser radar are divided into z max / m distance range categories, so that each distance range category has at least one two-dimensional projection point.

[0047] For example, the maximum effective distance z max is 200, and the distance separation threshold m is set to 10. Then all two-dimensional projection points can be divided into 20 distance range categories. The first distance range category contains all two-dimensional projection points with a distance of 0-10, the second distance range category contains all two-dimensional projection points with a distance of 10-20, and so on.

[0048] Step S5, for each two-dimensional projection point in each distance range category, traverse and calculate the Euler distance between the two-dimensional projection point and all two-dimensional imaging points of the event camera and all two-dimensional projection points of the laser radar in the same distance range category. When the Euler distance is less than the preset distance, accumulate the number of target points of the two-dimensional imaging point and the number of target points of the two-dimensional projection point.

[0049] Specifically, for each two-dimensional projection point in each distance range category, the two-dimensional projection point is used as a reference point, and the reference point is traversed with all two-dimensional imaging points of the event camera in the corresponding distance range category, as well as all two-dimensional projection points of the lidar in the same distance range category, and the Euler distance between the reference point and any point in these two points is calculated.

[0050] Next, each Euler distance is compared with the preset distance. When the Euler distance is less than the preset distance, the two types of points that meet the conditions are accumulated as the target points of the two-dimensional imaging points of the event camera or the target points of the two-dimensional projection points of the lidar according to their belonging to the event camera or lidar.

[0051] The Euler distance is calculated according to the following formula:

[0052]

[0053] Among them, (x i ,y i ) represents the coordinates of the currently selected i-th two-dimensional projection point; (x j ,y j ) represents the coordinates of any two-dimensional imaging point of the event camera or any two-dimensional projection point in the same distance range category as the i-th two-dimensional projection point; d(i, j) represents the Euler distance.

[0054] Specifically, for a 2D projection point of a LiDAR in the 50-60 range, use it as a reference point and calculate the Euler distance between that reference point and all 2D imaging points of the event camera on the projection plane at that time, as well as all 2D projection points of the LiDAR in the 50-60 range. If any of the calculated Euler distances is less than a preset distance (e.g., 1), the number of such points is recorded and represented by N1 and N2.

[0055] Based on the Euler formula above, i will eventually traverse every 2D projection point within the distance range category. That is, each time an i is selected, all j points are updated. Point j includes all event camera points and lidar points within the same distance range. This process is then repeated for the next i point until all lidar 2D projection points have been included in i.

[0056] Step S6, comparing the target point number of the two-dimensional imaging point and the target point number of the two-dimensional projection point with the preset first point number threshold and second point number threshold respectively, obtaining the motion state detection result of the two-dimensional projection point, and generating a corresponding motion state identifier.

[0057] In the embodiment of the present invention, the first point threshold and the second point threshold are different according to different usage scenarios or usage environments.

[0058] In this embodiment of the present invention, the motion state detection result includes motion or stillness. Based on this, when the target point count of the two-dimensional imaging point is greater than a first point count threshold, and the target point count of the two-dimensional projection point is greater than a second point count threshold, the motion state detection result of the two-dimensional projection point is motion; otherwise, it is stillness.

[0059] Then, according to the motion state detection result of the two-dimensional projection point, a corresponding motion state identifier is generated. For example, "0" can be represented as stationary, and "1" can be represented as moving.

[0060] Step S7, repeat the above steps S5 to S6 until the motion state identifications of all two-dimensional projection points of the laser radar are generated, and merge the three-dimensional point cloud of the laser radar and the corresponding motion state identifications and output them.

[0061] In the embodiment of the present invention, the detection result of the moving target is output according to the following data format:

[0062] data(x, y, z, p)

[0063] Among them, (x, y, z) is the coordinate of the three-dimensional point cloud of the laser radar; p is the motion state identifier, which has two values, "0" represents stillness and "1" represents motion.

[0064] Through the above embodiments, the present invention combines the advantages of event cameras and lidars, which can not only obtain the spatial position of an object, but also determine whether the object is moving, and realize the determination of whether each point is moving while outputting point cloud spatial information.

[0065] Compared with other motion detection algorithms that only use lidar or RGB cameras, the present invention only requires data from one scanning cycle (e.g., one frame) of the lidar to complete motion state detection.

[0066] Furthermore, the present invention can also simplify the implementation of many tasks by utilizing point cloud data containing motion information. For example, during map surveying, moving vehicles and pedestrians can interfere with the surveying process. The motion detection method of the present invention can filter out all moving point clouds, simplifying the mapping process. Furthermore, in certain target detection tasks, all point clouds can be screened based on whether they are moving or not. For example, to identify trees or road signs, recognition can be performed only in stationary point clouds, which can shorten the recognition time and simplify the work process.

[0067] Based on the above-mentioned motion detection method based on event camera and laser radar, the present invention also provides a motion detection device based on event camera and laser radar. Figure 6 The device is described in detail.

[0068] Figure 6 The block diagram of a motion detection device based on an event camera and a laser radar according to an embodiment of the present invention is schematically shown.

[0069] like Figure 6 As shown, the motion detection device 600 based on event camera and lidar according to this embodiment includes a configuration calibration module 610, a data acquisition module 620, a data projection module 630, a distance clustering module 640, a point accumulation module 650, a point comparison module 660 and a result output module 670.

[0070] Configuration and calibration module 610, for configuring and calibrating the event camera and lidar;

[0071] The data acquisition module 620 is used to acquire a moving target to be detected, scan the moving target, and acquire multiple two-dimensional imaging points of the event camera and multiple three-dimensional point clouds of the laser radar within one scanning cycle;

[0072] A data projection module 630 is configured to project the multiple three-dimensional point clouds onto the plane where the event camera is located to obtain multiple two-dimensional projection points of the laser radar;

[0073] The distance clustering module 640 is used to extract the distance data of the multiple two-dimensional projection points according to the farthest effective distance z of the laser radar.max and the preset distance separation threshold m, cluster the distance data to obtain z max / m distance range categories;

[0074] A point accumulation module 650 is configured to calculate, for each two-dimensional projection point in each distance range category, the Euler distance between the two-dimensional projection point and all two-dimensional imaging points of the event camera and all two-dimensional projection points in the same distance range category, and accumulate the target point count of the two-dimensional imaging point and the target point count of the two-dimensional projection point when the Euler distance is less than a preset distance;

[0075] a point number comparison module 660 for comparing the target point number of the two-dimensional imaging point and the target point number of the two-dimensional projection point with a preset first point number threshold and a preset second point number threshold, respectively, to obtain a motion state detection result of the two-dimensional projection point and generate a corresponding motion state identifier;

[0076] The result output module 670 is used to repeat the steps from the above-mentioned point accumulation module to the point comparison module until the motion state identifications of all two-dimensional projection points of the laser radar are generated, and the three-dimensional point cloud of the laser radar and the corresponding motion state identifications are merged and output.

[0077] It should be noted that the embodiment of the device part is similar to the embodiment of the method part, and the technical effects achieved are also similar. For specific details, please refer to the above-mentioned method embodiment part, which will not be repeated here.

[0078] According to an embodiment of the present invention, any multiple of the configuration calibration module 610, the data acquisition module 620, the data projection module 630, the distance clustering module 640, the point accumulation module 650, the point comparison module 660, and the result output module 670 can be combined into a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present invention, at least one of the configuration calibration module 610, the data acquisition module 620, the data projection module 630, the distance clustering module 640, the point accumulation module 650, the point comparison module 660, and the result output module 670 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware through any other reasonable means of circuit integration or packaging, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the configuration calibration module 610, the data acquisition module 620, the data projection module 630, the distance clustering module 640, the point accumulation module 650, the point comparison module 660, and the result output module 670 can be at least partially implemented as a computer program module, which can perform the corresponding function when executed.

[0079] Figure 7 A block diagram of an electronic device suitable for implementing an access control method according to an embodiment of the present disclosure is schematically shown.

[0080] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. The processor 701 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0081] Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and RAM 703. The processor 701 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0082] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage portion 708 including a hard disk; and a communication portion 709 including a network interface card such as a LAN card or a modem. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 710 as needed, so that a computer program read therefrom can be installed into the storage portion 708 as needed.

[0083] The accompanying drawings show some block diagrams and / or flow charts. It should be understood that some blocks in the block diagrams and / or flow charts, or combinations thereof, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when these instructions are executed by the processor, they can create a device for implementing the functions / operations described in the block diagrams and / or flow charts.

[0084] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly specifying the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined. Furthermore, the word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.

[0085] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A motion detection method based on event camera and laser radar, characterized in that: include: Step S1, configure and calibrate the event camera and lidar; Step S2: acquiring a moving target to be detected, scanning the moving target, and collecting multiple two-dimensional imaging points of the event camera and multiple three-dimensional point clouds of the laser radar within one scanning cycle; Step S3, projecting the multiple three-dimensional point clouds onto the plane where the event camera is located to obtain multiple two-dimensional projection points of the laser radar; Step S4, extracting the distance data of the multiple two-dimensional projection points, and calculating the maximum effective distance z of the laser radar. max and the preset distance separation threshold m, cluster the distance data to obtain z max / m distance range categories; Step S5: for each two-dimensional projection point in each distance range category, traversally calculate the Euler distance between the two-dimensional projection point and all two-dimensional imaging points of the event camera and all two-dimensional projection points of the lidar in the same distance range category; when the Euler distance is less than a preset distance, accumulate the number of target points of the two-dimensional imaging point and the number of target points of the two-dimensional projection point; Step S6, comparing the target point number of the two-dimensional imaging point and the target point number of the two-dimensional projection point with a preset first point number threshold and a preset second point number threshold, respectively, obtaining a motion state detection result of the two-dimensional projection point, and generating a corresponding motion state identifier; Step S7, repeat the above steps S5 to S6 until the motion state identifications of all two-dimensional projection points of the laser radar are generated, and merge the three-dimensional point cloud of the laser radar and the corresponding motion state identifications and output them.

2. The motion detection method based on event camera and laser radar according to claim 1, characterized in that: Before step S1, the event camera and laser radar are installed according to the following principles: The event camera and its supporting circuits are fixed in a protective shell, and the laser radar is fixed on the protective shell; The central axis of the laser radar coincides with the lens center of the event camera.

3. The motion detection method based on event camera and laser radar according to claim 1, characterized in that: The step S1 specifically includes: Controlling the horizontal viewing angle of the laser radar to be smaller than the horizontal viewing angle of the event camera; Controlling the vertical viewing angle of the laser radar to be 60% to 80% of the vertical viewing angle of the event camera, and the beams are evenly distributed; Ensure that the parallax between the event camera and the lidar when detecting objects at different distances is less than a preset parallax threshold.

4. The motion detection method based on event camera and laser radar according to claim 1, characterized in that: The multiple two-dimensional projection points of the laser radar are obtained according to the following formula: x′ li =Lx li / z li y′ li =Ly li / With li Among them, (x′ li , y′ li ) are the coordinates of multiple two-dimensional projection points of the laser radar; (x li ,y li , z li ) are the coordinates of multiple three-dimensional point clouds of the lidar; L is the distance from the center of the lidar to the event camera.

5. The motion detection method based on event camera and laser radar according to claim 1, characterized in that: When scanning the moving target, the sampling time of the event camera is greater than the sampling time of the laser radar.

6. The motion detection method based on event camera and laser radar according to claim 1, characterized in that: The Euler distance is calculated according to the following formula: Among them, (x i ,y i ) represents the coordinates of the currently selected i-th two-dimensional projection point; (x j ,y j ) represents the coordinates of any two-dimensional imaging point of the event camera or any two-dimensional projection point in the same distance range category as the i-th two-dimensional projection point; d(i, j) represents the Euler distance.

7. The motion detection method based on event camera and laser radar according to claim 1, characterized in that: The motion state detection result includes motion or stillness; The target number of the two-dimensional imaging point and the target number of the two-dimensional projection point are compared with a preset first point number threshold and a preset second point number threshold respectively to obtain a motion state detection result of the two-dimensional projection point, specifically including: When the target point number of the two-dimensional imaging point is greater than the first point number threshold, and the target point number of the two-dimensional projection point is greater than the second point number threshold, the motion state detection result of the two-dimensional projection point is motion.

8. The motion detection method based on event camera and laser radar according to claim 1, characterized in that: The first point threshold and the second point threshold are different according to different usage scenarios or usage environments.

9. A motion detection device based on an event camera and a laser radar, characterized in that: include: Configuration and calibration module, used to configure and calibrate event cameras and lidar; A data acquisition module is used to acquire a moving target to be detected, scan the moving target, and acquire multiple two-dimensional imaging points of the event camera and multiple three-dimensional point clouds of the laser radar within one scanning cycle; A data projection module, configured to project the multiple three-dimensional point clouds onto the plane where the event camera is located, to obtain multiple two-dimensional projection points of the laser radar; The distance clustering module is used to extract the distance data of the multiple two-dimensional projection points according to the farthest effective distance z of the laser radar. max and the preset distance separation threshold m, cluster the distance data to obtain z max / m distance range categories; a point accumulation module for traversing and calculating the Euler distance between each two-dimensional projection point in each distance range category and each of the two-dimensional imaging points of the event camera and each of the two-dimensional projection points of the lidar in the same distance range category; and accumulating the number of target points of the two-dimensional imaging point and the number of target points of the two-dimensional projection point when the Euler distance is less than a preset distance; a point number comparison module, configured to compare the target point number of the two-dimensional imaging point and the target point number of the two-dimensional projection point with a preset first point number threshold and a preset second point number threshold, respectively, to obtain a motion state detection result of the two-dimensional projection point and generate a corresponding motion state identifier; The result output module is used to repeat the steps of the above-mentioned point accumulation module to the point comparison module until the motion state identification of all two-dimensional projection points of the laser radar is generated, and the three-dimensional point cloud of the laser radar and the corresponding motion state identification are merged and output.

10. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 8.

Images