Motion target detection method, segmentation method and electronic device of event camera

Abstract

The application discloses a motion target detection method and segmentation method of an event camera and electronic equipment, and belongs to the technical field of event cameras; an event stream to be detected is projected to an image plane corresponding to the event camera, and a time graph is constructed, so that a two-dimensional global motion confidence graph is obtained after normalization; the two-dimensional global motion confidence graph can reflect the possibility of an independent motion target to a certain extent, threshold segmentation is performed on the two-dimensional global motion confidence graph, and the preliminary position of the motion target can be determined; an initial three-dimensional point cloud corresponding to the motion target is constructed, plane fitting is performed on the three-dimensional point cloud, and a fitted plane and points with a distance less than a preset distance from the plane are obtained; the fitted plane is used as a motion target point distribution plane, points with a distance less than the preset distance from the motion target point distribution plane in the three-dimensional point cloud are used as motion target points, and a motion target point cloud is formed, so that the detection of the motion target is realized. The application can accurately detect motion targets of any shape.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of event cameras, and more particularly, relates to an event camera motion target detection method, a segmentation method and an electronic device. BACKGROUND

[0002] An event camera is a new type of neuromorphic visual sensor. Unlike traditional frame cameras that capture images at a fixed frame rate, it asynchronously triggers events based on changes in pixel light intensity, has high dynamic range, high temporal resolution, low power consumption, and other advantages, and has wide application prospects in dynamic scene perception, high-speed obstacle avoidance, video surveillance and other fields. For example, in high dynamic scene perception, it is necessary to perceive and understand scene motion under complex lighting conditions such as daytime overexposure and nighttime low light; in the high-speed obstacle avoidance scene of a drone, it is necessary to quickly detect obstacles and respond. The key to the above tasks is to detect and segment independent motion targets from dynamic event streams. However, in the scene of a moving platform and a moving target, the background events generated by the camera's own motion are mixed with the events generated by the moving target, making it difficult to distinguish between them.

[0003] In existing event camera motion target detection methods, a common operation is to introduce an inertial measurement unit (IMU) into event camera motion segmentation, and use the background information provided by the IMU to compensate for the camera's own motion, thereby detecting independent motion targets. However, this method does not take into account the tailing effect, resulting in incomplete detection of the structure of the moving target. In order to solve the above problems, another event camera motion target detection method uses a three-dimensional tubular structure to complete the target structure, but this method can only complete regular targets, and there are still problems of incompleteness for irregular targets. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the present application provides an event camera motion target detection method, a segmentation method and an electronic device, which aims to accurately detect motion targets of any shape.

[0005] To achieve the above purpose, in a first aspect, the present application provides an event camera motion target detection method, comprising:

[0006] projecting a to-be-detected event stream to an image plane corresponding to the event camera, and constructing a time graph; the pixel value of each pixel point in the time graph is the average value of the time stamps of each event projected to the pixel point; performing normalization processing on the time graph to obtain a two-dimensional global motion confidence graph; wherein the to-be-detected event stream is an event stream obtained by performing motion compensation on the original event stream collected by the event camera;

[0007] The threshold segmentation is performed on the 2D global motion confidence map to obtain the preliminary position of the moving target, and the minimum connected domain where the preliminary position of the moving target is located is extracted from the 2D global motion confidence map; all the pixels in the connected domain are obtained with the connected domain as the boundary, and a 3D point cloud {(x i ,y i ,z i )} i=1,2,......,M ; M is the number of pixels in the connected domain; (x i ,y i ) and z i are the position of the i-th pixel in the connected domain and its pixel value in the two-dimensional global motion confidence map;

[0008] A plane fitting is performed on the three-dimensional point cloud to obtain the fitted plane and the points whose distance from the plane is less than a preset distance; the fitted plane is used as the moving target point distribution plane, and the points in the three-dimensional point cloud whose distance from the moving target point distribution plane is less than a preset distance are used as moving target points, and a moving target point cloud is formed, thereby realizing the detection of moving targets.

[0009] Further preferably, performing threshold segmentation on the two-dimensional global motion confidence map includes:

[0010] Construct a threshold segmentation image of the same size as the 2D global motion confidence map;

[0011] When the pixel value ρ of the pixel point in the i-th row and j-th column in the two-dimensional global motion confidence map ij When it is greater than the preset threshold, the pixel value of the pixel in the i-th row and j-th column in the threshold segmentation image is set to ρ ij , otherwise, set to 0; where i=1,2,......,H, j=1,2,......,W; H and W are the number of rows and columns of the two-dimensional global motion confidence map respectively;

[0012] The positions of pixels with non-zero values ​​in the threshold segmentation image are taken as the preliminary positions of the moving target;

[0013] The preset threshold is e|w|+f; e and f are both preset coefficients; and w is the average angular velocity of the event camera when collecting the original event stream.

[0014] Further preferably, the RANSAC algorithm is used to analyze the three-dimensional point cloud {(x i ,y i ,z i )} i=1,2,......,M Perform plane fitting.

[0015] Further preferably, the motion compensation on the raw event stream collected by the event camera comprises: obtaining a self-motion rotation matrix of the event camera; and substituting the self-motion rotation matrix into a geometric projection equation to perform spatio-temporal alignment on the raw event stream, to obtain the motion-compensated event stream.

[0016] Further preferably, the self-motion rotation matrix of the event camera is obtained by using an inertial measurement unit.

[0017] In a second aspect, the present application provides a motion target segmentation method of an event camera, comprising:

[0018] The motion target detection method provided in the first aspect of the present application is used to perform target detection on the event stream to be segmented, to obtain a motion target point cloud; wherein the event stream to be segmented is a motion-compensated event stream of a raw event stream collected by the event camera;

[0019] Events related to the motion target points in the motion target point cloud are extracted from the event stream to be segmented in time sequence, to form a target event stream;

[0020] The target event stream is aligned in space-time based on event motion parameters; the aligned target event stream is projected onto an image plane corresponding to the event camera, and a counting graph is constructed; the pixel value of each pixel in the counting graph is the number of events projected onto the pixel; wherein the event motion parameters comprise a motion displacement vector of each event in the target event stream;

[0021] An optimization objective comprising maximizing the gradient of the counting graph is constructed, and the event motion parameters are optimized based on the optimization objective, to obtain optimal event motion parameters and an optimal target event stream; the optimal target event stream is the target event stream aligned based on the optimal event motion parameters;

[0022] An event frame corresponding to the optimal target event stream is generated, and after binaryzation processing of the event frame, the event frame is taken as a motion target segmentation result.

[0023] Further preferably, the optimization objective further comprises: maximizing the similarity between the average motion displacement vector of the target event stream and a motion constraint vector;

[0024] The average motion displacement vector of the target event stream is the average value of the motion displacement vectors of the events in the target event stream;

[0025] The direction of the motion constraint vector is the opposite direction of the projection of the normal vector of the motion target point distribution plane on the xy plane of the point cloud three-dimensional coordinate system, and the size is ;

[0026] k is a preset constant; is the time interval from the first event to the last event in the target event stream; and a is the angle between the motion target point distribution plane and the xy plane in the three-dimensional coordinate system of the point cloud.

[0027] Further preferably, the motion target point distribution plane is ax+by+cz=d, wherein a, b, c, and d are all plane parameters. .

[0028] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to execute the method provided in the first aspect or the second aspect.

[0029] In a fourth aspect, the present application provides a computer readable storage medium, which comprises a stored computer program, wherein the computer program, when executed by a processor, controls a device where the storage medium is located to execute the method provided in the first aspect or the second aspect.

[0030] In a fifth aspect, the present application provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the method provided in the first aspect or the second aspect.

[0031] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0032] 1. The present application provides a motion target detection method for an event camera, which is based on the global distribution characteristics and motion consistency of a background event stream, and a two-dimensional global motion confidence map is constructed by using the motion-compensated event stream to complete background motion modeling. At the same time, the present application takes into account that in a three-dimensional space, the tailing effect of a motion target will form an event cluster with a planar distribution characteristic in a local area, regardless of the shape of the target body, the continuous motion trajectory of which presents a planar structure characteristic in a three-dimensional space. Therefore, by using the global motion confidence map and local plane fitting, the present application can not only eliminate background interference but also completely retain the motion trajectory shape of the target, and thus can achieve accurate detection of a motion target with any geometric shape.

[0033] 2、Further, in the motion target detection method of the event camera provided by the application, when the two-dimensional global motion confidence map is threshold segmented, the preset threshold is e|w|+f; since the background event is caused by the camera self-motion, the preset threshold is set as a dynamic threshold related to the camera self-motion angular velocity |w|, the segmentation threshold is dynamically adjusted by the size of the camera self-motion, the threshold segmentation can be dynamically performed according to the fast or slow degree of the camera self-motion, the threshold segmentation can be more accurately performed, and the preliminary position of the motion target can be accurately locked.

[0034] 3、Further, in the motion target detection method of the event camera provided by the application, the RANSAC algorithm is used to perform plane fitting on the three-dimensional point cloud {(x i ,y i ,z i )} i=1,2,......,M , compared with other plane fitting algorithms, the method has better robustness and more accurate results.

[0035] 4、Further, in the motion target detection method of the event camera provided by the application, the event stream to be detected is an event stream obtained by performing motion compensation on the original event stream collected by the event camera; in the motion compensation process, the self-motion rotation matrix of the event camera is obtained by using an inertial measurement unit, the acquisition speed is fast and the obtained parameters are more accurate, and the acquisition is not affected by the light of a complex scene.

[0036] 5、The application provides a motion target segmentation method of an event camera, which performs motion target segmentation based on position information and motion information of the motion target. Specifically, the motion target detection method provided by the first aspect of the application is used to perform target detection on the event stream to be segmented, accurate detection of the motion target of any shape is realized, and then the target event stream is obtained from the event stream to be segmented; based on the event motion parameters, the target event stream is aligned in space and time, and the aligned target event stream is projected onto the image plane corresponding to the event camera to construct a counting graph; since the sharper the target edge structure is, the more significant the counting graph generated by the corresponding event accumulation is in the gradient amplitude, and there is a positive correlation between this strong gradient feature and the edge sharpness. Therefore, the target edge is segmented by maximizing the gradient of the counting graph to determine the optimal event motion parameters; the process of maximizing the counting graph gradient is essentially searching for the optimal motion compensation parameters that can best enhance the contrast of the target edge in the parameter space. When the gradient response reaches the global maximum, it means that the space-time accumulation form of the event stream is most consistent with the real motion trajectory of the target, at this time, the sharpness of the target edge contour in the counting graph is the largest, and thus the accurate reconstruction and complete segmentation of the target edge structure are realized.

[0037] 6、Further, the motion target segmentation method of the event camera provided by the application considers that the average motion displacement vector of the target event stream to some extent reflects the real motion trajectory of the target. The application maximizes the gradient of the counting graph and also maximizes the similarity between the average motion displacement vector of the target event stream and the motion constraint vector, so that the estimated event motion parameters are as consistent as possible with the real motion trajectory of the target, which can improve the convergence speed under the premise of ensuring the accuracy of target edge segmentation and make the target edge contour edge sharp. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The motion target detection method flowchart of the event camera provided by the embodiment of the application.

[0039] Figure 2 The two-dimensional global motion confidence map provided by the embodiment of the application.

[0040] Figure 3 The result map after threshold segmentation of the two-dimensional motion confidence map provided by the embodiment of the application.

[0041] Figure 4 The three-dimensional point cloud schematic diagram provided by the embodiment of the application.

[0042] Figure 5 The plane extraction result schematic diagram provided by the embodiment of the application.

[0043] Figure 6 The motion target segmentation flowchart of the event camera provided by the embodiment of the application.

[0044] Figure 7 The first original event frame of the event camera provided by the embodiment of the application.

[0045] Figure 8 The motion target segmentation result schematic diagram corresponding to the first original event frame of the event camera provided by the embodiment of the application.

[0046] Figure 9 The second original event frame provided by the embodiment of the application.

[0047] Figure 10 The motion target segmentation result schematic diagram corresponding to the second original event frame of the event camera provided by the embodiment of the application. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0049] In order to achieve the above-mentioned purpose, in a first aspect, as shown in the specification, Figure 1 the present application provides a moving target detection method of an event camera, comprising:

[0050] projecting a to-be-detected event stream to an image plane corresponding to the event camera, and constructing a time graph; the pixel value of each pixel point in the time graph is the average value of the time stamps of each event projected to the pixel point; performing normalization processing on the time graph to obtain a two-dimensional global motion confidence graph; wherein the to-be-detected event stream is an event stream obtained by performing motion compensation on an original event stream collected by the event camera;

[0051] performing threshold segmentation on the two-dimensional global motion confidence graph to obtain a preliminary position of a moving target, and extracting a minimum connected domain in which the preliminary position of the moving target is located in the two-dimensional global motion confidence graph; taking the connected domain as a boundary, obtaining all pixel points therein, and constituting a three-dimensional point cloud {(x i ,y i ,z i )} i=1,2,......,M ; M is the number of pixel points in the connected domain; (x i ,y i ) and z i are the position of the i-th pixel point in the connected domain and the pixel value thereof in the two-dimensional global motion confidence graph, respectively;

[0052] performing plane fitting on the three-dimensional point cloud to obtain a fitted plane and points with a distance less than a preset distance from the plane; taking the fitted plane as a moving target point distribution plane, taking points with a distance less than a preset distance from the moving target point distribution plane in the three-dimensional point cloud as moving target points, and constituting a moving target point cloud, thereby realizing detection of the moving target.

[0053] It should be noted that the present invention models the motion characteristics of the target and background from both global and local perspectives. The background event stream generated by the self-motion of the camera is globally distributed and has consistent motion. Therefore, by constructing a two-dimensional global motion confidence map, the background event stream can be modeled and eliminated by threshold segmentation. The moving target is in a local area. The constructed two-dimensional global motion confidence map is expanded to three dimensions. The target event stream is distributed in the local area and exhibits a slope characteristic. The target can be extracted by plane fitting. Therefore, the global motion characteristics used to model the background event stream are reflected in the construction of a two-dimensional global motion confidence map, while the local motion characteristics used to model the target event stream are reflected in the plane fitting of the local area where the target is located.

[0054] Considering that the background event stream is globally distributed and has motion consistency, after motion compensation (the event stream to be detected in the present invention is the event stream after motion compensation of the original event stream collected by the event camera), the motion characteristics of the background event stream can be modeled by constructing a two-dimensional global motion confidence map. When the two-dimensional motion confidence map is expanded to three dimensions, the target happens to be a planar structure, so the target can be detected and extracted by fitting the plane. It is precisely because the target has a tailing effect that it presents a planar distribution in three-dimensional space. Regardless of the shape of the target, the planar distribution of the target in three-dimensional space is caused by tailing, which has nothing to do with the shape of the target. The present invention extracts the target through plane fitting, which can completely extract the event stream belonging to the target, thereby retaining the complete structure of the target. Therefore, it is possible to detect moving targets of any shape.

[0055] It should be noted that there are many methods for performing threshold segmentation on the two-dimensional global motion confidence map, such as global threshold segmentation, multi-threshold segmentation, adaptive threshold segmentation, etc. In an optional implementation, performing threshold segmentation on the two-dimensional global motion confidence map includes:

[0056] Construct a threshold segmentation image of the same size as the 2D global motion confidence map;

[0057] When the pixel value ρ of the pixel point in the i-th row and j-th column in the two-dimensional global motion confidence map ij When it is greater than the preset threshold, the pixel value of the pixel in the i-th row and j-th column in the threshold segmentation image is set to ρ ij , otherwise, set to 0; where i=1,2,......,H, j=1,2,......,W; H and W are the number of rows and columns of the two-dimensional global motion confidence map respectively;

[0058] The positions of pixels with non-zero values ​​in the threshold segmentation image are taken as the preliminary positions of the moving target;

[0059] Wherein, the preset threshold is e|w|+f; e and f are both preset coefficients; w is the average value of angular velocity of the event camera when collecting the original event stream.

[0060] It should be noted that the method of plane fitting for the three-dimensional point cloud {(x i ,y i ,z i )} i=1,2,......,M is various, such as RANSAC algorithm, least square method, principal component analysis (PCA) method, etc., which is not limited here. Preferably, the present application adopts RANSAC algorithm to perform plane fitting on the three-dimensional point cloud {(x i ,y i ,z i )} i=1,2,......,M , which has better robustness.

[0061] It should be noted that the way of motion compensation for the original event stream collected by the event camera is various;

[0062] In an optional implementation, the above motion compensation for the original event stream collected by the event camera includes: obtaining a self-motion rotation matrix of the event camera; substituting the self-motion rotation matrix into a geometric projection equation to perform space-time alignment on the original event stream, to obtain the motion-compensated event stream; wherein, the self-motion rotation matrix is constructed based on the motion parameters of the event camera; the motion parameters of the event camera include: the acceleration and angular velocity of the three axes in the event camera coordinate system; the method of obtaining the motion parameters of the event camera is various, which can be estimated by an optimization method, or directly measured by an inertial measurement unit (IMU) (this way is faster and the obtained parameters are more accurate). It should be noted that the inertial measurement unit (IMU) can be set on the event camera, or set outside the event camera, which is not limited here.

[0063] In another optional implementation, the mobile displacement can also be estimated by an optimization method, and then motion compensation is performed, but compared with the optional implementation, this way is slower.

[0064] In a second aspect, the present application provides a motion target segmentation method of an event camera, which includes:

[0065] The motion target detection method provided in the first aspect of the present application is adopted to perform target detection on the event stream to be segmented, to obtain a motion target point cloud; wherein, the event stream to be segmented is the event stream after motion compensation for the original event stream collected by the event camera;

[0066] Events related to the motion target points in the motion target point cloud are extracted from the event stream to be segmented according to time sequence, to form a target event stream;

[0067] align the target event stream in time and space based on the event motion parameters; project the aligned target event stream to the image plane corresponding to the event camera, and construct a counting graph; the pixel value of each pixel point in the counting graph is the number of events projected to the pixel point; wherein the event motion parameters include the motion displacement vector of each event in the target event stream;

[0068] construct an optimization target including maximizing the gradient of the counting graph, and optimize the event motion parameters based on the optimization target to obtain optimal event motion parameters and an optimal target event stream; the optimal target event stream is the target event stream aligned based on the optimal event motion parameters;

[0069] generate an event frame corresponding to the optimal target event stream, and after binarization processing of the event frame, use it as the motion target segmentation result.

[0070] Preferably, in an optional embodiment, the optimization target further includes: maximizing the similarity between the average motion displacement vector of the target event stream and the motion constraint vector;

[0071] wherein the average motion displacement vector of the target event stream is the average value of the motion displacement vectors of the events in the target event stream;

[0072] The direction of the motion constraint vector is the opposite direction of the projection of the normal vector of the motion target point distribution plane in the xy plane of the point cloud three-dimensional coordinate system, and the size is ;

[0073] k is a preset constant; is the time interval from the first event to the last event in the target event stream; α is the angle between the motion target point distribution plane and the xy plane in the point cloud three-dimensional coordinate system; the motion target point distribution plane is obtained by target detection on the event stream to be segmented using the motion target detection method provided in the first aspect of the application.

[0074] In an optional embodiment, the motion target point distribution plane is: ax+by+cz=d; wherein a, b, c, d are all plane parameters. .

[0075] The related technical solutions are the same as the motion target detection method provided in the first aspect of the application, and will not be repeated here.

[0076] In order to further illustrate the event camera motion target detection method and the motion target segmentation method provided by the application, specific embodiments will be described in detail below:

[0077] Embodiment 1

[0078] This embodiment provides a method for detecting moving targets using an event camera. The method obtains the target's position and motion information by analyzing and mining the two-dimensional and three-dimensional motion features of the moving target. The method specifically includes the following steps:

[0079] 1) Device calibration of event camera and inertial measurement unit

[0080] This example uses a hand-eye calibration method to calibrate the external parameters of the event camera and inertial measurement unit (IMU), determining the transformation relationship between the camera coordinate system and the IMU coordinate system. First, the event stream output by the event camera is synchronously recorded with the IMU angular velocity and acceleration data to ensure timestamp alignment. The event stream is accumulated at fixed time intervals from the IMU to generate corresponding event frames.

[0081] Estimating the rotation increment of the event camera using consecutive event frames and translation increments Similarly, the corresponding rotation increment is obtained by pre-integrating the IMU data and translation increments . Assume that the external parameter from IMU to camera is T IC ={R IC , T IC}, according to the hand-eye calibration method, the following formula is satisfied:

[0082]

[0083] in, , , through nonlinear optimization, minimize the camera reprojection error and IMU prediction error to obtain the accurate IMU to event camera external parameter rotation matrix R IC With the translation matrix T IC .

[0084] 2) IMU-based background event stream motion compensation

[0085] In event motion detection and segmentation, events are generated not only by moving targets but also by their own background motion. The IMU measures this background motion information. Therefore, the IMU can be used to perform spatiotemporal alignment of background events. Specifically, the IMU obtains six-degree-of-freedom motion parameters, solves and derives the event camera's self-motion rotation matrix, and then uses geometric projection equations to perform spatiotemporal alignment of the original event stream.

[0086] Specifically, given an event stream E={e t |e t = (x, p, t)} and the corresponding angular velocity w t , first, calculate the average angular velocity during this period:

[0087]

[0088] Among them, ∆t' is the time interval from the first event to the last event in the event stream. Therefore, the rotational motion matrix R relative to the three axes of the reference time is calculated according to the Rodriguez formula:

[0089]

[0090] Among them, R x (·), R y (·), R z (·) is the three-axis Rodriguez formula. According to the geometric projection equation, the event stream is motion compensated to achieve the spatiotemporal alignment of the background events:

[0091] x C =KRK -1 x

[0092] Among them, K is the camera internal parameter, x C =(x C , y C ) is the event coordinate after compensation.

[0093] 3) Constructing a 2D global motion confidence map

[0094] After motion compensation, the event stream is projected onto the two-dimensional image plane corresponding to the event camera. First, the number of superpositions of events at different pixel positions is counted:

[0095]

[0096] in, is the set of events with coordinates at (i, j), and E' is the compensated event stream. The average timestamp of the event points at each pixel position is counted to construct a two-dimensional global motion confidence map:

[0097]

[0098] The average timestamp is then normalized to obtain the motion confidence value of each pixel:

[0099]

[0100] in, For all T ij The average value of The time interval between the first event and the last event in the event stream.

[0101] The two-dimensional global motion confidence map obtained in this embodiment is as follows: Figure 2 shown.

[0102] 4) Adaptive threshold segmentation

[0103] After obtaining the two-dimensional motion confidence map, the confidence value can reflect the possibility of the independent motion target to a certain extent, so adaptive threshold segmentation is performed to obtain the position of the independent motion target. Therefore, the adaptive threshold is λ(w) = e|w| + f; e and f are both preset coefficients, e is a weight value of the adaptive angular velocity, and f adjusts the classification threshold; w is the average value of the angular velocity of the event camera when collecting the original event stream. In the embodiment, e and f take values of 0.1 and 0.6 respectively.

[0104] Threshold segmentation is performed on the two-dimensional motion confidence map to obtain a target region.

[0105]

[0106] The result of threshold segmentation on the two-dimensional motion confidence map in the embodiment is shown in FIG. 6. Figure 3

[0107] 5) Target motion slope extraction

[0108] According to the target position coordinates obtained by adaptive threshold segmentation, a minimum connected domain η ij containing the complete structure of the target is obtained by a connected domain algorithm, as shown in FIG. 7. The connected domain is taken as a boundary to obtain a three-dimensional point cloud set in the connected domain: Figure 4

[0109]

[0110] The motion target presents a slope structure in the three-dimensional confidence point cloud, and a plane geometric model is constructed using the following formula: ax + by + cz = d; wherein a, b, c, and d are all plane parameters; (a, b, c) is the normal vector of the plane; and d is the distance from the origin of the point cloud three-dimensional coordinate system to the plane.

[0111] RANSAC algorithm is used to randomly sample three points in the three-dimensional point cloud set obtained above, to solve the plane parameters according to the plane formula, and to solve the distance of other points to the plane according to the following formula, so as to divide the inliers and outliers corresponding to the plane:

[0112]

[0113] wherein (x0, y0, z0) is any non-sampling point in the three-dimensional point cloud set. A distance threshold (preset distance) is given, and if the distance of other points to the plane is less than the preset distance, the other points are considered as inliers, and otherwise, the other points are considered as outliers. The above process is repeated, and the plane model with the largest number of inliers is solved according to the following formula:

[0114] ​​

[0115] where I(·) is an indicator function, outputting 1 if the condition within the function is true, and 0 otherwise. The distance threshold in this embodiment is 0.03.

[0116] As Figure 5 shown is a plane extraction result schematic diagram provided by this embodiment.

[0117] The inlier points are taken as the motion target points, and a motion target point cloud is formed, so as to realize detection of the motion target.

[0118] Embodiment 2

[0119] This embodiment provides a motion target segmentation method of an event camera. After the position information and motion information of the target are acquired by analyzing and mining the motion characteristics of the motion target in two dimensions of two and three dimensions in the motion target detection in embodiment 1, the contrast maximization framework is used to realize accurate positioning and complete segmentation of the target edge. The whole process is shown in Figure 6 . Specifically, it includes:

[0120] The contrast maximization framework is used to segment the motion target completely and accurately, and a three-dimensional plane constraint contrast maximization optimization model is constructed. The event stream set belonging to the inlier points is obtained as , as the target event stream; wherein x k is the pixel coordinate of the kth event in the target event stream, t k is the timestamp of the kth event in the target event stream, p k is the polarity change direction of the kth event in the target event stream; N' is the number of events in the target event stream. The target event stream can be aligned in space-time through the event motion parameter θ, as shown in the following formula:

[0121]

[0122] where x k ' is the position of the event after the target event stream is aligned in space-time; the event motion parameter θ includes the motion displacement vector of each event in the target event stream.

[0123] The target event stream after alignment is projected on a two-dimensional plane to obtain a projection image, i.e., a counting graph, as shown in the following formula:

[0124]

[0125] x represents any pixel coordinate in the image plane corresponding to the event camera; δ is the Dirac function.

[0126] Maximizing the contrast as the optimization objective, i.e. maximizing the gradient of the counting map by an optimization iterative algorithm, the gradient of the counting map can be solved as follows:

[0127]

[0128] Wherein, Ω is the area of the non-zero position of the counting map.

[0129] The embodiment further adds the target motion information knowledge obtained by the slope into the target function, helps the target function to converge faster, and prevents the target function from overfitting:

[0130] The above optimization objective further includes: maximizing the similarity between the average motion displacement vector of the target event stream and the motion constraint vector; in the embodiment, it is represented as:

[0131]

[0132] Wherein, the average motion displacement vector θ' of the target event stream is the average value of the motion displacement vectors of each event in the target event stream, and is obtained based on the event motion parameter θ.

[0133] The direction of the motion constraint vector θ0 is the opposite direction of the projection of the normal vector of the motion target point distribution plane in the xy plane of the point cloud three-dimensional coordinate system, and the size is ;

[0134] K is a preset constant, and in the embodiment, the value is 0.3; is the time interval from the first event to the last event in the target event stream; α is the angle between the motion target point distribution plane and the xy plane in the point cloud three-dimensional coordinate system; the motion target point distribution plane is obtained by performing target detection on the to-be-segmented event stream by using the motion target detection method provided in the embodiment 1, and specifically, the motion target point distribution plane is ax+by+cz=d; wherein, a, b, c and d are all plane parameters; .

[0135] Based on this, the total optimization objective in the embodiment is:

[0136]

[0137] Wherein, λ is a preset weight value, and in the embodiment, the value is 0.5.

[0138] The embodiment uses the Gauss-Newton method to iteratively optimize the target function, and obtains the motion target with clear edge contour structure.

[0139] As shown in Figure 7 and Figure 8 , they are respectively the first original event frame of the event camera and the corresponding motion target segmentation result schematic diagram provided in the embodiment. As shown inFigure 9 and Figure 10 Fig. 2 and Fig. 3 respectively show a second raw event frame of an event camera and a corresponding moving object segmentation result according to the embodiment.

[0140] The embodiment adopts an event camera moving object segmentation method based on spatiotemporal motion saliency, extracts motion features in two dimensions and three dimensions of the moving object, obtains position information and motion information of the object, and realizes accurate positioning and complete segmentation of the object edge by using a contrast maximization framework.

[0141] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the method provided in the first aspect or the second aspect of the present application.

[0142] The related technical solutions are the same as the method provided in the first aspect or the second aspect of the present application, and will not be repeated here.

[0143] In a fourth aspect, the present application further provides a computer readable storage medium, which comprises a stored computer program, wherein when the computer program is run by a processor, the device where the storage medium is located executes the method provided in the first aspect or the second aspect of the present application.

[0144] The related technical solutions are the same as the method provided in the first aspect or the second aspect of the present application, and will not be repeated here.

[0145] In a fifth aspect, the present application further provides a computer program product, comprising computer programs / instructions, which are executed by a processor to implement the method provided in the first aspect or the second aspect of the present application.

[0146] The related technical solutions are the same as the method provided in the first aspect or the second aspect of the present application, and will not be repeated here.

[0147] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A moving target detection method for an event camera, characterized in that: include: Project the event stream to be detected onto the image plane corresponding to the event camera and construct a time graph; The pixel value of each pixel in the time map is the average of the timestamps of the events projected onto the pixel; the time map is normalized to obtain a two-dimensional global motion confidence map; the event stream to be detected is the event stream after motion compensation of the original event stream collected by the event camera; The threshold segmentation is performed on the two-dimensional global motion confidence map to obtain the preliminary position of the moving target, and the minimum connected domain where the preliminary position of the moving target is located is extracted from the two-dimensional global motion confidence map; all the pixels in the connected domain are obtained with the connected domain as the boundary, and a three-dimensional point cloud {(x i ,y i ,z i )} i=1,2,......,M ; M is the number of pixels in the connected domain; (x i ,y i ) and z i are respectively the position of the i-th pixel point in the connected domain and its pixel value in the two-dimensional global motion confidence map; The three-dimensional point cloud is plane-fitted to obtain a fitted plane and points whose distance from the plane is less than a preset distance; the fitted plane is used as a moving target point distribution plane, and points in the three-dimensional point cloud whose distance from the moving target point distribution plane is less than a preset distance are used as moving target points, and a moving target point cloud is formed, thereby realizing the detection of moving targets.

2. The moving target detection method according to claim 1, wherein: The threshold segmentation of the two-dimensional global motion confidence map includes: Construct a threshold segmentation image of the same size as the 2D global motion confidence map; When the pixel value ρ of the pixel point in the i-th row and j-th column in the two-dimensional global motion confidence map ij When it is greater than the preset threshold, the pixel value of the pixel in the i-th row and j-th column in the threshold segmentation image is set to ρ ij , otherwise, set to 0; where i=1,2,......,H, j=1,2,......,W; H and W are the number of rows and columns of the two-dimensional global motion confidence map respectively; The positions of pixels with non-zero values ​​in the threshold segmentation image are taken as the preliminary positions of the moving target; The preset threshold is e|w|+f; e and f are both preset coefficients; and w is the average angular velocity of the event camera when collecting the original event stream.

3. The moving target detection method according to claim 1, wherein: The RANSAC algorithm is used to quantize the three-dimensional point cloud {(x i ,y i ,z i )} i=1,2,......,M Perform plane fitting.

4. The moving target detection method according to any one of claims 1 to 3, characterized in that: The motion compensation of the original event stream captured by the event camera includes: obtaining the self-motion rotation matrix of the event camera; substituting the self-motion rotation matrix into the geometric projection equation to perform spatiotemporal alignment on the original event stream to obtain the motion-compensated event stream.

5. The moving target detection method according to claim 4, wherein: The inertial measurement unit is used to obtain the ego-motion rotation matrix of the event camera.

6. A moving target segmentation method for an event camera, characterized in that: include: Using the moving target detection method according to any one of claims 1 to 5, target detection is performed on the segmented event stream to obtain a moving target point cloud; The event stream to be segmented is an event stream obtained by performing motion compensation on the original event stream captured by the event camera; Extract events related to the moving target points in the moving target point cloud from the event stream to be segmented in time sequence to form a target event stream; Align the target event stream in time and space based on event motion parameters; Project the aligned target event stream onto the image plane corresponding to the event camera and construct a count map; The pixel value of each pixel in the counting map is the number of events projected onto the pixel; the event motion parameter includes a motion displacement vector of each event in the target event stream; Constructing an optimization objective including maximizing the gradient of the count graph, and optimizing the event motion parameters based on the optimization objective to obtain optimal event motion parameters and an optimal target event stream; the optimal target event stream is the target event stream aligned based on the optimal event motion parameters; An event frame corresponding to the optimal target event stream is generated, and the event frame is binarized to obtain a moving target segmentation result.

7. The moving object segmentation method according to claim 6, characterized in that: The optimization goal also includes: maximizing the similarity between the average motion displacement vector of the target event stream and the motion constraint vector; The average motion displacement vector is the average value of the motion displacement vectors of each event in the target event stream; The direction of the motion constraint vector is the opposite direction of the projection of the normal vector of the motion target point distribution plane on the xy plane of the point cloud three-dimensional coordinate system, and its magnitude is ; k is a preset constant; is the time interval from the first event to the last event in the target event stream; α is the angle between the moving target point distribution plane and the xy plane in the three-dimensional coordinate system of the point cloud; the moving target point distribution plane is obtained by performing target detection on the event stream to be segmented using the moving target detection method described in any one of claims 1 to 5.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instruction, which implements the method according to any one of claims 1 to 7 when executed by a processor.

Images