Method for spatial motion segmentation fusing depth estimation and three-dimensional motion compensation

Abstract

The embodiment of the application relates to a spatial motion segmentation method fusing depth estimation and three-dimensional motion compensation, and compared with a traditional frame-based and plane-based motion segmentation algorithm, the spatial motion segmentation method based on an event camera increases the accuracy and segmentation efficiency of the algorithm. In the absence of other sensing devices such as an RGB-D camera, an event data dimension upgrading model realizes event dimension upgrading, reduces the cost of event 3D to 4D, and is simple to operate. PSO is adopted to optimize theta, the global optimization effect of the algorithm is strengthened, good local search capability is considered, and the dependence of the algorithm on initialization is reduced.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a spatial motion segmentation method that integrates depth estimation and 3D motion compensation. Background Technology

[0002] In motion segmentation, individual motion-related pixels can be separated from a video sequence. Foreground objects are separated from the background by clustering these pixels according to different motion types. Traditional vision-based processing algorithms typically assume that the video was filmed under ideal conditions; they do not consider many factors (e.g., movement speed, stability, and lighting conditions).

[0003] In real-world applications, the performance of motion segmentation algorithms is severely impacted by fast-moving mobile devices such as drones and autonomous vehicles. With the advancement of science and technology and the increasing complexity of manufacturing processes, many fields are demanding higher accuracy and stability in feature detection and tracking.

[0004] Therefore, the limitations of 2D detection have gradually become apparent. With the help of depth estimation, more accurate detection can be achieved using 3D information while fully considering 2D information. When detecting and tracking a scene, all information is contained within its spatiotemporal context. However, tracking performance is affected by a large amount of redundant background information; only the region where the target is located is valuable. Summary of the Invention

[0005] The purpose of this invention is to provide a spatial motion segmentation method that integrates depth estimation and 3D motion compensation, which can meet the accuracy and stability requirements of visual tasks, separate the target from the redundant background, greatly reduce the amount of information to be processed, and improve the performance of visual tasks.

[0006] The technical solution of the present invention is as follows:

[0007] To achieve the above objectives, this invention provides a spatial motion segmentation method that integrates depth estimation and 3D motion compensation, comprising the following steps:

[0008] S1: The event camera simulator ESIM is used to preprocess the video sequence and establish an event data upscaling model, including depth estimation, linear interpolation and coordinate normalization, to realize the 3D to 4D transformation of the event;

[0009] S2: Calculate the motion flow of the event point cloud, cluster it, initialize the motion parameters θ, perform 3D motion compensation distortion processing on the events, accumulate the weights of the distorted events to obtain VWE, and get the initial motion segmentation result;

[0010] S3: PSO is used to update θ. By iterating θ and P, the image contrast of VWE is maximized, and the result is saved to complete the spatial motion segmentation.

[0011] Furthermore, step S1 above is specifically as follows:

[0012] S101: Taking the video sequence as input, the high frame rate video sequence is first broken down into individual frames of images. Then, these individual frames, along with the camera's motion trajectory, are fed into the event camera simulator ESIM. Finally, an asynchronous event stream is output.

[0013]

[0014] Among them, e k Let x represent the k-th event, (x k ,y k ) represents the coordinates of the event, t k Indicates the triggering event, N e Indicates the number of events;

[0015] S102: Depth estimation; Select keyframes from the output event stream, which consists of approximately 15,000 events; Use these keyframes as input to feed them into a deep learning-based depth estimation network to obtain the depth map of the keyframes.

[0016] S103: Linear interpolation; For the same pixel, assuming that the depth change of that pixel between two keyframes is linear, the pseudo depth of each event can be calculated through linear interpolation.

[0017]

[0018] Among them, Z k It is the depth value of the k-th event, t k This represents the time of the k-th event. This represents the time of the i-th keyframe. Indicates the depth of the i-th keyframe;

[0019] S103: Coordinate Normalization; The x, y, and Z coordinates of an event obtained through linear interpolation are not in the same coordinate system. Coordinate normalization can unify the event coordinates into a global coordinate system. The formula is as follows:

[0020]

[0021] Where f is a scaling factor, which can be freely adjusted to determine the size of the pseudo-depth.

[0022] Rewriting the above equation in matrix form gives us...

[0023]

[0024] Furthermore, step S2 above is specifically as follows:

[0025] S201: Motion flow estimation of event point clouds; In a two-dimensional plane, optical flow represents the instantaneous velocity of pixels in two dimensions; Analogous to optical flow, in three-dimensional space, motion flow represents the instantaneous velocity of the point cloud in three dimensions, and is used in three-dimensional motion compensation; The grayscale invariance assumption means that the grayscale value remains unchanged from time t1 to time t2, as shown below:

[0026]

[0027] in, It is the grayscale value at time i;

[0028] Therefore, from time t to time t+dt, we have

[0029] I(x+dx,y+dy,z+dz,t+dt)=I(x,y,z,t) (6)

[0030] Expanding the left side of the above equation using Taylor and retaining the first-order terms, we get...

[0031]

[0032] Combining equations (6) and (7), we have

[0033]

[0034] Dividing both sides of the above equation by the phase shift, we have:

[0035]

[0036] Where, θ x ,θ y ,θ z These represent the velocities in the x, y, and z directions, respectively.

[0037] To establish sufficient formulas to estimate θ, we establish three planes S1, S2, and S3 to represent the projections of the object onto (x,y,t), (y,z,t), and (z,x,t) respectively, and denote their gray values ​​as I1, I2, and I3; therefore, equation (9) can be written as

[0038]

[0039] Where (i,j)∈{(x,y),(y,z),(z,x)}, k is the index of the set; for example, when k=1, (i,j)=(x,y);

[0040] Writing equation (10) in matrix form, we have:

[0041]

[0042] Among them, I 1,x This represents the partial derivative of I1 with respect to x when k=1;

[0043] I k,(i,j) Let it be a k ,b k The above formula can be written as

[0044]

[0045] By calculating the optical flow, we can obtain θ x The motion flow can be represented as

[0046]

[0047] The calculated θ is clustered using the k-means algorithm to find cluster centers, with each cluster center representing a type of motion.

[0048] S202: 3D Motion Compensation and VWE; Distorting events along the direction of motion flow will make the VWE clearer, and vice versa; Through the above steps, we obtained the initial value of θ, and distorting the events has…

[0049] x k =(x k ,y k ,z k ) T (14)

[0050] x' k =x k -(t k -t ref )θ=x k -Δt k θ (15)

[0051] Where, x k Let x' represent the position before the k-th event is distorted. k Indicates the position after distortion, t ref Indicates the reference time to which the event is distorted;

[0052] The VWE is defined by weighting the distorted events.

[0053]

[0054] Among them, V j Represents the VWE of the j-th motion flow;

[0055] Image contrast represents the sharpness of an image; the higher the contrast, the clearer the image. Variance is used to evaluate contrast.

[0056]

[0057] in, V represents j The mean of Ω represents the image volume.

[0058] Furthermore, step S3 above is specifically as follows:

[0059] S301: Initialize parameter P to ensure that the probability of each event being assigned to each motion type is equal initially.

[0060]

[0061] Where, N l It is the number of clusters;

[0062] S302: θ is updated using PSO, and the particle velocity is updated as follows.

[0063]

[0064] Where i represents the i-th particle, d represents the dimension of the particle, w is the inertia factor (the larger the value, the wider the range the particle can fly, the stronger the global search capability, but the weaker the local capability), p i ,p g c1 and c2 represent the global and local optimal positions, respectively, c1 and c2 are non-negative constants, and r1 and r2 are random numbers belonging to [0,1].

[0065] The particle update equation is as follows:

[0066]

[0067] Here, α is a constraint factor, the purpose of which is to control the weight of the speed;

[0068] The update of P is represented as follows:

[0069]

[0070] Where, p kj c represents the probability that the k-th event belongs to the j-th motion stream. i Indicates the i-th VWE;

[0071] S303: Continuously iterate over P and θ to maximize the contrast of VWE and obtain the final output segmentation result.

[0072] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0073] 1) Compared with traditional frame-based and plane-based motion segmentation algorithms, the spatial motion segmentation method of this application increases the accuracy and segmentation efficiency of the algorithm.

[0074] 2) Without using other sensing devices such as RGB-D cameras, the present invention realizes the dimensionality upgrade of events through the event data upgrade model, reducing the cost of upgrading events from 3D to 4D and simplifying the operation.

[0075] 3) This invention uses PSO to optimize θ, which enhances the global optimization effect of the algorithm while also taking into account good local search capabilities, and reduces the algorithm's dependence on initialization. Attached Figure Description

[0076] Figure 1 This is a flowchart of the present invention;

[0077] Figure 2 The linear interpolation method of this invention obtains a pseudo-depth map for each event;

[0078] Figure 3 This invention uses global coordinate transformation to normalize the coordinates of the graph.

[0079] Figure 4 This is an assumption diagram of the invariance of grayscale during object motion in space, as presented in this invention.

[0080] Figure 5 It is a projection of the object of the present invention onto a plane;

[0081] Figure 6 This is the original image of the segmented scene according to the present invention;

[0082] Figure 7 This is a segmentation result diagram obtained by twisting the motion flow along the upper left corner of the event in this invention;

[0083] Figure 8 This is a segmentation result diagram obtained by twisting the motion flow along the lower right corner of the event in this invention. Detailed Implementation

[0084] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0085] Step 1: Establish a spatial motion segmentation method model that integrates depth estimation and 3D motion compensation, such as... Figure 1 As shown.

[0086] First, the video sequence is preprocessed using the event camera simulator ESIM to establish an event data upscaling model, including depth estimation, linear interpolation, and coordinate normalization, to realize the 3D to 4D transformation of the event; next...

[0087] Then, the motion flow of the event point cloud is calculated, the event is clustered, and the motion parameters θ are initialized. After the events are subjected to 3D motion compensation distortion processing, the distorted events are weighted and accumulated to obtain VWE, thus obtaining the initial motion segmentation result.

[0088] Finally, PSO is used to update θ, and the VWE image contrast is maximized by iterating between θ and P. The result is then saved to complete the spatial motion segmentation.

[0089] Step 2: First, take the video sequence as input, break the high frame rate video sequence into individual frames, then input the broken-down frames and the camera's motion trajectory into the event camera simulator ESIM, and finally output an asynchronous event stream.

[0090] Secondly, the output event stream is divided into sets of approximately 15,000 events. Keyframes are selected and used as input to a deep learning-based depth estimation network to obtain the depth map of the keyframes.

[0091] like Figure 2 As shown, for the same pixel, assuming that the depth change of the pixel between two keyframes is linear, the pseudo depth of each event can be calculated by linear interpolation.

[0092]

[0093] However, the event coordinates x, y and Z obtained through linear interpolation are not in the same coordinate system. Coordinate normalization can unify the event coordinates into a global coordinate system, such as... Figure 3 As shown.

[0094]

[0095] Step 3: Based on the assumption of constant grayscale, as follows Figure 4 As shown, a spatial motion flow model is established, through... Figure 5 The motion flow is calculated and clustered, and then the parameter θ is initialized.

[0096]

[0097] By using motion compensation to distort events along different motion flow directions, preliminary segmentation results can be obtained.

[0098] x' k =x k -(t k -t ref )θ=x k -Δt k θ

[0099]

[0100] Step 4: To achieve the clearest segmentation result, i.e., to maximize the image contrast of VWE, it is necessary to update and iterate θ and P. In this invention, image contrast is evaluated using variance; that is, the larger the contrast (variance), the clearer the image.

[0101]

[0102] θ is updated using PSO.

[0103]

[0104] The above equation is the velocity update equation for particles.

[0105]

[0106] The above equation is the particle renewal equation.

[0107] Update P using equation (21).

[0108]

[0109] Figure 6 The original scene to be segmented. Figure 7 and Figure 8 This is the result of dividing the "car" along different directions, such as the top left corner and the bottom right corner.

[0110] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0111] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the appended claims.

Claims

1. A spatial motion segmentation method integrating depth estimation and 3D motion compensation, characterized in that, include: S1: The event camera simulator ESIM is used to preprocess the video sequence and establish an event data upscaling model, including depth estimation, linear interpolation and coordinate normalization, to realize the 3D to 4D transformation of the event; The specific steps of S1 are as follows: S101: Taking the video sequence as input, the high frame rate video sequence is first broken down into individual frames of images. Then, these individual frames, along with the camera's motion trajectory, are fed into the event camera simulator ESIM. Finally, an asynchronous event stream is output. (1） in, Indicates the first One event, Represents the coordinates of the event. Indicates the event that triggered the event. Indicates the number of events; S102: Depth estimation; Select keyframes from the output event stream, which consists of approximately 15,000 events; Use these keyframes as input to feed them into a deep learning-based depth estimation network to obtain the depth map of the keyframes. S103: Linear interpolation; For the same pixel, assuming that the depth change of that pixel between two keyframes is linear, the pseudo depth of each event can be calculated through linear interpolation. (2) in, It is the first The depth value of an event. Indicates the first The time of the event Indicates the first Keyframe timing, Indicates the first Depth of keyframes; S103: Coordinate normalization; event coordinates obtained through linear interpolation. and Since they are not in the same coordinate system, coordinate normalization can unify the coordinates of events into a global coordinate system. The formula is as follows: (3) in, It is a scaling factor, which can freely adjust the size of the pseudo-depth; Rewriting the above equation in matrix form gives us... (4) S2: Calculate the motion flow of the event point cloud, cluster it, and initialize the motion parameters. After performing three-dimensional motion compensation distortion processing on the event, the distorted event is weighted and accumulated to obtain VWE, thus obtaining the initial motion segmentation result; Step S2 is as follows: S201: Motion flow estimation of event point clouds; in a two-dimensional plane, optical flow represents the two-dimensional instantaneous velocity of pixels; analogous to optical flow, in three-dimensional space, motion flow represents the three-dimensional instantaneous velocity of the point cloud, and is used in three-dimensional motion compensation; the grayscale invariance assumption refers to the object's... Always moving At any given time, the grayscale value remains unchanged, as shown below: (5) in, It is the first The grayscale value at any given moment; Therefore, from Always moving Always (6) Expanding the left side of the above equation using Taylor and retaining the first-order terms, we get... (7) Combining equations (6) and (7), we have (8) Dividing both sides of the above equation by the phase shift, we have: (9) in, They represent in Velocity in the direction; To establish sufficient pairs of formulas To make an estimate, we establish The three planes represent the object in The projections of the images are recorded as follows, and their grayscale values ​​are denoted as... Therefore, equation (9) can be written as (10) in, , It is the index of the set; for example, when hour, ; Writing equation (10) in matrix form, we have: (11) in, represent hour right Find the partial derivative; Recorded as The above formula can be written as (12) By calculating the optical flow, we can obtain The motion flow can be represented as (13) The calculation was performed using the k-means algorithm. Clustering is performed to find cluster centers, each representing a type of motion; S202: 3D Motion Compensation and VWE; Distortion compensation of events along the direction of motion flow will make VWE clearer, and vice versa; Through the above steps, we obtain... The initial value, to distort the event. (14) (15) in, Indicates the first The position before the event was distorted Indicates the position after distortion. Indicates the reference time to which the event is distorted; The VWE is defined by weighting the distorted events. (16) in, Indicates the first A VWE of motion flow; Image contrast represents the sharpness of an image; the higher the contrast, the clearer the image. Variance is used to evaluate contrast. (17) in, express The mean, Indicates image volume; S3: Using PSO Update, through iteration and Maximize the image contrast of VWE and save the results to complete spatial motion segmentation.

2. The spatial motion segmentation method integrating depth estimation and three-dimensional motion compensation according to claim 1, characterized in that, Step S3 is as follows: S301: Parameter Initialization is performed to ensure that each event has an equal probability of being assigned to each motion type initially. (18) in, It is the number of clusters; S302: Using PSO The particle velocity is updated as follows: (19) in, Indicates the first One particle, The dimension of a particle. The inertia factor is a factor whose larger value indicates a wider range of particle flight, resulting in stronger global search capabilities but weaker local search capabilities. These represent the global and local optimal positions, respectively. It is a non-negative constant. It belongs to Random numbers; The particle update equation is as follows: (20) in, It is a constraint factor, the purpose of which is to control the weight of speed; The update is as follows: (21) in, Indicates the first The event belongs to the first The probability of a motion flow. Indicates the first VWE; S303: Continuously adjusting and The process is iterated to maximize the contrast of the VWE to obtain the final output segmentation result.

Images