Object pose estimation method based on deep learning and electronic device

Abstract

A deep learning-based object pose estimation method and electronic device are provided. The object pose estimation method includes: extracting a feature vector sequence and a pose vector sequence from an image frame sequence corresponding to an event stream and a pose sequence corresponding to the image frame sequence, respectively; training a neural network model for pose estimation based on the feature vector sequence and the pose vector sequence; and performing object pose estimation using the trained neural network model. According to this disclosure, object pose estimation can be performed based on the image frame sequence corresponding to the event stream and the pose sequence corresponding to the image frame sequence, which can improve efficiency.

Description

Technical Field

[0001] This disclosure generally relates to the field of computer vision, and more specifically, to a deep learning-based method for object pose estimation and an electronic device. Background Technology

[0002] 3D target tracking and six-degree-of-freedom (6DOF) pose estimation play a crucial role in artificial intelligence and robotics. Considering the continuous trajectory of a moving object in 3D space, we aim to estimate the object's pose (position and angle) at any given moment through visual observation. To facilitate tracking and avoid background interference, we mount a set of light-emitting diodes (LEDs) on the object's surface, causing them to blink at a high frequency (e.g., 1 kHz) to distinguish them from ambient noise. Since traditional camera sensors (such as CMOS) cannot capture such high-frequency blinking, we utilize the extremely high temporal resolution of a dynamic vision sensor (DVS) to detect the blinking LEDs.

[0003] Recovering six-DOF pose from visual signals can be considered a classic stereo vision problem. However, it is a rather complex process, involving feature matching, tracking, geometric transformation, and global optimization; furthermore, we need precise knowledge of the 3D object model and camera calibration parameters. Noise is introduced at each step of the process, and inaccuracies in any step (including inaccuracies in the 3D object model and camera parameters) can lead to significant errors in the final pose estimation.

[0004] Furthermore, there are methods that utilize convolutional neural networks (CNNs) to extract deep features from single-frame images (this can be achieved by applying a pre-trained CNN to a single frame). However, landmark-based images acquired by DVS cameras contain little information (e.g., little or no texture or edges), with most of the content being a black background, and can be termed extremely sparse images. Extracting features from such extremely sparse images using CNNs is difficult, even infeasible, making it challenging to perform deep learning based on extremely sparse images, and consequently, impossible for object tracking and pose estimation. Summary of the Invention

[0005] Exemplary embodiments of this disclosure provide a deep learning-based object pose estimation method and electronic device for object pose estimation in extremely sparse images.

[0006] According to exemplary embodiments of this disclosure, a deep learning-based object pose estimation method is provided, comprising: extracting a feature vector sequence and a pose vector sequence from an image frame sequence corresponding to an event stream and a pose sequence corresponding to the image frame sequence, respectively; training a neural network model for pose estimation based on the feature vector sequence and the pose vector sequence; and performing object pose estimation using the trained neural network model. This object pose estimation method can perform object pose estimation for image frame sequences including extremely sparse images. The pose estimation object can be an object with landmarks, such as a handheld controller, rather than the camera itself that captured the object. Compared with traditional methods that require the use of 3D object models and camera parameters and require matching of target features and landmarks, this method has the advantages of simplified operation, improved efficiency, and more accurate estimation results.

[0007] Optionally, the step of extracting the feature vector sequence includes: decomposing the image matrix of each image frame in the image frame sequence into a first orthogonal matrix, a second orthogonal matrix, and a diagonal matrix through singular value decomposition; generating feature vectors for each image frame based on at least a portion of the column vectors of the first orthogonal matrix, at least a portion of the column vectors of the second orthogonal matrix, and at least a portion of the singular values ​​of the diagonal matrix, to form a feature vector sequence including the generated feature vectors. This method of selecting a portion of the vectors to form feature vectors based on the singular value decomposition results can reduce the vector dimension, and can effectively reflect image features through a portion of the vectors rather than all of them, thereby improving the execution efficiency of steps such as training models using the formed feature vectors.

[0008] Optionally, the first N singular values ​​of the diagonal matrix, ordered from largest to smallest, are at least a subset of the singular values ​​of the diagonal matrix. The at least a subset of column vectors of the first orthogonal matrix are N column vectors corresponding to the N singular values, and the at least a subset of column vectors of the second orthogonal matrix are N column vectors corresponding to the N singular values, where N is a natural number. Preferably, N is 25 or greater. After ordering the singular values ​​from largest to smallest, the feature vectors generated based on the first N singular values ​​and their corresponding column vectors can effectively represent features and reduce dimensionality to improve computational efficiency.

[0009] Optionally, the neural network model includes a sequence input layer, used to generate an input matrix for training and an output matrix for training based on the feature vector sequence and the pose vector sequence, respectively. Combining the feature vector sequence and the pose vector sequence into a single matrix for training effectively integrates the various vectors.

[0010] Optionally, the input matrix is ​​a matrix formed by combining the feature vectors in the feature vector sequence according to the temporal order of the event flow, and the output matrix is ​​a matrix formed by combining the pose vectors in the pose vector sequence according to the temporal order of the event flow. Combining the matrix according to the temporal order can effectively reflect the temporal characteristics of the event flow.

[0011] Optionally, the neural network model is used for six-degree-of-freedom estimation and further includes at least one of the following layers: a first long short-term memory layer, a second long short-term memory layer, a first fully connected layer, a Dropout layer, a second fully connected layer, and a regression layer, wherein the regression layer is used to obtain the estimated pose vector based on the feature vector corresponding to the object to be estimated. This improved long short-term memory (LSTM) network combines the advantages of LSTM networks with the suitability for deep learning on extremely sparse images.

[0012] Optionally, the first long short-term memory layer includes 256 units, the second long short-term memory layer includes 512 units, the first fully connected layer includes 512 units, and the second fully connected layer includes 7 units, with the 7 units corresponding to three-dimensional coordinates and quaternions. By utilizing the units included in the layers in a rationally designed neural network, the layers can be tightly integrated, and computational resources can be fully utilized.

[0013] Optionally, the object pose estimation method further includes: acquiring an event stream for an object equipped with light-emitting diodes having a specific flashing frequency using a dynamic visual sensor camera; and synthesizing image frames based on the acquired event stream to generate an image frame sequence. The dynamic visual sensor camera allows for the rapid and efficient acquisition of image frame sequences for object pose estimation.

[0014] According to another exemplary embodiment of this disclosure, an electronic device is provided, comprising: a vector extraction unit configured to extract a feature vector sequence and a pose vector sequence from an image frame sequence corresponding to an event stream and a pose sequence corresponding to the image frame sequence, respectively; a training unit configured to train a neural network model for pose estimation based on the feature vector sequence and the pose vector sequence; and an estimation unit configured to perform object pose estimation using the trained neural network model. This electronic device can perform object pose estimation for image frame sequences including extremely sparse images. The pose estimation object can be an object with landmarks, such as a handheld controller, rather than the camera itself that captures the object. Compared with traditional pose estimation devices that require the use of a 3D object model and camera parameters and require matching of target features and landmarks, this device has advantages such as simplified operation, improved efficiency, and more accurate estimation results.

[0015] Optionally, the vector extraction unit is configured to: decompose the image matrix of each image frame in the image frame sequence into a first orthogonal matrix, a second orthogonal matrix, and a diagonal matrix through singular value decomposition; and generate feature vectors for each image frame based on at least a portion of the column vectors of the first orthogonal matrix, at least a portion of the column vectors of the second orthogonal matrix, and at least a portion of the singular values ​​of the diagonal matrix, to form a feature vector sequence including the generated feature vectors. This method of selecting a portion of the vectors to form feature vectors based on the singular value decomposition results can reduce the vector dimension, and can effectively reflect image features through a portion of the vectors rather than all of them, thereby improving the execution efficiency of operations such as training models using the formed feature vectors.

[0016] Optionally, the first N singular values ​​of the diagonal matrix, ordered from largest to smallest, are at least a subset of the singular values ​​of the diagonal matrix. The at least a subset of column vectors of the first orthogonal matrix are N column vectors corresponding to the N singular values, and the at least a subset of column vectors of the second orthogonal matrix are N column vectors corresponding to the N singular values, where N is a natural number. Preferably, N is 25 or greater. After ordering the singular values ​​from largest to smallest, the feature vectors generated based on the first N singular values ​​and their corresponding column vectors can effectively represent features and reduce dimensionality to improve computational efficiency.

[0017] Optionally, the neural network model includes a sequence input layer, used to generate an input matrix for training and an output matrix for training based on the feature vector sequence and the pose vector sequence, respectively. Combining the feature vector sequence and the pose vector sequence into a single matrix for training effectively integrates the various vectors.

[0018] Optionally, the input matrix is ​​a matrix formed by combining the feature vectors in the feature vector sequence according to the temporal order of the event flow, and the output matrix is ​​a matrix formed by combining the pose vectors in the pose vector sequence according to the temporal order of the event flow. Combining the matrix according to the temporal order can effectively reflect the temporal characteristics of the event flow.

[0019] Optionally, the neural network model is used for six-degree-of-freedom estimation and further includes at least one of the following layers: a first long short-term memory layer, a second long short-term memory layer, a first fully connected layer, a Dropout layer, a second fully connected layer, and a regression layer, wherein the regression layer is used to obtain the estimated pose vector based on the feature vector corresponding to the object to be estimated. This improved long short-term memory (LSTM) network combines the advantages of LSTM networks with the suitability for deep learning on extremely sparse images.

[0020] Optionally, the first long short-term memory layer includes 256 units, the second long short-term memory layer includes 512 units, the first fully connected layer includes 512 units, and the second fully connected layer includes 7 units, with the 7 units corresponding to three-dimensional coordinates and quaternions. By utilizing the units included in the layers in a rationally designed neural network, the layers can be tightly integrated, and computational resources can be fully utilized.

[0021] Optionally, the electronic device further includes: a dynamic vision sensor camera configured to acquire an event stream for an object equipped with light-emitting diodes having a specific flashing frequency; and an image frame sequence generation unit configured to synthesize image frames based on the acquired event stream to generate an image frame sequence. The dynamic vision sensor camera allows for the rapid and efficient acquisition of image frame sequences for object pose estimation.

[0022] According to another exemplary embodiment of this disclosure, an electronic device is provided, comprising: a dynamic vision sensor camera configured to acquire an event stream for an object having light-emitting diodes having a specific flashing frequency; and a processor configured to: synthesize image frames based on the acquired event stream to generate an image frame sequence; extract a feature vector sequence and a pose vector sequence from the image frame sequence corresponding to the event stream and a pose sequence corresponding to the image frame sequence, respectively; train a neural network model for pose estimation based on the feature vector sequence and the pose vector sequence; and perform object pose estimation using the trained neural network model. This electronic device can perform object pose estimation for image frame sequences including extremely sparse images. The pose estimation object can be an object with landmarks, such as a handheld controller, rather than the camera itself that captures the object. Compared with traditional pose estimation devices that require the use of a 3D object model and camera parameters and require matching of target features and landmarks, this device offers advantages such as simplified operation, improved efficiency, and more accurate estimation results.

[0023] According to another exemplary embodiment of the present disclosure, a system is provided that includes a storage device comprising at least one computing device and at least one storage instruction, wherein the instruction, when executed by the at least one computing device, causes the at least one computing device to perform an object pose estimation method as described above.

[0024] According to another exemplary embodiment of the present disclosure, a computer-readable storage medium is provided that stores instructions, wherein when the instructions are executed by at least one computing device, the at least one computing device causes the at least one computing device to perform the object pose estimation method as described above.

[0025] According to this disclosure, object pose estimation, especially six-DOF pose estimation of objects (e.g., virtual reality controllers), can be performed based on deep learning. Compared with traditional methods, it can improve efficiency and / or accuracy in the following aspects, which facilitates tracking and pose estimation: (1) It does not require prior knowledge of the 3D model of the target object, for example, it only requires knowledge of the distribution of light-emitting diodes (LEDs) with a specific flashing frequency on the surface of the controller; (2) It does not require prior knowledge of the camera calibration parameters; (3) It does not require matching and tracking of target features and landmarks.

[0026] According to this disclosure, pose can also be estimated directly through visual image processing without needing to know the details of the hardware (such as the camera or the object to be estimated). In addition to handheld controllers, this disclosure can also be applied to six-degree-of-freedom tracking and pose estimation of various objects such as robots, drones, and unmanned lunar rovers.

[0027] Further aspects and / or advantages of the general concept of this disclosure will be set forth in part in the description which follows, and in part will be clear from the description or may be learned by practice of the general concept of this disclosure. Attached Figure Description

[0028] The above and other objects and features of exemplary embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, which exemplarily illustrate the embodiments, wherein:

[0029] Figure 1 A flowchart illustrating a deep learning-based object pose estimation method according to an exemplary embodiment of the present disclosure is shown.

[0030] Figure 2 A schematic diagram illustrating the polarity change of an event stream according to an exemplary embodiment of the present disclosure is shown.

[0031] Figure 3 A schematic diagram showing image frames before and after singular value decomposition according to an exemplary embodiment of the present disclosure;

[0032] Figure 4 A schematic diagram of the structure of a neural network for pose estimation according to an exemplary embodiment of the present disclosure is shown.

[0033] Figure 5 A schematic diagram showing a handle controller and its image frames according to exemplary embodiments of the present disclosure;

[0034] Figure 6 A schematic diagram illustrating the relationship between the estimation results based on the single_fast subset and the actual data;

[0035] Figure 7A schematic diagram showing the relationship between the estimation results based on the single_fast subset and the actual data on the three-dimensional trajectory;

[0036] Figure 8 This diagram illustrates the relationship between the estimation results based on the free_moving subset and the actual data. Detailed Implementation

[0037] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, examples of which are illustrated in the drawings, wherein the same reference numerals always refer to the same parts. The embodiments will now be described with reference to the accompanying drawings in order to explain this disclosure.

[0038] In exemplary embodiments of this disclosure, the six-degree-of-freedom (DOF) pose of a handheld controller can be tracked in virtual reality applications. More specifically, by waving a handheld controller in front of a device such as a dynamic vision sensor (DVS) camera, a user can control the movement of a virtual controller corresponding to the handheld controller in a virtual reality environment. To identify the pose of the handheld controller and reflect the identification result as the action of the virtual controller, identification or detection can be performed on the handheld controller or similar device. The handheld controller or similar device has LEDs with specific flashing frequencies arranged on its surface. The DVS camera only detects the LEDs with specific flashing frequencies and does not detect the background other than the LEDs. In this case, concise or sparse (even extremely sparse) video frames can be generated, in which only markers representing the LEDs are visible. Such video frames can be used as the target of six-degree-of-freedom pose estimation or as input to a six-degree-of-freedom pose estimation device or system.

[0039] In exemplary embodiments of this disclosure, six-DOF pose estimation can be performed based on deep learning. More specifically, to track the trajectory of a 3D object (e.g., a virtual reality (VR) controller, hand controller, or other device) in space, the six-DOF pose of the 3D object can be estimated using an end-to-end deep learning framework or method. A significant advantage of such a deep learning framework or method is its ability to provide end-to-end estimation of the object's pose; that is, by inputting the distribution pattern of landmarks (road markers) acquired by a DVS camera, the six-DOF pose of the object in a specific coordinate system can be estimated. Furthermore, to reflect the temporal and contextual relationships between different frames, time-series-based deep learning can be employed.

[0040] In the exemplary embodiments of this disclosure, a DVS camera can be used as an example for illustration. It should be noted that this is only for the convenience of illustration and for illustrative purposes, and is not intended to limit the scope of protection of this disclosure. Other types of shooting devices are also feasible, such as shooting devices that can acquire event streams, such as DVS cameras, and other shooting devices that acquire image frame sequences from devices such as the aforementioned handheld controllers.

[0041] Figure 1 A flowchart illustrating a deep learning-based object pose estimation method according to an exemplary embodiment of the present disclosure is shown.

[0042] like Figure 1 As shown, the deep learning-based object pose estimation method according to an exemplary embodiment of the present disclosure may include steps 110 to 130.

[0043] In step 110, feature vector sequences and pose vector sequences are extracted from the image frame sequences corresponding to the event stream and the pose sequences corresponding to the image frame sequences, respectively. In step 120, a neural network model for pose estimation is trained based on the feature vector sequences and pose vector sequences. In step 130, object pose estimation is performed using the trained neural network model.

[0044] As an example, an event stream for an object equipped with LEDs having a specific blinking frequency can be acquired using a DVS camera; image frames are then synthesized based on the acquired event stream to generate a sequence of image or video frames. When the blinking frequency reaches a certain level (e.g., 1 kHz), the DVS camera's detection of markers is no longer affected by motion blur and low-frequency noise. The portion of each video frame excluding the markers can be treated as background (e.g., set to black), thus removing complex backgrounds while allowing only the markers to be detected. This can be considered an ideal visual input for pose estimation.

[0045] In an exemplary embodiment of this disclosure, the raw output of the DVS camera includes an event stream, which can be represented as (t, x, y, p), comprising a timestamp t, pixel coordinates (x, y), and the polarity p of the event. p is the polarity of the event, representing a change in LED illumination intensity, including enhancement (also called On) and reduction (also called Off). For computer vision-based deep learning, a sequence of image frames (corresponding to a time period) needs to be generated from the event stream. For events within a time period from the DVS camera output, the events can be plotted as pixels on the same image frame according to the coordinates (x, y) included in such events, thereby obtaining a binary image in which pixels of one or more events have a pixel value of 1, and other pixels have a pixel value of 0. Pixels with a pixel value of 1 correspond to events associated with LEDs having the aforementioned flicker frequency. The polarity of the event can be utilized to display pixels with a pixel value of 1 on the image frame and to distinguish them from noise.

[0046] Figure 2 A schematic diagram illustrating the polarity change of an event stream according to an exemplary embodiment of the present disclosure is shown. Figure 2 The image shows events from regular flickering and events from random noise.

[0047] It can be assumed that motion within an image frame is negligible over a certain time interval (e.g., 1.5 microseconds). At a given pixel location, if the polarity change of a continuous event reaches a cycle, that is, from "increase" to "decrease" and then back to "increase," for example, from on to off and back to on, or from off to on and back to off, this can be considered as a cycle Δt at pixel location (x,y). p And the distribution p(Δt) p |f) can be considered as a Gaussian distribution (f is a pre-set flashing frequency). If the probability p(Δt) p |f)=N(1 / Δt p -f,σ 2 If σ = 30Hz, which is greater than a predetermined threshold, then the event is considered to have been generated by an LED with a flickering frequency exceeding this threshold. In this way, events originating from regular flickering can be distinguished from events originating from random noise.

[0048] The above distinction methods are for illustrative purposes only and are not intended to limit the scope of this disclosure. Other methods for obtaining image frames from events or distinguishing between events from regular flickering and events from random noise are also feasible. For example, events can be distinguished and image frame sequences can be obtained from an event stream that includes multiple events in the following ways.

[0049] For example, multiple cycles of polarity change can be observed at a given location. If all the cycle lengths are approximately the same or the deviations are small enough, then the average length of the cycle can be considered. The final change period of the pixel position (x,y) is used; otherwise, when the difference in the length of the change period is greater than the predetermined length, the event corresponding to the pixel position (x,y) can be regarded as noise rather than an event from regular flickering.

[0050] Additionally, the flicker frequency of a given pixel can be calculated. And the calculated flicker frequency is compared with the LED's preset flicker frequency f. LED The pixels whose differences are within a predetermined threshold range are determined as LED occurrence positions, thereby obtaining an image frame including pixels corresponding to the determined LED occurrence positions. Since image frames can be obtained from events, an image frame sequence can be obtained from an event stream including multiple events.

[0051] Next, feature extraction can be performed on image frames or sequences of image frames. The acquired image frames are sparse, meaning that the number of pixels corresponding to LEDs (markers) with a specific flashing frequency is sparse relative to the background (e.g., only 10 or fewer pixels corresponding to the above LEDs in a 640×480 pixel image frame). In this case, to highlight image features and reduce the computational cost of deep learning, features of the image frames can be extracted, especially using singular value decomposition-based feature extraction methods to reduce dimensionality or perform compression.

[0052] While convolutional neural networks (CNNs) can be used to extract image features, this approach may not be suitable for sparse image frames. Specifically, such sparse image frames contain limited information; for example, they may lack edges or textures, and most pixels are simply black background pixels. CNN-based image feature extraction methods cannot effectively extract feature vectors representing these sparse image frames. If the image matrix corresponding to a sparse image frame is converted into a column or row vector feature vector, a massive vector is typically obtained (e.g., a 43200×1-dimensional vector for a 180×240 pixel image frame). Using such a vector for deep learning may require significantly longer training times or even fail to produce an effective model.

[0053] Therefore, the image matrix of each image frame in the image frame sequence can be decomposed into a first orthogonal matrix, a second orthogonal matrix, and a diagonal matrix through singular value decomposition; based on at least a portion of the column vectors of the first orthogonal matrix, at least a portion of the column vectors of the second orthogonal matrix, and at least a portion of the singular values ​​of the diagonal matrix for each image frame, a feature vector for each image frame is generated to form a feature vector sequence including the generated feature vectors.

[0054] As an example, the first N singular values ​​of the diagonal matrix in descending order are the at least part of the singular values ​​of the diagonal matrix, the at least part of the column vectors of the first orthogonal matrix are the N column vectors corresponding to the N singular values, and the at least part of the column vectors of the second orthogonal matrix are the N column vectors corresponding to the N singular values, where N is a natural number.

[0055] Specifically, due to the sparsity of image frames—for example, most pixels are black background pixels—compression or dimensionality reduction can be performed, for example, through singular value decomposition (SVD). The singular value decomposition of the m×n image matrix X of the original image frame (the image frame obtained through an event) can be performed using the following formula:

[0056] X = USV T

[0057] Where U is an m×m matrix, V is an n×n matrix, S is an m×n diagonal matrix, and each element of the m×n image matrix X represents a pixel value (grayscale value).

[0058] The first N singular values ​​in matrix S, arranged in descending order of their singular values, where N is a natural number (e.g., 1, 5, 15, 25), are retained. Correspondingly, the first N column vectors of matrix U and matrix V are also retained. These retained singular values ​​and their corresponding singular vectors contain most of the image information. Then, based on the retained column vectors of matrices U and V, and the aforementioned N singular values, a long vector of size (m+n+1)×N can be obtained. This vector is typically much smaller than m×n.

[0059] Figure 3 A schematic diagram showing image frames before and after singular value decomposition according to an exemplary embodiment of the present disclosure. Figure 3 The image shows the original image frame, the image frame obtained by retaining 1 singular value, the image frame obtained by retaining 5 singular values, the image frame obtained by retaining 15 singular values, and the image frame obtained by retaining 25 singular values. It also shows the percentage of reconstructed energy to total energy. Figure 3As shown, retaining 25 (e.g., 5.2%) or a higher percentage (e.g., above 5.2%) of the 480 singular values ​​is sufficient to effectively reconstruct the original image frame.

[0060] The vectors obtained in the above manner (e.g., column vectors of length (m+n+1)×N) can be used as feature vectors. Therefore, a sequence of feature vectors can be obtained by arranging image frames in sequence.

[0061] Deep learning can be performed based on the obtained feature vector sequence to perform pose estimation. As an example, a neural network model for pose estimation may include: a sequence input layer for generating an input matrix for training and an output matrix for training based on the feature vector sequence and the pose vector sequence, respectively.

[0062] As an example, the input matrix is ​​a matrix formed by combining each feature vector in the feature vector sequence according to the temporal order of the event flow, and the output matrix is ​​a matrix formed by combining each pose vector in the pose vector sequence according to the temporal order of the event flow.

[0063] As an example, the neural network model is used for six-degree-of-freedom estimation and includes at least one of the following layers: a first Long Short-Term Memory (LSTM) layer, a second Long Short-Term Memory layer, a first fully connected layer, a Dropout layer, a second fully connected layer, and a regression layer, wherein the regression layer ultimately outputs the estimated pose vector. During training, the ground truth pose vector can be input into the regression layer in matrix form, and the matrix formed by the ground truth pose vector can be used as the above output matrix.

[0064] As an example, the first long short-term memory layer includes 256 units, the second long short-term memory layer includes 512 units, the first fully connected layer includes 512 units, and the second fully connected layer has 7 units, with the 7 units corresponding to the object's three-dimensional coordinates and angle quaternions.

[0065] Specifically, to apply a deep learning framework, feature vectors can be extracted from each image frame. These feature vectors are then stacked column-by-column to obtain the input matrix of the neural network. The pose vectors corresponding to the image frames are then stacked column-by-column to obtain the output / label matrix of the neural network. Various methods can be used to extract feature vectors from a given image frame, including but not limited to extracting feature vectors from image frames using a pre-trained CNN or methods based on singular value decomposition.

[0066] The above image frame sequences or feature vector sequences are time-series data. To fully explore the information contained in time-series data (e.g., moving objects form a continuous trajectory during motion, adjacent image frames are similar, and adjacent image frames also express similar object poses), time-series data can be processed based on Long Short-Term Memory (LSTM) networks (e.g., applying the feature vector sequence to network training). This processing method can fully uncover the connections between adjacent data. Therefore, during training, both the input and output matrices of the neural network can be formed according to the temporal order of the sequence, which is consistent with the temporal order of the event stream.

[0067] The neural network model according to an exemplary embodiment of the present disclosure may include multiple LSTM layers stacked together to process time-series data. However, in this disclosure, instead of processing each frame of image individually, the image time-series information is explored in the form of a "time series". Therefore, the feature vectors of adjacent image frames are combined together to form a sequence input (the feature vector of each image frame is a column of the input matrix), and the respective pose vectors corresponding to each image frame are combined together as the sequence output of the neural network (each pose vector is a column of the output matrix).

[0068] Figure 4 A schematic diagram of the structure of a neural network for pose estimation according to an exemplary embodiment of the present disclosure is shown. Figure 4 The neural network shown includes: a sequence input layer, a first long short-term memory layer, a second long short-term memory layer, a first fully connected layer, a dropout layer, a second fully connected layer, and a regression layer. It obtains a sequence of feature vectors based on the image frame sequence extracted from the event stream. This sequence of feature vectors is input into the sequence input layer to form an input matrix and an output matrix, and the network is trained through each layer. When estimation is required, the feature vectors corresponding to the object to be estimated can be input into this network to obtain the estimated pose vector.

[0069] As an example, the neural network is optimized using "adaptive moments" with an initial learning rate of 0.0001, which is kept constant, a gradient threshold (weight decay) set to 1, and a minimum batch size of 64.

[0070] Figure 5 A schematic diagram of a handle controller and its image frames according to an exemplary embodiment of the present disclosure is shown. Figure 5 The handle controller shown includes LEDs with a specific flashing frequency, which can be targeted to... Figure 5The handheld controller shown performs six-degree-of-freedom pose estimation. By swinging the handheld controller in front of the DVS camera, the movement of the virtual controller can be controlled in a virtual reality environment. The DVS camera can acquire an event stream corresponding to the handheld controller. Based on the above embodiment, an image frame sequence can be obtained from the acquired event stream, and a feature vector sequence can be obtained from the image frame sequence. Furthermore, to improve the estimation accuracy, a series of LED markers can be distributed on the surface of the handheld controller, ensuring that the LED distribution pattern observed from multiple different viewpoints is substantially different, such as... Figure 5 The LED distribution pattern of the handle controller shown in (a) may include 32 LEDs as waypoints (ALM). Figure 5 (b) is a DVS image frame generated from an event stream, where each non-gray pixel represents an event with a specific location, timestamp, and polarity (white: enhanced; black: diminished).

[0071] In exemplary embodiments of this disclosure, the DVS camera can detect only changes in brightness or illumination intensity within the field of view. If there is no change in illumination intensity at a pixel location, that location will not be activated. The output of the DVS camera is a stream of events, where each event corresponds to an intensity change at a certain location and time on the image plane. An event can be written as a quadruple vector (t, x, y, p), where t is the timestamp of the event, (x, y) is the pixel location of the event in the image frame, and p is the polarity of the event. The DVS camera outputs a discrete stream of events. Compared to conventional CMOS cameras, DVS cameras have higher temporal resolution (in microseconds) and higher dynamic range, making them more suitable for applications requiring low-latency and high-dynamic-range visual data. For example, DVS cameras are suitable for detecting LEDs with very high flicker frequencies (above 1 kHz). For example, the DVS camera used could be a Samsung third-generation video graphics array (VGA) device. Alternatively, a DVS camera with a resolution of 640 × 480 pixels can be connected to an electronic device via USB 3.0.

[0072] In an exemplary embodiment of this disclosure, the object has six degrees of freedom of motion in three-dimensional space, including three-dimensional translation / position (X, Y, Z) and three-dimensional rotation (e.g., Euler angles in three orthogonal directions). To avoid the inherent problems of Euler angles (such as gimbal lock), the three Euler angles can be replaced with corresponding quaternions with four elements (q0, q1, q2, q3). Therefore, the pose of the object in three-dimensional space can be described by a 7-dimensional vector (x, y, z, q0, q1, q2, q3).

[0073] In an exemplary embodiment of this disclosure, to verify the effectiveness of pose estimation, experiments can be conducted on an authoritative dataset (DAVIS dataset), and the experimental results are shown in Table 1:

[0074] Table 1

[0075]

[0076] The DAVIS dataset includes the shapes_rotation subset and the shapes_translation subset, and the two results obtained are the average position error and the average angle error, respectively. As shown in Table 1 above, the results of this disclosure are superior to those of traditional methods (i.e., PoseNet method, Bayesian PoseNet method, and SP-LSTM method).

[0077] Since the DAVIS dataset is not specifically designed for handheld device pose estimation, and it records camera pose rather than the pose of the object itself, a novel dataset specifically for handheld device pose estimation can be used. This dataset, which can be called the Real Handheld Device Pose Dataset, includes sub-datasets such as free_moving, single_fast, and shaking. In experiments for each sub-dataset, the first 70% of the image frames were used for training, and the remaining image frames were used for testing (estimation).

[0078] Table 2

[0079]

[0080] Referring to Table 2 above, the test dataset is used for testing. The entire dataset includes the training dataset for training and the test dataset for testing. Existing methods have difficulty effectively processing the dataset shown in Table 2 (or the processing difficulty is relatively high). However, according to the method of this disclosure, pose estimation can be effectively performed on this dataset with small error.

[0081] The following combination Figures 6 to 8 The ability of the trained neural network model to estimate pose was verified. Figure 6 This diagram illustrates the relationship between the estimation results based on the single_fast subset and the actual data. Figure 7 This diagram illustrates the relationship between the estimation results based on the single_fast subset and the actual data on the 3D trajectory. Figure 8 This diagram illustrates the relationship between the estimation results based on the free_moving subset and the actual data.

[0082] exist Figure 8 In this study, the model is trained on the first 40% of the image frames in the free_moving subset, and the pose of the remaining 60% of the image frames is estimated based on the trained neural network model.

[0083] In the attached diagram, dashed lines correspond to estimated data, and solid lines correspond to actual data. (Combined) Figures 6 to 8 Especially Figure 8 As shown in the case where the training set is smaller than the test set, it can be seen that the estimation results obtained by the object pose estimation method based on this disclosure are close to the actual data, can effectively capture the real pose, and have high estimation accuracy.

[0084] According to another exemplary embodiment of the present disclosure, an electronic device is provided, comprising: a vector extraction unit configured to extract a feature vector sequence and a pose vector sequence from an image frame sequence corresponding to an event stream and a pose sequence corresponding to the image frame sequence, respectively; a training unit configured to train a neural network model for pose estimation based on the feature vector sequence and the pose vector sequence; and an estimation unit configured to perform object pose estimation using the trained neural network model.

[0085] As an example, the vector extraction unit is configured to: decompose the image matrix of each image frame in the image frame sequence into a first orthogonal matrix, a second orthogonal matrix, and a diagonal matrix by singular value decomposition; and generate feature vectors for each image frame based on at least a portion of the column vectors of the first orthogonal matrix, at least a portion of the column vectors of the second orthogonal matrix, and at least a portion of the singular values ​​of the diagonal matrix, to form a feature vector sequence including the generated feature vectors.

[0086] As an example, the first N singular values ​​of the diagonal matrix in descending order are the at least part of the singular values ​​of the diagonal matrix, the at least part of the column vectors of the first orthogonal matrix are the N column vectors corresponding to the N singular values, and the at least part of the column vectors of the second orthogonal matrix are the N column vectors corresponding to the N singular values, where N is a natural number.

[0087] As an example, the neural network model includes: a sequence input layer for generating an input matrix for training and an output matrix for training based on a sequence of feature vectors and a sequence of pose vectors, respectively.

[0088] As an example, the input matrix is ​​a matrix formed by combining each feature vector in the feature vector sequence according to the temporal order of the event flow, and the output matrix is ​​a matrix formed by combining each pose vector in the pose vector sequence according to the temporal order of the event flow.

[0089] As an example, the neural network model is used to perform six-degree-of-freedom estimation and further includes at least one of the following layers: a first long short-term memory layer, a second long short-term memory layer, a first fully connected layer, a Dropout layer, a second fully connected layer, and a regression layer, wherein the regression layer is used to obtain an estimated pose vector based on the feature vector corresponding to the object to be estimated.

[0090] As an example, the first long short-term memory layer includes 256 units, the second long short-term memory layer includes 512 units, the first fully connected layer includes 512 units, and the second fully connected layer has 7 units, with the 7 units corresponding to three-dimensional coordinates and quaternions.

[0091] As an example, the electronic device further includes: a dynamic vision sensor camera configured to acquire an event stream for an object provided with light-emitting diodes having a specific flashing frequency; and an image frame sequence generation unit configured to synthesize image frames based on the acquired event stream to generate an image frame sequence.

[0092] According to another exemplary embodiment of the present disclosure, an electronic device is provided, including: a dynamic vision sensor camera configured to acquire an event stream for an object having light-emitting diodes having a specific flashing frequency; and a processor configured to: synthesize image frames based on the acquired event stream to generate an image frame sequence; extract a feature vector sequence and a pose vector sequence from the image frame sequence corresponding to the event stream and a pose sequence corresponding to the image frame sequence, respectively; train a neural network model for pose estimation based on the feature vector sequence and the pose vector sequence; and perform object pose estimation using the trained neural network model.

[0093] It should be understood that specific implementations of the electronic device according to exemplary embodiments of this disclosure can be referred to in conjunction with... Figures 1 to 4 The specific implementation methods described will not be elaborated here.

[0094] According to exemplary embodiments of this disclosure, deep learning based on a monocular event stream camera (e.g., a DVS camera) is applied to six-degree-of-freedom tracking and pose estimation of 3D objects such as virtual reality controllers. This enables end-to-end pose estimation of objects without cumbersome stereo vision and geometry-based computations. Furthermore, it eliminates the need to know hardware details such as 3D object models and camera calibration parameters.

[0095] Compared to existing deep learning-based methods using event-stream cameras, embodiments of this disclosure can distinguish between foreground objects and background because LED markers can be distributed on the surface of the object, and the markers are detected using algorithms such as DVS flicker detection. Based on this, the pose of the object can be estimated, rather than the pose of the camera itself.

[0096] This disclosure employs sequence-based deep learning, which can fully explore the temporal / contextual information of the input video sequence, unlike existing deep learning-based methods that only explore information in each individual image frame.

[0097] To handle sparse image frames (i.e., only landmarks or waypoints are visible), singular value decomposition can be used to extract features (feature vectors) from the image frames, rather than applying CNNs, which may not be suitable for feature extraction of sparse image frames. Therefore, embodiments of this disclosure are more suitable (but not limited to) six-DOF object tracking, as well as tracking of other target objects (e.g., robots and unmanned lunar rovers) and other applications.

[0098] Furthermore, in the embodiments of this disclosure, the LED distribution pattern also affects the pose estimation results. Continuous time-series data can be used for network training. This allows for end-to-end estimation of six-DOF pose in three-dimensional space, thus avoiding the cumbersome processes of traditional stereo vision processing (e.g., object feature extraction, matching, tracking calculations, and the need to understand 3D object models and camera parameters). The use of continuous time-series data for network training allows temporal continuity constraints to play a role in the learning process. Compared to traditional learning methods that do not consider time series and temporal continuity constraints, a more efficient network model can be obtained.

[0099] The components included in the electronic device according to exemplary embodiments of this disclosure can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these devices may correspond to dedicated integrated circuits, pure software code, or modules combining software and hardware. Furthermore, one or more functions implemented by these devices may also be uniformly performed by components in a physical entity device (e.g., a processor, client, or server).

[0100] It should be understood that the method according to the exemplary embodiments of the present disclosure can be implemented by a program recorded on a computer-readable medium. For example, according to the exemplary embodiments of the present disclosure, a computer-readable medium for object pose estimation can be provided, wherein a computer program for performing the following method steps is recorded on the computer-readable medium: extracting a feature vector sequence and a pose vector sequence from an image frame sequence corresponding to an event stream and a pose sequence corresponding to the image frame sequence, respectively; training a neural network model for pose estimation based on the feature vector sequence and the pose vector sequence; and performing object pose estimation using the trained neural network model.

[0101] The computer program in the aforementioned computer-readable medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, and servers. It should be noted that the computer program can also be used to perform additional steps in addition to the steps described above, or to perform more specific processing while performing the steps described above. The contents of these additional steps and further processing have been described with reference to the accompanying drawings, and will not be repeated here to avoid repetition.

[0102] It should be noted that the electronic device according to the exemplary embodiments of this disclosure can rely entirely on the operation of a computer program to realize the corresponding function. That is, each device corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., a lib library) to realize the corresponding function.

[0103] On the other hand, the various components included in the electronic device according to the exemplary embodiments of this disclosure can also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the corresponding operations can be stored in a computer-readable medium such as a storage medium, so that a processor can perform the corresponding operations by reading and running the corresponding program code or code segments.

[0104] For example, exemplary embodiments of this disclosure can also be implemented as a computing device including a storage component and a processor, wherein the storage component stores a set of computer-executable instructions that, when executed by the processor, perform an object pose estimation method.

[0105] Specifically, the computing device can be deployed on a server or client, or on a node device in a distributed network environment. Furthermore, the computing device can be a PC, tablet, personal digital assistant, smartphone, web application, or other device capable of executing the aforementioned set of instructions.

[0106] Here, the computing device is not necessarily a single computing device, but can be any collection of devices or circuits capable of executing the above instructions (or instruction sets) individually or in combination. The computing device can also be part of an integrated control system or system manager, or can be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0107] In the computing device, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0108] Some operations described in the object pose estimation method according to exemplary embodiments of the present disclosure can be implemented in software, some operations can be implemented in hardware, and some operations can be implemented in a combination of software and hardware.

[0109] The processor can execute instructions or code stored in one of its storage components, which can also store data. The instructions and data can also be sent and received over a network via a network interface device, which can employ any known transport protocol.

[0110] Storage components can be integrated with the processor, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, storage components can include separate devices, such as external disk drives, storage arrays, or other storage devices that can be used by any database system. Storage components and the processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the storage component.

[0111] In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the computing device may be interconnected via a bus and / or network.

[0112] The operations involved in the object pose estimation method according to exemplary embodiments of this disclosure can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logic device or operate according to non-precise boundaries.

[0113] For example, as described above, an electronic device according to an exemplary embodiment of the present disclosure may include a storage component and a processor, wherein the storage component stores a set of computer-executable instructions that, when executed by the processor, perform the following steps: extracting a feature vector sequence and a pose vector sequence from an image frame sequence corresponding to an event stream and a pose sequence corresponding to the image frame sequence, respectively; training a neural network model for pose estimation based on the feature vector sequence and the pose vector sequence; and performing object pose estimation using the trained neural network model.

[0114] The foregoing has described various exemplary embodiments of this disclosure. It should be understood that the foregoing description is exemplary only and not exhaustive, and this disclosure is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A deep learning-based object pose estimation method, comprising: Extract feature vector sequence and pose vector sequence from the image frame sequence corresponding to the event stream and the pose sequence corresponding to the image frame sequence, respectively; A neural network model for pose estimation is trained based on the feature vector sequence and the pose vector sequence. Object pose estimation is performed using a trained neural network model. The steps for extracting the feature vector sequence include: Singular value decomposition is used to decompose the image matrix of each image frame in the image frame sequence into a first orthogonal matrix, a second orthogonal matrix, and a diagonal matrix. Based on at least a portion of the column vectors of the first orthogonal matrix, at least a portion of the column vectors of the second orthogonal matrix, and at least a portion of the singular values ​​of the diagonal matrix for each image frame, a feature vector is generated for each image frame to form a feature vector sequence including the generated feature vectors.

2. The object pose estimation method according to claim 1, wherein, The first N singular values ​​of the diagonal matrix in descending order are the at least a subset of the singular values ​​of the diagonal matrix. The at least a subset of column vectors of the first orthogonal matrix are the N column vectors corresponding to the N singular values. The at least a subset of column vectors of the second orthogonal matrix are the N column vectors corresponding to the N singular values, where N is a natural number.

3. The object pose estimation method according to claim 1, wherein, The neural network model includes a sequence input layer, used to generate an input matrix for training and an output matrix for training based on a sequence of feature vectors and a sequence of pose vectors, respectively.

4. The object pose estimation method according to claim 3, wherein, The input matrix is ​​a matrix formed by combining the feature vectors in the feature vector sequence according to the temporal order of the event stream. The output matrix is ​​a matrix formed by combining each pose vector in the pose vector sequence according to the time order of the event flow.

5. The object pose estimation method according to claim 3, wherein, The neural network model is used for six-degree-of-freedom estimation and further includes at least one of the following layers: The system consists of a first long short-term memory layer, a second long short-term memory layer, a first fully connected layer, a Dropout layer, a second fully connected layer, and a regression layer. The regression layer is used to obtain the estimated pose vector based on the feature vector corresponding to the object to be estimated.

6. The object pose estimation method according to claim 5, wherein, The first long short-term memory layer includes 256 units, the second long short-term memory layer includes 512 units, the first fully connected layer includes 512 units, and the second fully connected layer has 7 units, with the 7 units corresponding to three-dimensional coordinates and quaternions.

7. The object pose estimation method according to claim 1 further includes: The event stream for an object equipped with LEDs having a specific flashing frequency is acquired using a dynamic vision sensor camera. Image frames are synthesized based on the acquired event stream to produce a sequence of image frames.

8. An electronic device, comprising: The vector extraction unit is configured to extract feature vector sequence and pose vector sequence from the image frame sequence corresponding to the event stream and the pose sequence corresponding to the image frame sequence, respectively. The training unit is configured to train a neural network model for pose estimation based on a sequence of feature vectors and a sequence of pose vectors. The estimation unit is configured to perform object pose estimation using a trained neural network model. The vector extraction unit is configured to: decompose the image matrix of each image frame in the image frame sequence into a first orthogonal matrix, a second orthogonal matrix, and a diagonal matrix through singular value decomposition; and generate feature vectors for each image frame based on at least a portion of the column vectors of the first orthogonal matrix, at least a portion of the column vectors of the second orthogonal matrix, and at least a portion of the singular values ​​of the diagonal matrix, to form a feature vector sequence including the generated feature vectors.

9. The electronic device according to claim 8, wherein, The first N singular values ​​of the diagonal matrix in descending order are the at least a subset of the singular values ​​of the diagonal matrix. The at least a subset of column vectors of the first orthogonal matrix are the N column vectors corresponding to the N singular values. The at least a subset of column vectors of the second orthogonal matrix are the N column vectors corresponding to the N singular values, where N is a natural number.

10. The electronic device according to claim 8, wherein, The neural network model includes a sequence input layer, used to generate an input matrix for training and an output matrix for training based on a sequence of feature vectors and a sequence of pose vectors, respectively.

11. The electronic device according to claim 10, wherein, The input matrix is ​​a matrix formed by combining the feature vectors in the feature vector sequence according to the temporal order of the event stream. The output matrix is ​​a matrix formed by combining each pose vector in the pose vector sequence according to the time order of the event flow.

12. The electronic device according to claim 10, wherein, The neural network model is used for six-degree-of-freedom estimation and further includes at least one of the following layers: The system comprises a first long short-term memory layer, a second long short-term memory layer, a first fully connected layer, a Dropout layer, a second fully connected layer, and a regression layer, wherein the regression layer is used to obtain an estimated pose vector based on the feature vector corresponding to the object to be estimated.

13. The electronic device according to claim 12, wherein, The first long short-term memory layer includes 256 units, the second long short-term memory layer includes 512 units, the first fully connected layer includes 512 units, and the second fully connected layer has 7 units, with the 7 units corresponding to three-dimensional coordinates and quaternions.

14. The electronic device according to claim 8, wherein, The electronic device also includes: A dynamic vision sensor camera is configured to acquire an event stream for an object equipped with light-emitting diodes having a specific flashing frequency; The image frame sequence generation unit is configured to synthesize image frames based on the acquired event stream to generate an image frame sequence.

15. An electronic device comprising: A dynamic vision sensor camera is configured to acquire an event stream for an object equipped with light-emitting diodes having a specific flashing frequency; The processor is configured as follows: Image frames are synthesized based on the acquired event stream to generate a sequence of image frames; Extract feature vector sequence and pose vector sequence from the image frame sequence corresponding to the event stream and the pose sequence corresponding to the image frame sequence, respectively; A neural network model for pose estimation is trained based on the feature vector sequence and the pose vector sequence. Object pose estimation is performed using a trained neural network model. The processor is configured to decompose the image matrix of each image frame in the image frame sequence into a first orthogonal matrix, a second orthogonal matrix, and a diagonal matrix by singular value decomposition; and to generate feature vectors for each image frame based on at least a portion of the column vectors of the first orthogonal matrix, at least a portion of the column vectors of the second orthogonal matrix, and at least a portion of the singular values ​​of the diagonal matrix, to form a feature vector sequence including the generated feature vectors.

16. A system comprising at least one computing device and at least one storage device for storing instructions, wherein, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the object pose estimation method as described in any one of claims 1-7.

17. A computer-readable storage medium for storing instructions, wherein, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the object pose estimation method as described in any one of claims 1-7.

Images