Object detection and tracking

Abstract

A method and computing device for detecting and tracking objects from video input are disclosed. For example, the method and computing device can be used to track objects of interest such as lane markings in traffic. A plurality of frames corresponding to a video can be analyzed in a spatio-temporal domain by a neural network. The neural network can be trained using data synthesized in the spatio-temporal domain.

Description

Technical Field

[0001] This disclosure relates to a method, specifically a method for detecting and tracking objects from video input. Furthermore, the invention relates to corresponding computing devices and computer programs. Background Technology

[0002] The increasing prevalence of intelligent driving assistance systems in modern vehicles has presented numerous computer vision tasks, such as lane detection, road segmentation, object detection, and object tracking. These tasks require fast, efficient, and robust solutions. Summary of the Invention

[0003] This application provides a method and computing device for detecting and tracking objects from video input, which can be used to track objects of interest such as lane markings in traffic.

[0004] According to a first aspect, a method includes: acquiring a plurality of frames corresponding to a video, wherein the plurality of frames include features of interest; forming a spatiotemporal data volume based on the plurality of frames, wherein two dimensions of the spatiotemporal data volume correspond to the spatial dimensions of the plurality of frames, and one dimension of the spatiotemporal data volume corresponds to the temporal dimension of the plurality of frames; slicing the spatiotemporal data volume along a plurality of surfaces to generate a plurality of spatiotemporal images, wherein each of the plurality of spatiotemporal images corresponds to the spatiotemporal data volume along a corresponding surface of the plurality of surfaces; and enhancing the features of interest in the plurality of spatiotemporal images using a neural network to generate processed plurality of spatiotemporal images. For example, the method can efficiently enhance the features of interest. Because the neural network can analyze features of interest in the spatiotemporal domain, the structure of the neural network may be simpler and computationally more efficient.

[0005] In one implementation of the first aspect, acquiring multiple frames includes: acquiring multiple input frames corresponding to a video, and performing feature extraction on the multiple input frames to generate multiple frames and features of interest in the multiple frames. For example, the method can extract the features of interest using appropriate and efficient algorithms, and then enhance the features of interest using the neural network.

[0006] In another implementation of the first aspect, enhancing the features of interest in the plurality of spatiotemporal images includes at least one of the following: removing noise from the plurality of spatiotemporal images; connecting disconnected portions of at least one geometric shape in the plurality of spatiotemporal images; extracting at least one geometric shape in the plurality of spatiotemporal images; or classifying at least one geometric shape in the plurality of spatiotemporal images; wherein the at least one geometric shape corresponds to one of the features of interest. For example, the method can enhance the features of interest more efficiently because the features of interest may correspond to simpler geometric shapes in the spatiotemporal domain.

[0007] In another implementation of the first aspect, the method further includes projecting the enhanced features of interest from the processed plurality of spatiotemporal images onto the plurality of input frames or the plurality of frames. For example, the method is capable of projecting the enhanced features of interest (such as lane markings) back onto traffic images in a driver assistance application. Thus, for example, the enhanced features of interest can be visually displayed to the user.

[0008] In another implementation of the first aspect, the feature of interest corresponds to an object of interest in traffic. For example, the method can enhance the feature of interest corresponding to an object of interest in traffic.

[0009] In another implementation of the first aspect, the object of interest in the traffic includes at least one of the following: lane markings; a section of road; or an object in the traffic to be tracked. For example, the method can flexibly enhance various different features of interest corresponding to the object of interest in the traffic.

[0010] In another implementation of the first aspect, the neural network includes a convolutional neural network. For example, the method is capable of efficiently enhancing features of interest.

[0011] In another implementation of the first aspect, the neural network is trained using a synthetic spatiotemporal image that includes synthesized features of interest. Since the neural network can enhance the features of interest in the spatiotemporal domain, it can also be trained in the spatiotemporal domain. Synthesizing realistic training data in the spatiotemporal domain may be easier than synthesizing realistic images of traffic.

[0012] According to the second aspect, a computer program includes program code that, when the computer program is executed on a computer, performs the method provided in the first aspect.

[0013] According to a third aspect, a computing device is used to: acquire a plurality of frames corresponding to a video, wherein the plurality of frames include features of interest; form a spatiotemporal data volume based on the plurality of frames, wherein two dimensions of the spatiotemporal data volume correspond to the spatial dimensions of the plurality of frames, and one dimension of the spatiotemporal data volume corresponds to the temporal dimension of the plurality of frames; slice the spatiotemporal data volume along a plurality of surfaces to generate a plurality of spatiotemporal images, wherein each of the plurality of spatiotemporal images corresponds to the spatiotemporal data volume along a corresponding surface of the plurality of surfaces; and enhance the features of interest in the plurality of spatiotemporal images using a neural network to generate processed plurality of spatiotemporal images. For example, the computing device can efficiently enhance the features of interest. Because the neural network can analyze features of interest in the spatiotemporal domain, the structure of the neural network may be simpler and the computational efficiency higher.

[0014] In one implementation of the third aspect, the computing device is further configured to acquire the plurality of frames by performing the following steps: acquiring a plurality of input frames corresponding to a video, and performing feature extraction on the plurality of input frames to generate a plurality of frames and features of interest in the plurality of frames. For example, the computing device is capable of extracting the features of interest using appropriate and efficient algorithms, and then using the neural network to enhance the features of interest.

[0015] In another implementation of the third aspect, the computing device is further configured to enhance the features of interest in the plurality of spatiotemporal images by performing at least one of the following: removing noise from the plurality of spatiotemporal images; connecting disconnected portions of at least one geometry in the plurality of spatiotemporal images; extracting at least one geometry in the plurality of spatiotemporal images; or classifying at least one geometry in the plurality of spatiotemporal images; wherein the at least one geometry corresponds to one of the features of interest. For example, the computing device can enhance the features of interest more efficiently because the features of interest may correspond to simpler geometry in the spatiotemporal domain.

[0016] In another implementation of the third aspect, the computing device is further configured to project the enhanced features of interest from the processed plurality of spatiotemporal images onto the plurality of input frames or the plurality of frames. For example, the computing device may project the enhanced features of interest (such as lane markings) back onto traffic images in a driver assistance application. Thus, for example, the enhanced features of interest may be visually displayed to a user.

[0017] In another implementation of the third aspect, the feature of interest corresponds to an object of interest in traffic. For example, the computing device can enhance the feature of interest corresponding to an object of interest in traffic.

[0018] In another implementation of the third aspect, the object of interest in the traffic includes at least one of the following: lane markings; a section of road; or an object in the traffic to be tracked. For example, the computing device can flexibly enhance various different features of interest corresponding to the objects of interest in the traffic.

[0019] In another implementation of the third aspect, the neural network includes a convolutional neural network. For example, the computing device can efficiently enhance the features of interest.

[0020] In another implementation of the third aspect, the neural network has been trained using a synthetic spatiotemporal image that includes synthesized features of interest. Since the neural network can enhance the features of interest in the spatiotemporal domain, it can also be trained in the spatiotemporal domain. Synthesizing realistic training data in the spatiotemporal domain may be easier than synthesizing realistic images of traffic.

[0021] According to the fourth aspect, the vehicle includes the computing device provided in the third aspect.

[0022] Many of the accompanying features become better understood by referring to the following detailed description taken in conjunction with the accompanying drawings. Attached Figure Description

[0023] This description will be better understood by referring to the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a method provided in one embodiment is shown;

[0025] Figure 2 A schematic diagram of a computing device provided in one embodiment is shown;

[0026] Figure 3 A schematic diagram of a data flow provided in one embodiment is shown;

[0027] Figure 4 A schematic diagram of the main data flow provided in one embodiment is shown;

[0028] Figure 5 A schematic diagram of slicing a spatiotemporal data volume is shown in one embodiment;

[0029] Figure 6 A schematic diagram of a data preparation module provided in one embodiment is shown;

[0030] Figure 7 A schematic diagram of neural network training data provided in one embodiment is shown;

[0031] Figure 8 shows a schematic diagram of lane detection provided in one embodiment;

[0032] Figure 9 A schematic diagram of road segmentation provided in one embodiment is shown;

[0033] Figure 10 A schematic diagram of object tracking provided in one embodiment is shown.

[0034] In the accompanying drawings, the same reference numerals are used to denote the same parts. Detailed Implementation

[0035] The following detailed description, provided in conjunction with the accompanying drawings, is intended as a description of embodiments and is not intended to represent the only form in which embodiments can be constructed or used. However, the same or equivalent functions and structures can be implemented through different embodiments.

[0036] Figure 1 A flowchart of a method 100 provided in one embodiment is shown.

[0037] According to one embodiment, method 100 includes acquiring (101) a plurality of frames corresponding to a video, wherein the plurality of frames include features of interest.

[0038] For example, multiple frames can include multiple grayscale images. Each pixel in a grayscale image can correspond to a numerical value, where the value represents the pixel's color on a scale between white and black. Features of interest can correspond to lighter areas of such grayscale images. For example, features of interest can be obtained by performing feature extraction on images acquired from a camera. Alternatively, multiple frames can include images acquired from a camera without requiring separate feature extraction. Such images can include color information (e.g., in the form of RGB values) instead of grayscale information.

[0039] Each frame in a series of frames can correspond to a different moment in time. For example, multiple frames can correspond to consecutive frames in a video.

[0040] Method 100 may further include forming a (102) spatiotemporal data volume based on multiple frames, wherein two dimensions of the spatiotemporal data volume correspond to the spatial dimensions of the multiple frames, and one dimension of the spatiotemporal data volume corresponds to the temporal dimension of the multiple frames.

[0041] For example, a spatiotemporal data volume can be formed by stacking multiple frames in the time dimension.

[0042] The two dimensions of a spatiotemporal data volume can correspond to the spatial dimension of a video, while the one dimension of a spatiotemporal data volume corresponds to the temporal dimension of a video.

[0043] Spatiotemporal data volumes can also be called video volumes, video data volumes, etc.

[0044] Method 100 may further include slicing the spatiotemporal data volume along multiple surfaces (103) to generate multiple spatiotemporal images. Each of the multiple spatiotemporal images may correspond to a spatiotemporal data volume along a corresponding surface among the multiple surfaces.

[0045] According to one embodiment, each of the plurality of surfaces is curved. Therefore, one or both spatial locations of such surfaces can change as a function of the time dimension. Consequently, at least one spatial location of such surfaces can be non-constant with respect to the time dimension. Curved surfaces can have non-zero curvature.

[0046] Spatiotemporal images can also be called time slice (TS) images, generalized time slice (GTS) images, etc.

[0047] Since each of the multiple spatiotemporal images can correspond to a spatiotemporal data volume along a corresponding surface in the multiple surfaces, each spatiotemporal image can include values ​​of the spatiotemporal data volume along the corresponding surface, such as grayscale values ​​or RGB values.

[0048] Method 100 may also include using a neural network to enhance (104) the features of interest in multiple spatiotemporal images to generate multiple processed spatiotemporal images.

[0049] For example, enhancement can remove noise and / or connect disconnected parts of features of interest from multiple spatiotemporal images.

[0050] When executed on a computer, at least some of the operations of method 100 can be performed by a computer program.

[0051] Figure 2 A schematic diagram of a computing device 200 provided in one embodiment is shown.

[0052] According to one embodiment, computing device 200 is used to acquire multiple frames corresponding to a video, wherein the multiple frames include features of interest.

[0053] The computing device 200 can also be used to form a spatiotemporal data volume based on multiple frames, wherein two dimensions of the spatiotemporal data volume correspond to the spatial dimensions of the multiple frames, and one dimension of the spatiotemporal data volume corresponds to the temporal dimension of the multiple frames.

[0054] The computing device 200 can also be used to slice the spatiotemporal data volume along multiple surfaces to generate multiple spatiotemporal images. Each of the multiple spatiotemporal images can correspond to a spatiotemporal data volume along a corresponding surface among the multiple surfaces.

[0055] The computing device 200 can also use neural networks to enhance features of interest in multiple spatiotemporal images, generating multiple processed spatiotemporal images.

[0056] The computing device 200 may include a processor 201. The computing device 200 may also include a memory 202.

[0057] In some embodiments, at least some portions of the computing device 200 may be implemented as a system on a chip (SoC). For example, the processor 201, memory 202, and / or other components of the computing device 200 may be implemented using a field-programmable gate array (FPGA).

[0058] Components of computing device 200, such as processor 201 and memory 202, may not be discrete components. For example, if computing device 200 is implemented using a SoC, the components may correspond to different units of the SoC.

[0059] For example, processor 201 may include one or more of various processing devices, such as coprocessors, microprocessors, controllers, digital signal processors (DSPs), processing circuitry with or without an accompanying DSP, or various other processing devices including integrated circuits such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontroller units (MCUs), hardware accelerators, dedicated computer chips, etc.

[0060] The memory 202 can be used to store computer programs, etc. The memory 202 may include one or more volatile memory devices, one or more non-volatile memory devices, and / or a combination of one or more volatile memory devices and non-volatile memory devices. For example, the memory 202 may be implemented as a magnetic storage device (e.g., hard disk drive, floppy disk, magnetic tape, etc.), an optical-magnetic storage device, and a semiconductor memory (e.g., mask ROM, programmable ROM (PROM), erasable PROM (EPROM), flash memory ROM, random access memory (RAM), etc.).

[0061] The functions described herein can be implemented by various components of computing device 200. For example, memory 202 may include program code for performing any of the functions disclosed herein, and processor 201 may be used to perform functions based on the program code included in memory 202.

[0062] When computing device 200 is used to implement certain functions, some components of computing device 200 (such as one or more processors 201 and / or memory 202) can be used to implement those functions. Furthermore, when one or more processors 201 are used to implement certain functions, those functions can be implemented using program code, etc., included in memory 202. For example, if computing device 200 is used to perform an operation, one or more memories 202 and computer program code can be used together with one or more processors 201 to enable computing device 200 to perform that operation.

[0063] According to an embodiment, the vehicle includes a computing device 200. For example, the computing device 200 can perform intelligent driving assistance system tasks within the vehicle. For instance, the vehicle may include one or more cameras, and the computing device 200 can acquire video from these cameras. In another embodiment, the computing device 200 can be implemented for offline processing. Therefore, the computing device 200 may not be connected to the vehicle.

[0064] Figure 3 A schematic diagram of a data flow provided in one embodiment is shown.

[0065] Method 100 may further include acquiring multiple input frames corresponding to the video and performing feature extraction on the multiple input frames to generate multiple frames and features of interest in the multiple frames. For example, the multiple input frames may be acquired from a camera.

[0066] For example, in Figure 3 In some embodiments, multiple input frames 301 can be fed into the preprocessing module 303. For example, the multiple input frames 301 may correspond to images acquired from one or more cameras of the vehicle.

[0067] The preprocessing module 303 can preprocess multiple input frames 301 to generate multiple frames.

[0068] The preprocessing module 303 may be optional. Therefore, in some embodiments, multiple frames may include multiple input frames 301.

[0069] Multiple frames can be fed into the main processing module 304. The main processing module 304 can generate an output frame 305. The output frame 305 may include multiple processed spatiotemporal images.

[0070] The preprocessing module 303 and the main processing module 304 can be referred to as the main data stream 302. For example, the main data stream 302 can be executed by the computing device 200. The computing device 200 can be implemented in a vehicle, and when the vehicle is in traffic, the computing device 200 can execute the main data stream 302.

[0071] In the data preparation module 310, training data can be generated by the data generation module 306. The generated training data can be used in the neural network training module 308 within the neural network module 307. Optionally, other training data, such as training data corresponding to real images, can also be obtained from the preprocessing module 303. The trained neural network can be stored in the model memory 309. Then, the trained neural network model can be deployed on a vehicle for driver assistance applications.

[0072] The data preparation module 310 can be implemented by the computing device 200 or any other computing device. The functionality of the data preparation module 310 can be performed, for example, before the computing device 200 is used for traffic purposes. The trained neural network from the model memory 309 can be used in the main data stream 302 of the driver assistance application.

[0073] Figure 4 A schematic diagram of a main data stream 302 provided in one embodiment is shown.

[0074] The preprocessing module 303 can preprocess multiple input frames 301 to generate multiple frames 401. For example, the input frame 301 may correspond to a video of a frontal view (FV) of a vehicle or any other view. The preprocessing module 303 can extract relevant data and obtain an initial image with the object of interest. In the case of a lane detection system, for example, the preprocessing module 303 can use a lane detector to obtain a grayscale image with brighter pixels, which indicates that the pixel is more likely to belong to a road lane marking.

[0075] In the main processing module 304, multiple frames 401 obtained from the preprocessing module 303 can be merged into a spatiotemporal data volume.

[0076] In the main processing module 304, the spatiotemporal data volume can generate multiple spatiotemporal images 402 along multiple surface slices. Each spatiotemporal image in the multiple spatiotemporal images 402 can correspond to a spatiotemporal data volume along a corresponding surface in multiple curved surfaces.

[0077] A neural network can be used to enhance features of interest in multiple spatiotemporal images 402. This can generate multiple processed spatiotemporal images 403.

[0078] According to one embodiment, enhancing (104) the features of interest in a plurality of spatiotemporal images 402 includes removing noise from the plurality of spatiotemporal images 402.

[0079] Alternatively, enhancing (104) the features of interest in the multiple spatiotemporal images 402 may include connecting disconnected portions of at least one geometry in the multiple spatiotemporal images 402.

[0080] Alternatively, enhancing (104) the features of interest in the multiple spatiotemporal images 402 may include extracting at least one geometry from the multiple spatiotemporal images 402.

[0081] Alternatively, enhancing (104) the features of interest in the multiple spatiotemporal images 402 may include classifying at least one geometry in the multiple spatiotemporal images 402.

[0082] At least one geometry may correspond to one of the features of interest. For example, lines / curves in multiple spatiotemporal images 402 may correspond to lane markings. Other geometries may correspond to objects to be tracked (such as cars, pedestrians, or bicycles) or parts to be segmented (such as roads or the sky).

[0083] Enhanced features of interest in the processed spatiotemporal images 403 can be projected back onto the input frames 301 to generate multiple output frames 404. For example, if the features of interest correspond to lane markings in traffic, enhanced lane markings can be projected onto the input frames 301 to highlight the lane markings in the input frames 301. The multiple output frames 404 can correspond to the output frames 305 of the main data stream 302.

[0084] According to one embodiment, method 100 further includes projecting enhanced features of interest from a plurality of processed spatiotemporal images onto a plurality of input frames or a plurality of frames.

[0085] Figure 5 A schematic diagram of slicing spatiotemporal data volume 501 provided in one embodiment is shown.

[0086] exist Figure 5 In one embodiment, three curved surfaces 502 are shown, along which the spatiotemporal data volume 501 is sliced. Figure 5 In this embodiment, the y-position of the curved surface 502 changes as a function of the time dimension, while the x-position remains in its original position. Therefore, when the spatiotemporal data volume 501 is sliced ​​along the curved surface 502, one row of pixels from each frame in the multiple frames 401 is copied to the corresponding spatiotemporal image. Because the y-position of the curved surface 502 changes, the pixel rows can be different for different frames in the multiple frames 401.

[0087] In other embodiments, the y-position of surface 502 may be constant, while the x-position varies as a function of time. In other embodiments, the x-position and y-position of surface 502 may vary as a function of time. In other embodiments, the x-position and y-position of surface 502 may be constant as a function of time.

[0088] The shape of surface 502 can be adjusted based on the application, the object to be tracked, and other available information.

[0089] Spatiotemporal image 402 includes spatiotemporal slices of spatiotemporal data volume 501. Therefore, spatiotemporal image 402 includes a two-dimensional image that combines spatial and temporal dimensions. Figure 5 The embodiments illustrate three examples of such slices extracted from spatiotemporal data volume 501 corresponding to video that has been preprocessed by a lane detector.

[0090] Multiple spatiotemporal images 402 can be processed by a neural network. This processing may include cleaning and data extraction, etc. In the case of lane detection, the neural network can take a noisy grayscale spatiotemporal image 402 with curves that may have missing parts as input. The neural network can then output multiple processed spatiotemporal images 403 with clean connection lines. Figure 5 In one embodiment, examples are shown of a spatiotemporal image 402 before neural network enhancement and a processed spatiotemporal image 403 after neural network enhancement. The processed spatiotemporal images 403 can then be combined into the spatiotemporal volume of the original structure.

[0091] By acquiring slices from spatiotemporal data volume 501, information from many frames across multiple frames can be combined into a single spatiotemporal image 402. Therefore, a broader temporal context of the scene can be used in a single shot.

[0092] Furthermore, the effects of occlusion can be reduced by using slices of the spatiotemporal data volume 501. If an object is occluded in one frame and becomes visible again in another frame, the object can have a footprint in the spatiotemporal image 402, and the effects of occlusion can be reduced by a neural network. Therefore, method 100 and / or computing device 200 are robust to handling occlusion. Moreover, this robustness can be fine-tuned to handle different types and durations of occlusion.

[0093] Figure 6 A schematic diagram of a data preparation module 310 provided in one embodiment is shown.

[0094] The synthesized spatiotemporal image can be sent from the data generator module 306 to the neural network module 307. In the neural network module 307, neural networks such as convolutional neural networks can be trained to enhance the spatiotemporal image 402.

[0095] According to one embodiment, neural network 602 includes a convolutional neural network (CNN). Alternatively, the neural network may include any other type of neural network.

[0096] For example, a CNN can be implemented as an hourglass network, with an encoder followed by a decoder, and an optional generative adversarial network (GAN) component. The GAN component can be used to better approximate synthetic data using real images 601. The output of neural network module 307 is the trained neural network 602. Neural network 602 can also be referred to as a model, neural network model, etc. The trained neural network 602 can be used in the main processing module 304.

[0097] The neural network 602 can accept noisy grayscale images as input and learn to enhance images by linking broken lines and removing noise. If the synthesized image cannot perfectly simulate the real input, and a certain number of real images are available, GAN components can be used for better learning.

[0098] One drawback of neural networks is the need for a large annotated dataset. To overcome this obstacle, neural networks 602 can be used to process spatiotemporal images. Therefore, training data can be synthesized directly in the spatiotemporal domain, instead of using real-world training data. This eliminates the need to synthesize realistic traffic images for training, which requires significant computational resources. By using non-natural images in the spatiotemporal domain, any type of behavior can be mathematically modeled (and fine-tuned via adversarial networks if needed). For example, occlusion can be modeled by removing random patches from spatiotemporal images in the training data. Similarly, false detections and missed detections can be modeled by adding different types of noise to the spatiotemporal training images.

[0099] The dataset used for network training can be fully synthesized by mixing mathematical functions such as polynomials and / or oscillatory functions.

[0100] For example, training data can include non-realistic grayscale images in the spatiotemporal domain, which is easier for the neural network 602 to analyze. Therefore, a simpler, shallower network structure can be used without affecting performance, thus saving computational resources.

[0101] According to one embodiment, the neural network 602 is trained using a synthetic spatiotemporal image that includes synthesized features of interest.

[0102] Method 100 and computing device 200 can be used for a variety of labeling and tracking tasks and may not require human-system interaction as in initial detection or manual dataset labeling.

[0103] Method 100 and computing device 200 may not need to assume any other knowledge about the multiple frames 401, such as camera calibration or odometer measurement information. Furthermore, it may not be necessary to make assumptions about the number of objects to be detected / tracked or their properties (e.g., the number of lanes or their shape in the case of a lane detector).

[0104] Because the 602 neural network can be trained in the spatiotemporal domain, the resulting neural network model can be concise and efficient. Therefore, it can process large amounts of data in a short time.

[0105] Method 100 and computing device 200 can have a constant processing time independent of video length. By using spatiotemporal image 402, method 100 and computing device 200 can simultaneously consider a wider temporal context, thus the video duration corresponds to the spatiotemporal image height. Therefore, the difference between long and short videos is the size of the spatiotemporal image, which may not have a significant impact on computational performance.

[0106] Method 100 and computing device 200 can be implemented at a reduced cost. Method 100 and computing device 200 are capable of automatically and unsupervisedly extracting data from video sequences. Furthermore, neural network 602 can be trained in an unsupervised manner.

[0107] Figure 7 A schematic diagram of neural network training data according to an embodiment is shown.

[0108] exist Figure 7 In the embodiment, a ground real spatiotemporal image 701 and a corresponding spatiotemporal image 702 are shown. For example... Figure 7 As shown, noise has been added to the ground real spatiotemporal image 701, and some parts of the features of interest have been disconnected to obtain spatiotemporal image 702.

[0109] When the spatiotemporal image 702 is fed into the neural network 602, the neural network 602 can be trained to output a processed spatiotemporal image similar to the corresponding ground real spatiotemporal image 701.

[0110] The ground-based real-time image 701 can be generated through mathematical simulation with different statistical parameters simulating real-time images. For example, the ground-based real-time image 701 can be generated by combining various sine and polynomial functions. Different types of noise (e.g., detection noise) can then be added to simulate this. Furthermore, random additional lines can be added, and random patches can be erased from the ground-based real-time image 701 to obtain a real-time image 702 for training.

[0111] During the training of neural network 602, its parameters can be adjusted so that when spatiotemporal images 702 from the training dataset are fed into neural network 602, the output of neural network 602 is similar to a processed spatiotemporal image of the corresponding ground truth spatiotemporal image 701. The adjustments can be repeated iteratively until pre-configured conditions are met. For example, pre-configured conditions may include comparing the output of neural network 602 with ground truth, and satisfying the pre-configured conditions when the parameter of the difference between the quantized output and the ground truth spatiotemporal image 701 is below a pre-configured threshold.

[0112] Figure 8 shows a schematic diagram of lane detection provided in one embodiment.

[0113] Lane markings are an essential component of autonomous navigation. Lanes should be accurately detected, and they should not be confused with arrows or other road markings. Lanes should be predicted even if the lines themselves are not visible in the image.

[0114] Method 100 and / or computing device 200 can be applied to lane detection. Multiple frames 401 may include images acquired from a front-facing camera of a vehicle with initial lane detection. For example, the initial lane detection may be acquired by a preprocessing module 303. The slicing process of the spatiotemporal data volume 501 disclosed herein can be applied to multiple frames to obtain spatiotemporal representations. Neural networks such as CNNs can process the spatiotemporal images 402 to enhance lane markings, generating multiple processed spatiotemporal images 403. The multiple processed spatiotemporal images 403 can be projected back onto the multiple frames 401 to generate multiple output frames 305, in which lane markings are highlighted.

[0115] In some embodiments, the post-processing algorithm may take one of the processed spatiotemporal images 403 as input and assign a label ID to each lane marker. The label can then be propagated to the rest of the spatiotemporal images so that the ID of each lane marker remains consistent across the spatiotemporal images.

[0116] Figure 8 illustrates two examples of the execution of method 100 and computing device 200. In Figure 8(a), the initial lane detection contains numerous errors. For example, road arrows are confused with actual lane markings. The spatiotemporal image 402 is then augmented using a CNN to generate a processed spatiotemporal image 403. As shown in Figure 8, the features of interest, i.e., the lines corresponding to the lane markings, are augmented. Broken sections of the features of interest are reconnected, and noise is removed. The final correct lane detection result is shown in output frame 305.

[0117] Figure 8(b) shows another example where lane markings are obscured by a truck and are missing. After spatiotemporal neural network enhancement, the lane markings are redetected even when they are behind the truck.

[0118] Figure 9 A schematic diagram of road segmentation provided in one embodiment is shown.

[0119] For example, road segmentation is an important task in intelligent driving assistance systems. In this task, the input can be an image or video of traffic, and the output can be a binary classification of the image or video into road / non-road pixels. Per-pixel annotation is a very time-consuming task, potentially taking several minutes per frame. Method 100 and computing device 200 are able to achieve time-consistent road segmentation masking by reducing the number of false positives and false negatives.

[0120] The road segmentation algorithm can be applied to each of the multiple input frames 301 in the preprocessing module 303, generating multiple frames 401. These multiple frames 401 can be formed into a spatiotemporal data volume 501 as disclosed herein. The spatiotemporal data volume 501 can be sliced ​​into spatiotemporal images 402 as disclosed herein. The spatiotemporal images 402 can be fed into a neural network 602, such as a CNN, which can perform denoising to fill in holes caused by road obstructions, clean up small inaccuracies, and correct road boundaries. Similar to lane detection applications, the neural network 602 can be trained using fully synthetic data that can be mathematically generated. Information from the processed spatiotemporal images 403 can be projected back onto the multiple input frames 301 to create a clean and consistent output.

[0121] The road segmentation process can be automated, potentially without any manual labeling of the data. By utilizing the spatiotemporal domain, clean output can be generated, reducing inaccuracies that may occur in single-frame analysis.

[0122] Figure 10 A schematic diagram of object tracking provided in one embodiment is shown.

[0123] Object tracking is another possible application of method 100 and / or computing device 200. For this task, a semantic segmentation neural network can be applied to each frame of the video sequence in preprocessing, etc., generating multiple frames 401. This frame-by-frame prediction is prone to errors and inaccuracies.

[0124] The spatiotemporal data volume 501 can be sliced ​​as disclosed herein to generate a spatiotemporal image 402. The spatiotemporal image 402 can be fed into a neural network 602, such as a CNN, to generate a processed spatiotemporal image 403. Figure 10 As shown in the embodiment, the processed spatiotemporal image 403 includes a clean "path" of the object being tracked over time. The trajectory can be projected back onto the image plane to obtain a consistent trajectory of the object over time. This process can be applied to any semantic class that needs to be tracked, such as cars, bicycles, pedestrians, etc.

[0125] Although a portion of this subject matter has been described in specific language of structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as embodiments for implementing the claims, and other equivalent features and actions are intended to be within the scope of the claims.

[0126] The functions described herein may be performed at least in part by one or more computer program components. Alternatively, the functions described herein may be performed at least in part by one or more hardware logic components. For example (but not limited to), illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), and graphics processing units (GPUs).

[0127] It should be understood that the benefits and advantages described above may relate to one embodiment or several embodiments. These embodiments are not limited to those that solve any or all of the described problems, or those that have any or all of the described benefits and advantages. It will also be understood that a reference to “one” can refer to one or more of these items. The term “and / or” can be used to indicate that one or more of their connections may occur. Two or more connections may occur, or only one connection may occur.

[0128] The operations of the methods described herein can be performed in any suitable order, or simultaneously where appropriate. Furthermore, individual boxes can be removed from any method without departing from the objectives and scope of the subject matter described herein. Aspects of any of the embodiments described above can be combined with aspects of any other embodiments described to form other embodiments without losing the desired effects.

[0129] As used herein, the term "comprising" means including the identified method, box, or element, but such boxes or elements do not include an exclusive list, and the method or apparatus may include other boxes or elements.

[0130] It should be understood that the above description is given by way of example only, and various modifications can be made by those skilled in the art. The above specification, embodiments, and data provide a complete description of the structure and use of exemplary embodiments. Although various embodiments have been described above with a degree of specificity or by reference to one or more individual embodiments, those skilled in the art can make many modifications to the disclosed embodiments without departing from the spirit or scope of this specification.

Claims

1. A method (100), characterized in that, include: Obtain (101) multiple frames corresponding to the video, wherein the multiple frames include features of interest; A spatiotemporal data volume (102) is formed based on the plurality of frames, wherein two dimensions of the spatiotemporal data volume correspond to the spatial dimensions of the plurality of frames, and one dimension of the spatiotemporal data volume corresponds to the temporal dimension of the plurality of frames; The spatiotemporal data volume is sliced ​​(103) along multiple surfaces to generate multiple spatiotemporal images, wherein each of the multiple spatiotemporal images corresponds to the spatiotemporal data volume along a corresponding surface among the multiple surfaces; The neural network is used to enhance (104) the features of interest in the multiple spatiotemporal images to generate multiple processed spatiotemporal images.

2. The method (100) according to claim 1, characterized in that, The acquisition of multiple frames includes: Obtain multiple input frames corresponding to the video; Feature extraction is performed on the multiple input frames to generate the multiple frames and the features of interest in the multiple frames.

3. The method (100) according to claim 1, characterized in that, The enhancement of the features of interest in the plurality of spatiotemporal images includes at least one of the following: Remove noise from the plurality of spatiotemporal images; Connect the broken portions of at least one geometric shape in the plurality of spatiotemporal images; Extract at least one geometric shape from the plurality of spatiotemporal images; or Classify at least one geometric shape among the plurality of spatiotemporal images; Wherein, the at least one geometric shape corresponds to one of the features of interest.

4. The method (100) according to claim 2, characterized in that, The method further includes: The enhanced features of interest from the processed spatiotemporal images are projected onto the input frames or the multiple frames.

5. The method (100) according to claim 1, characterized in that, The features of interest correspond to objects of interest in traffic.

6. The method (100) according to claim 5, characterized in that, The objects of interest in the traffic include at least one of the following: Lane markings; A stretch of road; or Objects in traffic to be tracked.

7. The method (100) according to claim 1, characterized in that, The neural network includes a convolutional neural network.

8. The method (100) according to claim 1, characterized in that, The neural network is trained using a synthetic spatiotemporal image that includes synthesized features of interest.

9. A computer program, characterized in that, Includes program code, which, when the computer program is executed on a computer, performs the method according to any one of claims 1 to 8.

10. A computing device (200), characterized in that, Used for: Acquire multiple frames (401) corresponding to the video, wherein the multiple frames include features of interest; A spatiotemporal data volume (501) is formed based on the plurality of frames, wherein two dimensions of the spatiotemporal data volume correspond to the spatial dimensions of the plurality of frames, and one dimension of the spatiotemporal data volume corresponds to the temporal dimension of the plurality of frames; The spatiotemporal data volume is sliced ​​along multiple surfaces (502) to generate multiple spatiotemporal images (402), wherein each of the multiple spatiotemporal images corresponds to the spatiotemporal data volume along the corresponding surface of the multiple surfaces; The neural network (602) is used to enhance the features of interest in the plurality of spatiotemporal images to generate a plurality of processed spatiotemporal images (403).

11. The computing device (200) according to claim 10, characterized in that, It is also used to acquire the plurality of frames by performing the following steps: Acquire multiple input frames corresponding to the video (301); Feature extraction is performed on the multiple input frames to generate the multiple frames and the features of interest in the multiple frames.

12. The computing device (200) according to claim 10, characterized in that, It is also used to enhance the features of interest in the plurality of spatiotemporal images by performing at least one of the following: Remove noise from the plurality of spatiotemporal images; or Connect the broken portions of at least one geometric shape in the plurality of spatiotemporal images; Extract at least one geometric shape from the plurality of spatiotemporal images; or Classify at least one geometric shape among the plurality of spatiotemporal images; Wherein, the at least one geometric shape corresponds to one of the features of interest.

13. The computing device (200) according to claim 11, characterized in that, Also used for: The enhanced features of interest from the processed spatiotemporal images are projected onto the input frames or the multiple frames.

14. The computing device (200) according to claim 10, characterized in that, The features of interest correspond to objects of interest in traffic.

15. The computing device (200) according to claim 14, characterized in that, The objects of interest in the traffic include at least one of the following: Lane markings; A stretch of road; or Objects in traffic to be tracked.

16. The computing device (200) according to claim 10, characterized in that, The neural network includes a convolutional neural network.

17. The computing device (200) according to claim 10, characterized in that, The neural network has been trained using synthetic spatiotemporal images that include synthesized features of interest.

18. A vehicle, characterized in that, Includes the computing device according to any one of claims 10 to 17.

Images