Warp-free motion guidance feature reference for consistent video segmentation

Abstract

An example system includes one or more memories and one or more processors coupled to the one or more memories. The one or more processors are configured to perform a first segmentation operation on a previous frame of image data to generate first segmented data. The one or more processors are configured to perform a deformable convolution operation based on the first segmentation data to generate a deformable convolution output. The one or more processors are configured to perform a second segmentation operation on a current frame of the image data to generate second segmented data. The one or more processors are configured to combine the deformable convolution output with the second segmentation data to generate third segmentation data, and control operation of the device based on the third segmentation data.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 461,366, filed September 5, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to image segmentation. Background Technology

[0003] Accurate image segmentation can be important for a variety of use cases. For example, accurate image segmentation (e.g., identifying and classifying pixels in an image into specific categories belonging to objects) can be important for navigation (such as obstacle avoidance). Navigation can be associated with autonomous driving, robotics, extended reality (XR) scene synthesis, and more. Accurate image segmentation is also important for 3D environment construction, spatial scene understanding (e.g., for image editing), and other use cases. Summary of the Invention

[0004] This disclosure relates in general to the use of pseudo-optical flow operations to add temporal information to improve the temporal consistency of image segmentation models. By using pseudo-optical flow operations, rather than traditional or recently developed optical flow operations such as Recursive Full-Pair Transform (RAFT), the techniques of this disclosure address temporal inconsistencies in image segmentation models and can be operated on-device because the pseudo-optical flow operations described herein are likely less computationally demanding than optical flow operations. The techniques of this disclosure can be used in several types of devices, such as autonomous vehicles, XR devices, robots, etc.

[0005] In one example, this disclosure describes a system comprising: one or more memories configured to store image data captured by at least one camera; and one or more processors communicatively coupled to the one or more memories, the processors being configured to: perform a first segmentation operation on a previous frame of the image data to generate first segmented data; perform a deformable convolution operation based on the first segmented data to generate a deformable convolution output; perform a second segmentation operation on a current frame of the image data to generate second segmented data; combine the deformable convolution output with the second segmented data to generate third segmented data; and control the operation of a device based on the third segmented data.

[0006] In another example, this disclosure describes a method comprising: performing a first segmentation operation on a previous frame of image data to generate first segmented data; performing a deformable convolution operation based on the first segmented data to generate a deformable convolution output; performing a second segmentation operation on a current frame of image data to generate second segmented data; combining the deformable convolution output with the second segmented data to generate third segmented data; and controlling the operation of a device based on the third segmented data.

[0007] In another example, the disclosure describes a non-transitory computer-readable storage medium including instructions that, when executed, cause one or more processors to: perform a first segmentation operation on a previous frame of image data to generate first segmented data; perform a deformable convolution operation based on the first segmented data to generate a deformable convolution output; perform a second segmentation operation on a current frame of image data to generate second segmented data; combine the deformable convolution output with the second segmented data to generate third segmented data; and control the operation of a device based on the third segmented data.

[0008] In another example, this disclosure describes a system comprising: means for performing a first segmentation operation on a previous frame of image data to generate first segmented data; means for performing a deformable convolution operation based on the first segmented data to generate a deformable convolution output; means for performing a second segmentation operation on a current frame of image data to generate second segmented data; means for combining the deformable convolution output with the second segmented data to generate third segmented data; and means for controlling the operation of a device based on the third segmented data.

[0009] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating an example vehicle according to one or more aspects of this disclosure.

[0011] Figure 2 This is a conceptual diagram illustrating an example architecture of a system for providing depth prediction based on image data from multiple cameras, according to one or more aspects of this disclosure.

[0012] Figure 3 This is a conceptual diagram illustrating an example of a decoder-side architecture associated with a single camera among a plurality of cameras, according to one or more aspects of this disclosure.

[0013] Figure 4 This is a conceptual diagram illustrating an example neural network according to one or more aspects of this disclosure.

[0014] Figure 5 This is a flowchart illustrating a depth prediction technique according to one or more aspects of this disclosure. Detailed Implementation

[0015] Pre-trained semantic segmentation models can generate "flickering" artifacts, where a given pixel of a specific object in one image frame may be represented as belonging to one category of the object, and then in the next image frame, the same relative pixel of the same object may be represented as belonging to a different category of the object, even if the object is stationary. For example, a pixel belonging to the street may be classified as a street pixel in one image frame and as a sidewalk pixel in the next. Such correct and incorrect classifications can "flicker" back and forth across several image frames in the output of the segmentation-oriented model, resulting in temporal inconsistencies. Temporal inconsistencies can degrade service quality and increase security or other risks.

[0016] Currently, optical flow can be used to improve temporal consistency. However, optical flow typically requires significant computation and can become a major bottleneck in data processing for real-time services. Therefore, it may be desirable to address or improve temporal inconsistencies (e.g., "flickering" artifacts) in semantic segmentation models in a less computationally intensive manner, enabling the models to run in real-time and on devices.

[0017] According to the techniques disclosed herein, given a pre-trained image segmentation model (such as a semantic segmentation model), a device can utilize pseudo-optical flow operations to add temporal information to improve temporal consistency. The techniques disclosed herein can be used in several types of devices, such as autonomous vehicles, XR (extended reality) devices, robots, etc.

[0018] Most urban segmentation models used in autonomous vehicles and / or advanced driver assistance systems (ADAS) are trained on clean real-world datasets or simulated game datasets. The techniques disclosed herein can be used for online adaptation to real-world scenes, where accurate segmentation masks, along with depth information, will allow vehicles to take appropriate control actions, such as speed control, steering, braking, etc.

[0019] For XR use cases, indoor segmentation can be used for human occlusion rendering and / or semantic reconstruction. The indoor environment on which the segmentation model is trained may have different characteristics than the indoor environment in which the XR device is actually used. Changes may exist in layout, brightness, and other aspects. In such cases, online adaptation is important to address the domain shift between the pre-training dataset and the test dataset.

[0020] Similarly, for robotic use cases, the segmentation model may be trained in an environment different from the actual environment in which the deployed robot can operate. Accurate semantic segmentation enables various capabilities in robots, such as navigation, localization, and interaction with physical objects in the environment. The techniques disclosed herein can be used to address domain offset characteristics between source and target datasets, such as changes in object characteristics and environmental changes such as lighting.

[0021] According to the technology disclosed herein, instead of using optical flow operations to improve temporal consistency, the device can use pseudo-optical flow operations on-device to improve temporal consistency, such as reducing or eliminating "flickering" artifacts in the output of the image segmentation model. The pseudo-optical flow operations may take the form of deformable convolution operations or include deformable convolution operations.

[0022] The techniques disclosed herein have been tested against other specific implementations, such as image-based models, video-based models, and video-based models with raft warping (e.g., recently developed forms of optical flow). Image-based models may use information from the current frame or image of video data but not information from other frames or images. Video-based models may use information from the current frame or image and information from one or more previous frames (e.g., temporal information). Table 1 shows the test results. It can be seen that the video-based model with pseudo-optical flow of this disclosure outperforms other specific implementations. The mean intersection-over-union ratio (mIoU) is the average of the IoU values ​​calculated for each class in a multi-class segmentation problem, which can be used to determine the performance of specific implementations across different classes of an object.

[0023]

[0024] Table 1

[0025] The technique disclosed herein can improve the quality of semantic segmentation for novel environments through online adaptation. This can improve the accuracy of those categories whose characteristics change across source and target datasets.

[0026] Figure 1 This is a block diagram illustrating an example device according to one or more aspects of this disclosure. Device 100 may include autonomous driving vehicles, driver-assisted driving vehicles (e.g., vehicles with ADAS), robots, XR headsets, mobile devices, etc. It should be understood that, depending on the actual type of device, Figure 1 The components discussed herein may vary. For example, a vehicle or robot may include, but may not include, LiDAR system 102, XR headset or mobile device (such as a smartphone).

[0027] The following is about Figure 1 The description generally refers to examples where the device is a vehicle or a robot. In the example where device 100 represents an autonomous vehicle, device 100 may be referred to as a self-driving vehicle.

[0028] Device 100 may include a LiDAR system 102, one or more cameras 104, a controller 106, one or more sensors 108, an input / output device 120, a wireless connectivity component 130, and a memory 160. The LiDAR system 102 may include one or more light emitters and one or more light sensors. The LiDAR system 102 may be deployed in or around device 100. For example, the LiDAR system 102 may be mounted on top of device 100, in the bumper of device 100, and / or in other locations on device 100. The LiDAR system 102 may be configured to emit light pulses and sense light pulses reflected from objects in the environment of device 100. The LiDAR system 102 (and / or processor 110) may determine the distance to such objects based on the time between the emission of a light pulse and the sensing of its reflection. The LiDAR system 102 may emit such pulses within a 360-degree field of view around device 100 to detect objects within the 360-degree field of view, such as objects in front of, behind, or beside device 100. Although this document describes the inclusion of LiDAR system 102, it should be understood that another distance or depth sensing system may be used instead of LiDAR system 102.

[0029] Depth estimation can be relatively important for a variety of applications, such as autonomous driving, assisted robotics, extended reality scene synthesis, and image editing. For example, in autonomous driving scenarios, depth estimation can provide an estimated distance from one vehicle (e.g., the first device) to another vehicle (e.g., the second device), which can be important for the operating system of the first vehicle (e.g., acceleration, braking, steering, etc.). Depth estimation can be achieved by utilizing any of a number of different principles (such as those associated with camera 104, sensor 108, and / or LiDAR system 102).

[0030] Camera 104 may include one or more camera sensors (also referred to as cameras) located in or on a vehicle, such as in or on a mirror, bumper, and / or other location of device 100. Camera 104 may be configured to capture video or image data of the environment 195 surrounding device 100. Image data may include still images and / or one or more video frames or pictures. Controller 106 may use information from camera 104 to determine depth information of objects that may be in the field of view of one or more cameras in camera 104.

[0031] Controller 106 may be an autonomous or assisted driving controller (e.g., ADAS) configured to control the operation of device 100. For example, controller 106 may control the acceleration, braking, and / or navigation of device 100 through its surrounding environment. Controller 106 may include one or more processors, such as processor 110. The processor may include one or more central processing units (CPUs), such as a single-core or multi-core CPU, a graphics processing unit (GPU), a digital signal processor (DSP), a neural processing unit (NPU), a multimedia processing unit, etc. Instructions executed by processor 110 may be loaded, for example, from memory 160, and may cause processor 110 to perform operations attributed to the processor in this disclosure. In some examples, one or more processors in processor 110 may be based on the ARM or RISC-V instruction set.

[0032] An NPU is typically a dedicated circuit configured to implement all the necessary control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RF), kernel methods, etc. An NPU may also be alternatively referred to as a neural signal processor (NSP), tensor processing unit (TPU), neural network processor (NNP), intelligent processing unit (IPU), or vision processing unit (VPU).

[0033] Processor 110 can be configured to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other tasks. In some examples, multiple processors 110 can be instantiated on a single chip such as a system-on-a-chip (SoC), while in other examples, one or more processors among processors 110 can be part of a dedicated machine learning accelerator device.

[0034] In some examples, one or more processors in processor 110 may be optimized for training or inference, or in some cases configured to balance performance between the two. For processor 110 capable of performing both training and inference, these two tasks can generally still be performed independently.

[0035] Although depicted as part of device 100, in some examples, such as when processor 110 is used for training, processor 110 may be external to the device (e.g., outside of device 100). For example, processor 110 may reside in a cloud computing environment. In some examples, processor 110 may be partially on the device (e.g., part of device 100) and partially outside the device.

[0036] In some examples, processor 110, designed to accelerate training, is generally configured to speed up the optimization of new models, which involves taking an existing dataset (typically labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters 184 (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backpropagating through the layers of the model and determining gradients to reduce prediction errors. In some examples, some or all of the tuning of model parameters 184 (e.g., training) can be performed external to device 100, such as in a cloud computing environment.

[0037] In some examples, the processor 110, designed to accelerate inference, is generally configured to operate on the full model. Such a processor 110 can thus be configured to take a new piece of data as input and rapidly process the data through a trained model to generate model output (e.g., inference).

[0038] In some examples, processor 110 can operate on predictive models such as artificial neural networks (ANNs) or random forests (RFs). An ANN can include hardware and / or software components comprising multiple connected nodes (e.g., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another (like a physical synapse in the brain). When a node receives a signal, it processes it and then sends the processed signal to other connected nodes. In some cases, the signals between nodes consist of real numbers, and the output of each node is computed as a function of the sum of its inputs. Each node and edge can be associated with one or more node weights that determine how the signal is processed and sent. During training, these weights are adjusted to improve the accuracy of the results (i.e., by minimizing a loss function that corresponds in some way to the difference between the current result and the target result). Edge weights increase or decrease the strength of the signal sent between nodes. In some cases, nodes have a threshold below which no signal is sent at all. In some examples, nodes are clustered into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal passes through certain layers multiple times.

[0039] Convolutional Neural Networks (CNNs) are a class of neural networks commonly used in computer vision or image classification systems. In some cases, CNNs enable the processing of digital images with minimal preprocessing. CNNs can be characterized by using convolutional (or cross-correlated) hidden layers. These layers apply convolutional operations to the input before signaling the results to the next layer. Each convolutional node can process a finite field of data in the input (i.e., the receiving field). During the forward pass of a CNN, filters at each layer convolve across the input volume, thus computing the dot product between the filter and the input. During training, filters can be modified so that they activate when they detect specific features within the input.

[0040] The term "loss function" refers to a function that influences how a machine learning model is trained in a supervised learning model. Specifically, during each training iteration, the model's output is compared to known ground truth values. The loss function provides a measure of how close the predicted data is to the actual data (e.g., the ground truth). After calculating the loss function, the model's parameters are updated accordingly, and a new set of predictions is made during the next iteration.

[0041] In some aspects, the wireless connectivity 130 component may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity 130 processing component is also connected to one or more antennas 135.

[0042] The processor 110 may also include one or more sensor processing units associated with the LiDAR system 102, camera 104, and / or sensor 108. For example, the processor 110 may include one or more image signal processors associated with camera 104 and / or sensor 108, and / or a navigation processor associated with sensor 108, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0043] Device 100 may also include one or more input and / or output devices 120, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0044] Device 100 also includes memory 160, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, memory 160 includes a computer-executable component that can be executed by one or more of the aforementioned components of device 100.

[0045] Examples of memory 160 include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory 160 include solid-state memory and hard disk drives. In some examples, memory 160 is used to store computer-readable, computer-executable software including instructions that, when executed, cause a processor to perform the various functions described herein. In some cases, among others, memory 160 includes a basic input / output system (BIOS) that controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, a memory controller operates the memory cells. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells within memory 160 store information in the form of logical states.

[0046] This disclosure describes a technique for determining motion information using deformable convolution operations. Previous work computes 4D correlated volumes to obtain optical flow. The technique of this disclosure utilizes a deformable convolution operation implemented with feature references, which can leverage previous features and estimate motion information to guide current predictions.

[0047] Optical flow is a technique that can be used to describe motion such as motion between two video frames (e.g., time-adjacent video frames). For example, optical flow can determine the corresponding velocity of corresponding pixels within a frame and determine an estimate of where the pixel might be in the next frame. Thus, optical flow can be a representation of the motion of various parts of a frame caused by motion (such as the motion of device 100, environment 195, and / or objects in environment 195).

[0048] Optical flow can be used to determine or estimate the 3D motion of objects within environment 195 and / or device 100 and / or other objects within environment 195. Optical flow can also be used to determine the structure of objects within environment 195. Recursive Total Field Transform (RAFT) is a relatively new technique for optical flow.

[0049] Generally speaking, device 100 and / or its components may be configured to perform the techniques described herein. Figure 1 Device 100 is merely an example, and alternative devices 100 with more, fewer, and / or different components may be used in other examples.

[0050] In some aspects, camera 104 and / or sensor 108 may include optical instruments (e.g., image sensors, cameras, etc.) for recording or capturing images, which may be stored locally, transmitted to another location, etc. For example, an image sensor may use one or more photosensitive elements to capture visual information, the one or more photosensitive elements being tunable for sensitivity to the visible spectrum of electromagnetic radiation. The resolution of this visual information can be measured in pixels, where each pixel may relate to a separate piece of capture information. In some cases, each pixel may therefore correspond to, for example, a component of a two-dimensional (2D) Fourier transform of the image. Computational methods can use the pixel information to reconstruct the image captured by the device. In a camera, an image sensor converts light incident on the camera lens into an analog or digital signal. Electronic devices may then display the image on a display panel based on the digital signal. Image sensors are typically mounted on electronic devices such as smartphones, tablet PCs, laptop PCs, and wearable devices.

[0051] In some aspects, sensor 108 may include a direct depth sensing sensor, which can be used to determine the depth of an object or the distance to an object within the environment surrounding device 100. Data from such a depth sensing sensor can be used to supplement the depth map generation techniques discussed herein.

[0052] Input / output device 120 (e.g., it may include an I / O controller) manages the input and output signals of device 100. In some cases, input / output device 120 may represent a physical connection or port to an external peripheral device. In some cases, input / output device 120 may utilize an operating system. In other cases, input / output device 120 may represent or interact with a modem, keyboard, mouse, touchscreen, or similar device. In some cases, input / output device 120 may be implemented as part of a processor (e.g., the processor in processor 110). In some cases, a user may interact with the device via input / output device 120 or via hardware components controlled by input / output device 120.

[0053] Specifically, in this example, memory 160 includes model parameters 184 (e.g., weights, biases, and other machine learning model parameters 184) of any machine learning model discussed herein. Memory 160 may also include: segmented data 162, which may include segmented data for flicker error correction; losses 182, each of which can be used to train one or more machine learning models described herein; and inference 186, which may include inferences made by any machine learning model discussed herein. One or more of the depicted components and other undepicted components may be configured to perform aspects of the methods described herein.

[0054] Figure 2 This is a conceptual diagram illustrating an example high-level architecture of a device for video segmentation according to one or more aspects of this disclosure. Figure 2 The high-level architecture can be provided by Figure 1 The device 100 implements this. For example, the processor 110 can compare previously determined features and / or incremental information 200 with feature reference 204 and determine motion representation 210. The processor 110 can then compare the motion representation 210 with L... t-1 The previous estimate 202 was used as motion guidance 206 to generate L t-1 To the predicted L' t Mapping of 212.

[0055] Figure 3 This is a conceptual diagram illustrating an example architecture of a device for video segmentation according to one or more aspects of this disclosure. Figure 3 The architecture can be derived from Figure 1 The device 100 is implemented in this way. For example, the processor 110 may extract features from a previous image I via the feature extraction unit 304. t-1 300 and current image I t Feature extraction. Previous image I t-1 300 and current image I t The image can be captured by camera 104. Feature extraction unit 304 can be implemented using neural networks or other machine learning models. The output of feature extraction unit 304 can be compared with the feature reference of feature reference unit 306, and the output of feature reference unit 306 can be input to pseudo optical flow unit 310 to generate pseudo optical flow.

[0056] In some examples, feature extraction unit 304 may be configured to perform semantic segmentation. Semantic segmentation is a technique used to classify each pixel in an image (e.g., a video data frame) into a category or object. For example, the output of semantic segmentation may include a pixel-by-pixel segmentation map of the image, where each pixel in the image is assigned to a specific category or object. For example, each pixel may be assigned a category label, which may be an identifier whose value is associated with a specific category to which the particular pixel belongs, such as 0 (tree), 1 (road), 2 (building), 3 (car), 4 (person), 5 (bicycle), etc. In some examples, such a segmentation map may not separate different objects within the category, such as two different bicycles. In other examples, such as when using an instance segmentation model, the output segmentation map may distinguish between different objects within a category. In some examples, the features of a pixel may include the category to which the pixel belongs.

[0057] Category-specific objects can be located, for example, by drawing a bounding box around the object. Segmentation masks can be used to group pixels of a specific object in the located image. Image features can be extracted using semantic segmentation to divide the image into multiple segments.

[0058] A convolutional network used to extract features from an image may be referred to as an encoder. In some examples, the encoder also downsamples the image, while the convolutional network used for upsampling may be referred to as a decoder. In some examples, the feature extraction unit 304 or the segmentation unit may include both an encoder and / or a decoder.

[0059] The output of the feature extraction unit 304 may include a segmentation estimate 308. The segmentation estimate 308 may be the current image I... t The estimation of segmentation (e.g., segmentation prediction).

[0060] The outputs of segmentation estimation 308 and pseudo-optical flow unit 310 can be combined by adder 312 and input to motion guidance unit 314 to determine motion guidance 314.

[0061] Current image I t 302 can be input to segmentation estimation unit 316 (which may be the same as or different from segmentation estimation unit 308). Motion guidance information from motion guidance unit 314 can also be input to segmentation estimation unit 316. Segmentation estimation unit 316 can output the predicted current image I'. t 318.

[0062] Figure 4 This is a conceptual diagram illustrating a more detailed example architecture of a device for video segmentation according to one or more aspects of this disclosure. Figure 4 The architecture can be derived from Figure 1 The device 100 is implemented.

[0063] It can be a previous image I t-1 Previous image I of the 300 example t-1 400 and can be the current image I t Example 302: Current image I t 402 can be captured by camera 104. The stitching unit 420 can stitch together previous images I... t-1 400. Current Image I t 402 and the difference between the two are concatenated. Feature extraction unit 404 can extract features from the output of concatenation unit 420. Feature extraction unit 404 can be an example of feature extraction unit 304 and can include machine learning models, such as encoders.

[0064] Previous image I t-1 Image 400 can be input to feature extraction unit 422, which may include a machine learning model having an encoder portion and a decoder portion. The output of the encoder portion of feature extraction 422 may represent the previous image I. t-1 400's characteristic F t-1 435. Similarly, the current image I t 402 can be input to feature extractor 424 (which may be the same as or different from feature extractor 422). Feature extractor 424 may include a machine learning model with an encoder portion and a decoder portion. The output of the encoder portion of feature extractor 424 may represent the current image I. t 402's characteristic F t 426.

[0065] The splicing unit 428 (which may be the same as or different from the splicing unit 420) can transmit feature F t 426. Feature F t-1 435 and the difference between the two are concatenated.

[0066] The outputs of feature extraction unit 404 and stitching unit 428 can be input to offset convolution unit 432 and modulation convolution unit 434. Offset convolution unit 432 can be configured to determine the features extracted from the previous image I. t-1 400 and the current image I t How many corresponding pixels have been moved between 402? The modulation convolutional unit 434 can be configured to determine the relative importance (e.g., weights) of one or more features for each pixel in the corresponding pixel. The output Δp of the offset convolutional unit 432... k and the output Δm of the modulation convolution unit 434 k It can be input into deformable convolutional unit 440, which can process Δp k and Δmk Perform deformable convolution operations. Although the offset convolution unit 432 and the modulation convolution unit 434 are shown separately from the deformable convolution unit 440, in some examples, either or both of the offset convolution unit 432 or the modulation convolution unit 434 may be implemented as one or more layers of the deformable convolution unit 440.

[0067] Feature extraction unit 422 can output segmented data A t-1 430, the segmented data can be input into the deformable convolutional unit 440. The feature extraction unit 424 can output segmented data A. t 436. Output 460 of deformable convolutional unit 440 and segmented data A t 436 can be combined with adder 412, which can be an example of adder 312, which can output segmented data A". t 442. Split data A" t 442 may include the current image I t The 402 segmented data has better flicker error performance than segmented data A. t 436 Improved features.

[0068] When training the disclosed machine learning model, processor 110 can generate losses 450, 452, and / or 454. Losses 450, 452, and / or 454 can be used to influence the implementation... Figure 4 The architecture of the device 100 pairs, for example, split data A" t 442 makes a prediction or estimate. For example, losses 450, 452, and / or 454 can be used to train offset convolutional unit 432, modulation convolutional unit 434, deformable convolutional unit 438, and / or deformable convolutional unit 440. In some examples, training occurs off-device.

[0069] For example, the outputs of the offset convolution unit 432 and the modulation convolution unit 434, as well as the feature F t-1 435 can be input to deformable convolutional unit 438 (which may be the same as or different from deformable convolutional unit 440). Although offset convolutional unit 432 and modulation convolutional unit 434 are shown separately from deformable convolutional unit 438, in some examples, either or both of offset convolutional unit 432 or modulation convolutional unit 434 may be implemented as one or more layers of deformable convolutional unit 438.

[0070] Deformable convolutional units 438 can determine the predicted feature F' t 439. Processor 110 can be based on feature F t 426 and predicted features F' t439 is used to generate a loss of 450. Processor 110 can be based on segmented data A t 436 is used to generate loss 452. Processor 110 can generate loss 454 based on the output 460 of deformable convolution unit 440.

[0071] Figure 5 This is a flowchart illustrating an example temporal consistency technique for image segmentation according to one or more aspects of this disclosure. Figure 5 The technology is about Figure 1 and Figure 4 This is under discussion. However, these techniques can be implemented by any system or device capable of doing so.

[0072] Processor 110 may perform a first segmentation operation on a previous frame of image data to generate first segmented data (500). For example, processor 110 may segment the previous image I via segmentation unit 422. t-1 400 Perform the first splitting operation to generate split data A t-1 430. Segmentation unit 422 may include a machine learning model.

[0073] Processor 110 may perform deformable convolution operations based on the first segmented data to generate deformable convolution output (502). For example, processor 110 may perform deformable convolution operations based on segmented data A via deformable convolution unit 440. t-1 430 is used to perform deformable convolution operations to generate deformable convolution output.

[0074] Processor 110 can perform a second segmentation operation on the current frame of image data to generate second segmentation data (504). For example, processor 110 can segment the current image I via segmentation unit 424. t 402 Perform the second segmentation operation to generate segmented data A t 436. Segmentation unit 424 may include a machine learning model.

[0075] Processor 110 can combine the deformable convolution output with the second segmented data to generate third segmented data (506). For example, processor 110 can use adder 412 to combine the deformable convolution output 460 with the segmented data A t 436 combinations to generate segmented data A" t 442.

[0076] Processor 110 can control the operation of the device based on the third segmented data (508). For example, processor 110 can control the operation of the device based on segmented data A". t 442 is used to control device 100. For example, to split data A. t442 can represent semantic segmentation that has been corrected to reduce or eliminate flickering errors in the segmented data. For example, processor 110 can base segmentation on segmented data A". t 442 is used to control one's own vehicle to remain on the road and off the sidewalk.

[0077] In some examples, at least one of the first segmentation operation or the second segmentation operation is a semantic segmentation operation. In some examples, processor 110 uses a machine learning model to perform the first segmentation operation and the second segmentation operation.

[0078] In some examples (e.g., the deformable convolution operation of deformable convolution unit 440), the deformable convolution operation is further based on offset convolution operation and modulation convolution operation. For example, offset convolution unit 432 and modulation convolution unit 434 can provide corresponding outputs that can be used to perform the deformable convolution operation.

[0079] In some examples, processor 110 can retrieve image data (e.g., previous image I) from image data. t-1 Features are extracted from previous frames (400) to generate the first extracted feature F. t-1 Processor 110 can retrieve image data (e.g., current image I) from image data. t Extract features from the current frame of (402) to generate a second extracted feature F. t 426. Processor 110 may (e.g., using splicing unit 420) process the first extracted feature F t-1 435. Second extracted feature F t 426 The difference between the second extracted feature and the first extracted feature (e.g., F) t 426 - F t-1 435) are stitched together to generate stitched extracted features. Processor 110 can process image data (e.g., previous image I) to generate stitched extracted features. t-1 Previous frames (e.g., current image I) of 400) t 402) The difference between the current frame of the image data and the previous frame of the image data (e.g., the current image I) t 402 - Previous Image I t-1 The processor 110 (400) performs stitching to generate stitched image data. The processor 110 may (e.g., via feature extraction unit 404) extract features from the stitched image data to generate third extracted features. In some examples, the processor 110 may perform an offset convolution operation on the third extracted features and the stitched extracted features. In some examples, the processor 110 may perform a modulation convolution operation on the third extracted features and the stitched extracted features.

[0080] In some examples, the first extracted feature F t-1435 includes the output of the encoder of the first machine learning model, and the second extracted feature F t 426 includes the output of the encoder of the second machine learning model, and the third extracted feature includes the output of the encoder of the third machine learning model.

[0081] In some examples, the deformable convolution operation is a first deformable convolution operation, and the deformable convolution output 460 is a first deformable convolution output. In such examples, the processor 110 may determine a first loss 450, which is based on a second extracted feature F. t 426 and the second deformable convolution output (e.g., deformable convolution unit 438 (which may be the same as or different from the output of deformable convolution unit 440), predicted feature F') t In some examples, the output of the second deformable convolution is based on the first extracted feature F. t-1 435. Offset convolution output and modulation convolution output. In some examples, processor 110 trains at least one of an offset convolution operation, a modulation convolution operation, or a first deformable convolution operation based on a first loss 450.

[0082] In some examples, processor 110 may determine a second loss 452 (the second loss 452 is based on the second segmentation data A). t 436) and based on the second loss 452, train at least one of the machine learning model of the offset convolution operation, the machine learning model of the modulation convolution operation, or the machine learning model of the first deformable convolution operation. In some examples, the processor 110 may determine a third loss 454 (the third loss 454 is based on the deformable convolution output (e.g., the output of the deformable convolution unit 440)) and based on the third loss 454, train at least one of the machine learning model of the offset convolution operation, the machine learning model of the modulation convolution operation, or the machine learning model of the first deformable convolution operation.

[0083] In some examples, device 100 includes a vehicle or robot, and wherein, as part of controlling the operation of the vehicle or robot, processor 110 enables the vehicle or robot to navigate in environment 195.

[0084] Examples in various aspects of this disclosure may be used individually or in any combination.

[0085] This disclosure includes the following terms.

[0086] Clause 1. A system comprising: one or more memories configured to store image data captured by at least one camera; and one or more processors communicatively coupled to the one or more memories, the one or more processors configured to: perform a first segmentation operation on a previous frame of the image data to generate first segmented data; perform a deformable convolution operation based on the first segmented data to generate a deformable convolution output; perform a second segmentation operation on a current frame of the image data to generate second segmented data; combine the deformable convolution output with the second segmented data to generate third segmented data; and control operation of a device based on the third segmented data.

[0087] Clause 2. The system according to Clause 1, wherein at least one of the first segmentation operation or the second segmentation operation is a semantic segmentation operation.

[0088] Clause 3. The system according to Clause 1 or Clause 2, wherein one or more processors are configured to use a machine learning model to perform the first segmentation operation and the second segmentation operation.

[0089] Clause 4. The system according to any one of Clauses 1 to 3, wherein the deformable convolution operation is further based on an offset convolution operation and a modulation convolution operation.

[0090] Clause 5. The system according to Clause 4, wherein the one or more processors are further configured to: extract features from the previous frame of image data to generate a first extracted feature; extract features from the current frame of image data to generate a second extracted feature; concatenate the first extracted feature, the second extracted feature, and the difference between the second extracted feature and the first extracted feature to generate a concatenated extracted feature; concatenate the previous frame of image data, the current frame of image data, and the difference between the current frame of image data and the previous frame of image data to generate concatenated image data; and extract features from the concatenated image data to generate a third extracted feature, wherein the one or more processors are configured to perform the offset convolution operation on the third extracted feature and the concatenated extracted feature, and wherein the one or more processors are configured to perform the modulation convolution operation on the third extracted feature and the concatenated extracted feature.

[0091] Clause 6. The system according to Clause 5, wherein the first extracted feature includes the output of the encoder of a first machine learning model, the second extracted feature includes the output of the encoder of a second machine learning model, and the third extracted feature includes the output of the encoder of a third machine learning model.

[0092] Clause 7. The system according to Clause 5 or Clause 6, wherein the deformable convolution operation is a first deformable convolution operation, the deformable convolution output is a first deformable convolution output, and wherein the one or more processors are further configured to: determine a first loss based on a second extracted feature and a second deformable convolution output, wherein the second deformable convolution output is based on the first extracted feature, the output of the offset convolution operation, and the output of the modulation convolution operation; and train at least one of a machine learning model of the offset convolution operation, a machine learning model of the modulation convolution operation, or a machine learning model of the first deformable convolution operation based on the first loss.

[0093] Clause 8. The system according to Clause 7, wherein the one or more processors are further configured to: determine a second loss based on the second segmented data; and train at least one of the machine learning model of the offset convolution operation, the machine learning model of the modulation convolution operation, or the machine learning model of the first deformable convolution operation based on the second loss.

[0094] Clause 9. The system according to Clause 7 or Clause 8, wherein the one or more processors are further configured to: determine a third loss based on the deformable convolution output; and train at least one of the machine learning model of the offset convolution operation, the machine learning model of the modulation convolution operation, or the machine learning model of the first deformable convolution operation based on the third loss.

[0095] Clause 10. A system according to any one of Clauses 1 to 9, wherein the device includes a vehicle or robot, and wherein, as part of controlling the operation of the vehicle or robot, the one or more processors are configured to enable the vehicle or robot to navigate in an environment.

[0096] Clause 11. A method comprising: performing a first segmentation operation on a previous frame of image data to generate first segmentation data; performing a deformable convolution operation based on the first segmentation data to generate a deformable convolution output; performing a second segmentation operation on a current frame of the image data to generate second segmentation data; combining the deformable convolution output with the second segmentation data to generate third segmentation data; and controlling operation of a device based on the third segmentation data.

[0097] Clause 12. The method according to Clause 11, wherein at least one of the first segmentation operation or the second segmentation operation is a semantic segmentation operation.

[0098] Clause 13. The method according to Clause 11 or Clause 12, wherein performing the first segmentation operation and performing the second segmentation operation includes using a machine learning model.

[0099] Clause 14. The method according to any one of Clauses 11 to 13, wherein the deformable convolution operation is performed further based on an offset convolution operation and a modulation convolution operation.

[0100] Clause 15. The method according to Clause 14, the method further comprising: extracting features from the previous frame of the image data to generate a first extracted feature; extracting features from the current frame of the image data to generate a second extracted feature; concatenating the first extracted feature, the second extracted feature, and the difference between the second extracted feature and the first extracted feature to generate a concatenated extracted feature; concatenating the previous frame of the image data, the current frame of the image data, and the difference between the current frame of the image data and the previous frame of the image data to generate concatenated image data; extracting features from the concatenated image data to generate a third extracted feature; performing the offset convolution operation on the third extracted feature and the concatenated extracted feature; and performing the modulation convolution operation on the third extracted feature and the concatenated extracted feature.

[0101] Clause 16. The method according to Clause 15, wherein the first extracted feature includes the output of the encoder of a first machine learning model, the second extracted feature includes the output of the encoder of a second machine learning model, and the third extracted feature includes the output of the encoder of a third machine learning model.

[0102] Clause 17. The method according to Clause 15 or Clause 16, wherein the deformable convolution operation is a first deformable convolution operation, the deformable convolution output is a first deformable convolution output, the method further comprising: determining a first loss based on a second extracted feature and a second deformable convolution output, wherein the second deformable convolution output is based on the first extracted feature, the offset convolution output, and the modulation convolution output; and training at least one of a machine learning model of the offset convolution operation, a machine learning model of the modulation convolution operation, or a machine learning model of the first deformable convolution operation based on the first loss.

[0103] Clause 18. The method according to Clause 17, the method further comprising: determining a second loss based on the second segmentation data; and training at least one of the machine learning model of the offset convolution operation, the machine learning model of the modulation convolution operation, or the machine learning model of the first deformable convolution operation based on the second loss.

[0104] Clause 19. The method according to Clause 17 or Clause 18, the method further comprising: determining a third loss based on the deformable convolution output; and training at least one of the machine learning model of the offset convolution operation, the machine learning model of the modulation convolution operation, or the machine learning model of the first deformable convolution operation based on the third loss.

[0105] Clause 20. The method according to any one of Clauses 11 to 19, wherein the device includes a vehicle or robot, and wherein, as part of controlling the operation of the vehicle or robot, the one or more processors are configured to enable the vehicle or robot to navigate in an environment.

[0106] Clause 21. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: perform a first segmentation operation on a previous frame of image data to generate first segmented data; perform a deformable convolution operation based on the first segmented data to generate a deformable convolution output; perform a second segmentation operation on a current frame of the image data to generate second segmented data; combine the deformable convolution output with the second segmented data to generate third segmented data; and control the operation of a device based on the third segmented data.

[0107] Clause 22. The storage medium according to Clause 21, wherein at least one of the first segmentation operation or the second segmentation operation is a semantic segmentation operation.

[0108] Clause 23. The storage medium pursuant to Clause 21 or Clause 22, wherein the instructions further cause the one or more processors to use a machine learning model to perform the first partitioning operation and the second partitioning operation.

[0109] Clause 24. The storage medium according to any one of Clauses 21 to 23, wherein the deformable convolution operation is further based on an offset convolution operation and a modulation convolution operation.

[0110] Clause 25. The storage medium according to Clause 24, wherein the instructions further cause the one or more processors to: extract features from the previous frame of the image data to generate a first extracted feature; extract features from the current frame of the image data to generate a second extracted feature; concatenate the first extracted feature, the second extracted feature, and the difference between the second extracted feature and the first extracted feature to generate a concatenated extracted feature; concatenate the previous frame of the image data, the current frame of the image data, and the difference between the current frame of the image data and the previous frame of the image data to generate concatenated image data; and extract features from the concatenated image data to generate a third extracted feature, wherein the instructions cause the one or more processors to perform the offset convolution operation on the third extracted feature and the concatenated extracted feature, and wherein the instructions cause the one or more processors to perform the modulation convolution operation on the third extracted feature and the concatenated extracted feature.

[0111] Clause 26. The storage medium according to Clause 25, wherein the first extracted feature includes the output of the encoder of a first machine learning model, the second extracted feature includes the output of the encoder of a second machine learning model, and the third extracted feature includes the output of the encoder of a third machine learning model.

[0112] Clause 27. The storage medium according to Clause 25 or Clause 26, wherein the deformable convolution operation is a first deformable convolution operation, the deformable convolution output is a first deformable convolution output, and wherein the instructions further cause the one or more processors to: determine a first loss based on a second extracted feature and a second deformable convolution output, wherein the second deformable convolution output is based on the first extracted feature, the offset convolution output, and the modulation convolution output; and train at least one of a machine learning model of the offset convolution operation, a machine learning model of the modulation convolution operation, or a machine learning model of the first deformable convolution operation based on the first loss.

[0113] Clause 28. The storage medium pursuant to Clause 27, wherein the instructions further cause the one or more processors to: determine a second loss based on the second segmented data; and train at least one of the machine learning model of the offset convolution operation, the machine learning model of the modulation convolution operation, or the machine learning model of the first deformable convolution operation based on the second loss.

[0114] Clause 29. The storage medium pursuant to Clause 27 or Clause 28, wherein the one or more processors are further configured to: determine a third loss based on the deformable convolution output; and train at least one of the machine learning model of the offset convolution operation, the machine learning model of the modulation convolution operation, or the machine learning model of the first deformable convolution operation based on the third loss.

[0115] Clause 30. A system comprising: means for performing a first segmentation operation on a previous frame of image data to generate first segmented data; means for performing a deformable convolution operation based on the first segmented data to generate a deformable convolution output; means for performing a second segmentation operation on a current frame of the image data to generate second segmented data; means for combining the deformable convolution output with the second segmented data to generate third segmented data; and means for controlling operation of a device based on the third segmented data.

[0116] It should be recognized that, based on the examples, certain actions or events of any technique described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary for implementing the technique). Furthermore, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt handling, or multiple processors) rather than sequentially.

[0117] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0118] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs utilize lasers to optically reproduce data. The combinations described above should also be included within the scope of computer-readable media.

[0119] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0120] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Rather, as described above, various units may be combined in a codec hardware unit, or various units may be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0121] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A system comprising: One or more memories, the one or more memories being configured to store image data captured by at least one camera; and One or more processors, communicatively coupled to one or more memories, are configured to: Perform a first segmentation operation on the previous frame of the image data to generate first segmentation data; Perform deformable convolution operations based on the first segmented data to generate deformable convolution output; Perform a second segmentation operation on the current frame of the image data to generate second segmentation data; The deformable convolution output is combined with the second segmentation data to generate the third segmentation data; as well as The operation of the device is controlled based on the third segmented data.

2. The system according to claim 1, wherein at least one of the first segmentation operation or the second segmentation operation is a semantic segmentation operation.

3. The system of claim 1, wherein the one or more processors are configured to use a machine learning model to perform the first segmentation operation and the second segmentation operation.

4. The system of claim 1, wherein the deformable convolution operation is further based on an offset convolution operation and a modulation convolution operation.

5. The system of claim 4, wherein the one or more processors are further configured to: Features are extracted from the previous frame of the image data to generate first extracted features; Features are extracted from the current frame of the image data to generate second extracted features; The first extracted feature, the second extracted feature, and the difference between the second extracted feature and the first extracted feature are concatenated to generate a concatenated extracted feature; The previous frame of the image data, the current frame of the image data, and the difference between the current frame of the image data and the previous frame of the image data are concatenated to generate concatenated image data; as well as Features are extracted from the stitched image data to generate a third extracted feature. The one or more processors are configured to perform the offset convolution operation on the third extracted feature and the concatenated extracted feature, and The one or more processors are configured to perform the modulation convolution operation on the third extracted feature and the concatenated extracted feature.

6. The system of claim 5, wherein the first extracted feature includes the output of the encoder of the first machine learning model, the second extracted feature includes the output of the encoder of the second machine learning model, and the third extracted feature includes the output of the encoder of the third machine learning model.

7. The system of claim 5, wherein the deformable convolution operation is a first deformable convolution operation, the deformable convolution output is a first deformable convolution output, and wherein the one or more processors are further configured to: A first loss is determined, the first loss being based on the second extracted features and the second deformable convolution output, wherein the second deformable convolution output is based on the first extracted features, the output of the offset convolution operation, and the output of the modulation convolution operation; and At least one of the machine learning model for the offset convolution operation, the machine learning model for the modulation convolution operation, or the machine learning model for the first deformable convolution operation is trained based on the first loss.

8. The system of claim 7, wherein the one or more processors are further configured to: Determine a second loss, which is based on the second segmentation data; and The machine learning model for the offset convolution operation, the machine learning model for the modulation convolution operation, or the machine learning model for the first deformable convolution operation is trained based on the second loss.

9. The system of claim 8, wherein the one or more processors are further configured to: Determine a third loss, said third loss based on the deformable convolution output; and The machine learning model for the offset convolution operation, the machine learning model for the modulation convolution operation, or the machine learning model for the first deformable convolution operation is trained based on the third loss.

10. The system of claim 1, wherein the device includes a vehicle or robot, and wherein, as part of controlling the operation of the vehicle or robot, the one or more processors are configured to enable the vehicle or robot to navigate in an environment.

11. A method, the method comprising: Perform a first segmentation operation on the previous frame of the image data to generate first segmentation data; Perform deformable convolution operations based on the first segmented data to generate deformable convolution output; Perform a second segmentation operation on the current frame of the image data to generate second segmentation data; The deformable convolution output is combined with the second segmentation data to generate the third segmentation data; as well as The operation of the device is controlled based on the third segmented data.

12. The method according to claim 11, wherein at least one of the first segmentation operation or the second segmentation operation is a semantic segmentation operation.

13. The method of claim 11, wherein performing the first segmentation operation and performing the second segmentation operation includes using a machine learning model.

14. The method of claim 11, wherein performing the deformable convolution operation is further based on an offset convolution operation and a modulation convolution operation.

15. The method according to claim 14, further comprising: Features are extracted from the previous frame of the image data to generate first extracted features; Features are extracted from the current frame of the image data to generate a second extracted feature; The first extracted feature, the second extracted feature, and the difference between the second extracted feature and the first extracted feature are concatenated to generate a concatenated extracted feature; The previous frame of the image data, the current frame of the image data, and the difference between the current frame of the image data and the previous frame of the image data are concatenated to generate concatenated image data; Features are extracted from the stitched image data to generate a third extracted feature; Perform the offset convolution operation on the third extracted feature and the concatenated extracted features; and The modulation convolution operation is performed on the third extracted feature and the concatenated extracted feature.

16. The method of claim 15, wherein the first extracted feature includes the output of the encoder of a first machine learning model, the second extracted feature includes the output of the encoder of a second machine learning model, and the third extracted feature includes the output of the encoder of a third machine learning model.

17. The method of claim 15, wherein the deformable convolution operation is a first deformable convolution operation, the deformable convolution output is a first deformable convolution output, and the method further comprises: A first loss is determined, the first loss being based on the second extracted features and the second deformable convolutional output, wherein the second deformable convolutional output is based on the first extracted features, the offset convolutional output, and the modulation convolutional output; as well as At least one of the machine learning model for the offset convolution operation, the machine learning model for the modulation convolution operation, or the machine learning model for the first deformable convolution operation is trained based on the first loss.

18. The method according to claim 17, further comprising: A second loss is determined, which is based on the second segmentation data; as well as The machine learning model for the offset convolution operation, the machine learning model for the modulation convolution operation, or the machine learning model for the first deformable convolution operation is trained based on the second loss.

19. The method according to claim 18, further comprising: A third loss is determined, which is based on the deformable convolution output; as well as The machine learning model for the offset convolution operation, the machine learning model for the modulation convolution operation, or the machine learning model for the first deformable convolution operation is trained based on the third loss.

20. The method of claim 11, wherein the device comprises a vehicle or a robot, and wherein, as part of controlling the operation of the vehicle or the robot, the one or more processors are configured to enable the vehicle or the robot to navigate in an environment.

21. A non-transitory computer-readable storage medium storing instructions, which, when executed, cause one or more processors to: Perform a first segmentation operation on the previous frame of the image data to generate first segmentation data; Perform deformable convolution operations based on the first segmented data to generate deformable convolution output; Perform a second segmentation operation on the current frame of the image data to generate second segmentation data; The deformable convolution output is combined with the second segmentation data to generate the third segmentation data; as well as The operation of the device is controlled based on the third segmented data.

22. The storage medium of claim 21, wherein at least one of the first segmentation operation or the second segmentation operation is a semantic segmentation operation.

23. The storage medium of claim 21, wherein the instructions further cause the one or more processors to use a machine learning model to perform the first partitioning operation and the second partitioning operation.

24. The storage medium of claim 21, wherein the deformable convolution operation is further based on an offset convolution operation and a modulation convolution operation.

25. The storage medium of claim 24, wherein the instructions further cause the one or more processors to: Features are extracted from the previous frame of the image data to generate first extracted features; Features are extracted from the current frame of the image data to generate a second extracted feature; The first extracted feature, the second extracted feature, and the difference between the second extracted feature and the first extracted feature are concatenated to generate a concatenated extracted feature; The previous frame of the image data, the current frame of the image data, and the difference between the current frame of the image data and the previous frame of the image data are concatenated to generate concatenated image data; as well as Features are extracted from the stitched image data to generate a third extracted feature. The instructions wherein the one or more processors perform the offset convolution operation on the third extracted feature and the concatenated extracted features, and The instructions wherein the one or more processors perform the modulation convolution operation on the third extracted feature and the concatenated extracted feature.

26. The storage medium of claim 25, wherein the first extracted feature includes the output of the encoder of a first machine learning model, the second extracted feature includes the output of the encoder of a second machine learning model, and the third extracted feature includes the output of the encoder of a third machine learning model.

27. The storage medium of claim 25, wherein the deformable convolution operation is a first deformable convolution operation, the deformable convolution output is a first deformable convolution output, and wherein the instructions further cause the one or more processors to: A first loss is determined, the first loss being based on the second extracted features and the second deformable convolutional output, wherein the second deformable convolutional output is based on the first extracted features, the offset convolutional output, and the modulation convolutional output; and At least one of the machine learning model for the offset convolution operation, the machine learning model for the modulation convolution operation, or the machine learning model for the first deformable convolution operation is trained based on the first loss.

28. The storage medium of claim 27, wherein the instructions further cause the one or more processors to: Determine a second loss, which is based on the second segmentation data; and The machine learning model for the offset convolution operation, the machine learning model for the modulation convolution operation, or the machine learning model for the first deformable convolution operation is trained based on the second loss.

29. The storage medium of claim 28, wherein the one or more processors are further configured to: Determine a third loss, said third loss based on the deformable convolution output; and The machine learning model for the offset convolution operation, the machine learning model for the modulation convolution operation, or the machine learning model for the first deformable convolution operation is trained based on the third loss.

30. A system comprising: A component for performing a first segmentation operation on a previous frame of image data to generate first segmented data; A component for performing deformable convolution operations based on the first segmented data to generate deformable convolution output; A component for performing a second segmentation operation on the current frame of the image data to generate second segmentation data; A component for combining the deformable convolution output with the second segmented data to generate the third segmented data; and Components used to control the operation of the device based on the third segmented data.

Images