Autoencoder with non-uniform spreading recursion

Abstract

A non-uniform video encoder system for generating multi-depth encoded data of a scene is provided. The non-uniform video encoder system is configured to receive a sequence of video frames of a video of a scene and transform the sequence of video frames into sequence input data. The sequence input data is analyzed to identify changes in scene evolution by dividing the sequence input data into a series of non-uniform segments. Each segment in the series of non-uniform segments is encoded by an encoder of an autoencoder architecture with a non-uniform unrolling recursion to produce multi-depth encoding of the sequence input data. To encode a current segment at a current iteration to produce a current encoding, the non-uniform unrolling recursion combines the current segment with a previous encoding produced at a previous iteration and encodes the combination with the encoder.

Description

Technical Field

[0001] This disclosure generally relates to robot control, and more specifically, to training and controlling robots to perform tasks based on hierarchical scene graph processing. Background Technology

[0002] The goal of robotics and artificial intelligence (AI) is to create robotic agents that can coexist with, assist, and interact naturally with humans. With the development of deep neural networks, agents or robots capable of autonomously navigating realistic 3D environments to solve real-world tasks have been constructed. As examples, tasks could involve audio-target navigation (i.e., visual navigation for locating objects emitting sounds in the environment), vision and language navigation (VLN), i.e., navigating to a target location following instructions provided in natural language or exploring the visual world seeking answers in given natural language.

[0003] However, robots deployed and operated in both real and virtual worlds may not be able to reliably navigate through such environments. To address this shortcoming, reinforcement learning (RL) policies can be trained to navigate using the 3D spatial directionality of the visual environment and audio (if available).

[0004] Efficiently searching for objects of interest in natural 3D environments is a crucial capability that autonomously implemented robotic agents must possess. Some example tasks of this nature include: (i) searching for trapped people at disaster sites, (ii) searching for lost objects in factory environments, and (iii) searching for, then picking up and placing objects in a rearrangement task, and so on. Robots deployed to solve such real-world tasks have learned policies (or inductive biases) to develop efficient and effective strategies for solving them. We also expect such learned policies to generalize to new environments that may differ from the training data.

[0005] Some tasks use the RGB and depth maps of a scene (views from an agent) as input to a learning model to learn navigation strategies (via reinforcement learning or imitation learning), for example, to decide where to move next to search for objects. However, such input (with raw pixels capturing the scene) is not very accurate in learning which scene regions to use to generate an effective navigation strategy, and may therefore require huge training sets and millions of training episodes. One approach to address this scenario is to extract relevant semantic details from the scene and then train the agent using only a subset of image regions, resulting in faster training. A standard way to sparsely model scenes (without sacrificing semantic content) is via scene graph representation. In this graph representation of a scene, the nodes of the graph correspond to objects in the scene generated using an object detector (which is trained to detect objects of interest in the scene) and a relation detector that can capture 3D spatial semantic relationships between each node (e.g., a table is "behind" a chair, etc.). Furthermore, representing a scene as a semantic graph allows for unwinding pixel details and abstracting the scene with higher semantic granularity to improve generalization.

[0006] While scene graphs may seem like a useful representation, they are not advantageous for concrete implementation settings, especially when the appearance of the scene changes significantly with each agent move. As a result, a scene graph is needed for every agent move, causing the number of such graphs per turn to increase linearly with the number of navigation steps. This linear increase leads to a quadratic increase in the number of graph edges, thus slowing down graph inference. Furthermore, constructing scene graphs requires performing object detectors on the scene view, which can potentially slow down decision-making during navigation in concrete implementations.

[0007] Therefore, a computationally efficient and feasible solution is needed to circumvent these problems and provide efficient robot agent control for different tasks. Summary of the Invention

[0008] Some implementations aim to provide a solution to the efficient proxy navigation problem using computationally efficient and feasible navigation methods.

[0009] Some implementations are based on the understanding that artificial intelligence systems (such as neural network-based systems) are capable of learning complex patterns and relationships in visual data, thereby allowing them to efficiently construct and understand scene graphs. The choice of a specific neural network architecture and method depends on the complexity of the task and the desired level of detail in the scene representation.

[0010] Some implementations are based on the understanding that the performance of a neural network depends on the amount of data provided for processing. Very large and frequently changing datasets present challenges related to the computation time and training efficiency of neural networks. These challenges are further amplified when processing sequential data that expands in time and / or space (such as in the visual scenario described above).

[0011] Other examples of such time-series data include video and / or audio signals, GNSS measurements, data packet switching, etc. Even when not processing the raw time-series data, but rather extracting features from it for subsequent processing by downstream neural networks, the sheer volume of extracted features can prevent their joint processing due to computational and storage constraints. To address this, many different applications use a so-called sliding window method to process time-series data partially (i.e., segmentally). This method segments the data, but it is acceptable when the task of the downstream neural network is based on local analysis of the data (such as generating captions for only parts of a video scene). However, when the task is global, such as generating a summary or key points of the entire video, this segmentation alone may be suboptimal.

[0012] The same problem exists in applications dealing with spatial data representing complex environments. For example, reinforcement learning (RL), a field of machine learning, involves concepts about how intelligent agents should act in an environment to maximize cumulative rewards. The concept of environment differs for different applications, but for many practical applications, such as robot control or drone navigation, the size of the environment can be very large for a complete consideration at once. Consider, for example, a robot search operation within a multi-story building. With the limited resources of a search agent (such as a search robot), maintaining a complete map of the building to make decisions at every step of control is computationally expensive.

[0013] Therefore, some embodiments aim to provide systems and methods suitable for downstream neural network processing for representing large amounts of data into a compact and fixed-size representation. Additionally or alternatively, some embodiments aim to disclose systems and methods that can encode incoming sequence data that expands temporally and / or spatially into a fixed-size global representation. Some embodiments address this problem by utilizing an unfolded recursive autoencoder.

[0014] An autoencoder is an artificial neural network used to learn efficient encodings (unsupervised learning) of unlabeled data. An autoencoder learns two functions: an encoding function that transforms the input data and a decoding function that recreates the input data from the encoded representation. Autoencoders learn efficient representations (encodings) of datasets and are often used for dimensionality reduction.

[0015] The encoder's output (i.e., encoding the input data into the latent space) can be forced to a fixed size. The encoding in the latent space may not have physical meaning, but due to the principle of autoencoders, the latent space preserves the original information in a way that allows the decoder to decode it into the original space.

[0016] Some implementations are based on the understanding that the principles of autoencoders can be extended to recursive encoding in the latent space. Specifically, encodings in the latent space can be encoded again by the same encoder and then recursively decoded by the autoencoder's decoder. For example, if input data is encoded twice by an encoder of an autoencoder architecture, the data encoded twice can have the same dimension in the latent space, regardless of the number of encodings, and can be recursively decoded by executing the autoencoder architecture's decoder twice.

[0017] Some implementations are based on another understanding: the recursive encoding of an autoencoder can be used by downstream neural networks even without subsequent recursive decoding. It should be recognized that this is one of the advantages of the autoencoder paradigm in encoding in a way that allows decoding of the encoding to retrieve the original input. As a result, the recursive encoding retains enough information for decoding and can therefore be used by downstream neural networks with and / or without decoding. However, recursive autoencoders offer little benefit for many practical applications because they do not necessarily make the data more compact or conducive to subsequent processing.

[0018] However, some implementations are based on the understanding that the same fundamental principles used for recursive autoencoders are effective for autoencoders utilizing unwinding recursion, where the encoder's input consists of raw (unencoded) data and data previously encoded by the encoder. Unwinding recursion is performed by combining segments of the input stream with previously encoded data in the latent space and encoding this combination into the latent space, repeating this process until a termination condition is met.

[0019] As a result, the encoder's output includes codes of varying depths, and it can encode increasingly new incoming data into multi-depth codes of the same fixed size. Furthermore, these multi-depth codes can be submitted to downstream neural networks without recursive decoding, allowing downstream neural networks to perform tasks involving processing lengthy data of unknown lengths by processing fixed-length multi-depth codes.

[0020] Additionally or alternatively, some implementations are based on the recognition that the size or amount of the original uncoded data to be combined with previously encoded data can vary between different iterations. It is recognized that this non-uniform unwinding recursion does not compromise the performance guarantees of the autoencoder architecture. Furthermore, allowing the size of the uncoded data to change in different encoding iterations can give each encoding additional meaning specific to and / or beneficial to downstream applications.

[0021] For example, in downstream applications related to navigation, the uncoded data in each encoding iteration can represent a space with specific semantic meaning. Examples of such spaces include rooms, streets, towns, etc. For instance, in one implementation where the semantic space is rooms and the downstream application is robot navigation within a building with multiple rooms, each encoding iteration includes adding uncoded features of the rooms (e.g., represented as a local scene map) to previously encoded information indicating the features of multiple rooms in the building that the robot has traversed. Doing so in this way allows the navigation application to rely on additional semantic meaning during each decoding. However, different rooms can include different numbers of objects and local scene maps of different sizes representing different objects in different rooms. Therefore, some implementations use non-uniform unfolding recursion to capture variations in the data representing different rooms.

[0022] Some implementations are based on the understanding that, using the autoencoder architecture described above, a computationally feasible representation can be used to abstract the scene graph of a room at different points in time. Some implementations provide solutions involving the hierarchy of scene graph representations. In some implementations, a three-step approach is used to obtain the hierarchy of scene graph representations. The three-step approach includes the following steps: (i) for each move of the agent, the pose difference of the agent (with its previous pose) is used to determine whether to construct a new local scene graph of the scene; (ii) if the pose changes significantly, the agent computes the local scene graph and then uses the pose to register the graph with the global scene graph by matching overlapping objects in the new view with objects already existing in the global graph (using their 3D proximity). This registration allows the addition of new nodes in the graph that do not exist in the global scene graph. While these two steps avoid redundant nodes in the graph, the graph size can still grow significantly when the agent explores large areas. To address this, (iii) the third step involves abstracting the global scene graph into supernodes based on predefined criteria. For example, if the number of nodes in the global scene graph exceeds a threshold, a supernode algorithm is invoked.

[0023] Some implementations address an alternative scenario based on the agent taking a certain number of steps (i.e., a fixed-size time motion window). Therefore, some implementations provide a supernode generation algorithm implemented using an unfolded recursive graph neural autoencoder that takes the scene graph as input and generates feature vectors in a suitable latent space, which can be decoded back to its input scene graph. These embedded features summarize the basic properties of the graph nodes and their semantic relationships. After this supernode construction, for the agent's next move, a new local scene graph is constructed, where additional nodes correspond to the supernodes computed in previous steps. The supernodes are fully connected to all nodes in the local graph, and the graph construction process recursively creates supernodes according to the predefined criteria described above.

[0024] Some implementations are based on the understanding that abstracting scene details in supernodes hierarchically without losing information (via an autoencoder), while implicitly using that information for reasoning, allows for limiting the size of the graph, making reasoning computationally efficient and feasible.

[0025] In another example of speech recognition, uncoded data of varying sizes can come from spoken utterances of different lengths, such as sentences. In another example of video processing, uncoded data of varying sizes can come from scenes of varying durations. In different implementations, the uncoded input data is segmented into distinct non-uniform semantic segments based on rules that favor the downstream neural network.

[0026] Therefore, one embodiment discloses a non-uniform video encoder system. The non-uniform video encoder system includes: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the non-uniform video encoder system to receive a sequence of video frames of a scene. The non-uniform video encoder system is also configured to transform the sequence of video frames into time-series input data indicating the evolution of the scene in time, space, or both. By dividing the time-series input data into a sequence of non-uniform segments, the time-series input data is analyzed to identify changes in scene evolution. Each segment in the non-uniform segment sequence is encoded by an encoder utilizing a non-uniform unrolling recursive autoencoder architecture to produce a multi-depth code of the time-series input data. To this end, in order to encode the current segment at the current iteration to produce the current code, the non-uniform unrolling recursion combines the current segment with a previous code generated at a previous iteration, and encodes the combination using the encoder. The multi-depth code of the sequence input data is output accordingly.

[0027] Another embodiment discloses a controller for controlling a robot to perform tasks. The controller is configured to receive a sequence of video frames of a scene. The controller is configured to transform the video frame sequence into time-series input data indicating the evolution of the scene in time, space, or both. The controller is also configured to analyze the time-series input data to identify changes in scene evolution by dividing the sequence input data into a sequence of non-uniform segments. The controller is further configured to encode each segment in the non-uniform segment sequence by an encoder utilizing a non-uniform unrolling recursive autoencoder architecture to produce a multi-depth code for the time-series input data. To encode the current segment at the current iteration to produce the current code, the non-uniform unrolling recursion combines the current segment with a previous code generated at a previous iteration, and encodes the combination using the encoder. Furthermore, the controller is configured to output the multi-depth code for the time-series input data.

[0028] Another embodiment discloses a non-transitory computer-readable storage medium having a program specifically implemented thereon that can be executed by a processor to perform a method comprising: receiving a video frame sequence of a scene; transforming the video frame sequence into time-series input data indicating the evolution of the scene in time, space, or both; analyzing the time-series input data to identify changes in the scene's evolution by dividing the time-series input data into a non-uniform segment sequence; encoding each segment in the non-uniform segment sequence by an encoder utilizing a non-uniform unfolding recursive autoencoder architecture to generate a multi-depth code of the time-series input data, wherein, in order to encode the current segment at the current iteration to generate the current code, the non-uniform unfolding recursion combines the current segment with a previous code generated at a previous iteration, and encodes the combination using an encoder; and outputting the multi-depth code of the time-series input data.

[0029] The currently disclosed embodiments will be further explained with reference to the accompanying drawings. The drawings shown are not necessarily drawn to scale, but generally focus on illustrating the principles of the currently disclosed embodiments. Attached Figure Description

[0030] [ Figure 1 ]

[0031] Figure 1 A block diagram of a non-uniform video encoder system according to some embodiments of the present disclosure is shown.

[0032] [ Figure 2 ]

[0033] Figure 2 A block diagram illustrating operations performed by a non-uniform video encoder system for processing a frame sequence according to an embodiment of the present disclosure is shown.

[0034] [ Figure 3 ]

[0035] Figure 3 This is a block diagram illustrating details of the components of a non-uniform video encoder system according to an example embodiment.

[0036] [ Figure 4 ]

[0037] Figure 4 A block diagram of a scene diagram generation pipeline according to an embodiment of the present disclosure is shown.

[0038] [ Figure 5 ]

[0039] Figure 5 A block diagram illustrating the operation of a non-uniform video encoder based on scene graph input according to an embodiment of the present disclosure is shown.

[0040] [ Figure 6 ]

[0041] Figure 6 A method for generating a recursive graph using the SuGE model according to an embodiment of this disclosure is shown.

[0042] [ Figure 7 ]

[0043] Figure 7 A schematic diagram illustrating the merging of supernodes and local graphs according to an embodiment of the present disclosure is shown.

[0044] [ Figure 8 ]

[0045] Figure 8 A schematic diagram of a use case for performing downstream tasks of a non-uniform video encoder system according to an embodiment of the present disclosure is shown.

[0046] [ Figure 9 ]

[0047] Figure 9 A schematic diagram is shown of a use case according to an embodiment of the present disclosure, including an automatic speech recognition (ASR) task performed by a downstream neural network using a non-uniform video encoder system.

[0048] [ Figure 10 ]

[0049] Figure 10 A schematic diagram of a use case of a non-uniform video encoder system for performing downstream navigation tasks according to an embodiment of the present disclosure is shown.

[0050] [ Figure 11 ]

[0051] Figure 11 This is a schematic diagram illustrating a computing device for implementing a non-uniform video encoder system according to an embodiment of the present disclosure. Detailed Implementation

[0052] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, apparatus and methods are shown only in block diagram form to avoid obscuring this disclosure. Various changes to the function and arrangement of the elements will be contemplated without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims.

[0053] As used in this specification and claims, the terms “for example,” “like,” and “such as,” as well as the verbs “comprising,” “having,” “including,” and other verb forms thereof, when used in conjunction with a list of one or more components or other items, are each interpreted as open-ended, meaning that the list is not considered to exclude other additional components or items. The term “based on” means at least partially based on. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0054] Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Furthermore, the same reference numerals and names in the various figures indicate the same elements.

[0055] Some embodiments aim to disclose a non-uniform video encoder system that processes sequential data extended in time, space, or both in a computationally feasible manner. To this end, the non-uniform video encoder system includes an autoencoder that performs non-uniform unfolding recursion to efficiently process input data and generate output that can be used by another computational system, such as a downstream neural network for performing a task. A detailed description of the non-uniform video encoder system and its various applications is provided in the following disclosure and suitable accompanying drawings.

[0056] Figure 1 A block diagram 100 of a non-uniform video encoder system 106 that performs a task using non-uniform unrolling recursion is shown. The non-uniform video encoder system 106 can be implemented as a computing system or can be specifically implemented within a controller. For this purpose, the non-uniform video encoder system 106 may include at least one processor; and a memory thereon storing instructions that, when executed by the at least one processor, cause the non-uniform video encoder system 106 to perform a series of operations to enable the execution of the task.

[0057] For this purpose, the non-uniform video encoder system 106 receives a sequence of video frames 102 of the scene as input. For example, the scene can be captured using RGB images and depth maps associated with a specific navigation task. Alternatively, the non-uniform video encoder system 106 transmits data as video and / or audio signals, GNSS measurements, data packet exchange, etc. In some examples, the scene may be associated with the execution of a task, such as searching for trapped people at a disaster site, searching for lost objects in a factory environment, and picking up and placing objects in a rearrangement task after a search. The task may be performed by an agent (such as a robot) equipped with a controller that generates control commands to control the robot to perform tasks related to the robot's specific navigation implementation within the scene.

[0058] In some implementations, the video frame sequence 102 is transformed into sequential input data indicating the evolution of a scene in time, space, or both. To this end, each frame of the video frame sequence 102 is first extracted from the video of the scene. Furthermore, from each frame, relevant features are extracted using a learning model or neural network, or any processing algorithm configured for feature extraction. Subsequently, the extracted features are organized, such as by arranging the extracted features or pixel values ​​from each frame in a sequential manner, to create sequential input data. The sequential input data includes rows of specific time points (frames) and columns of features or pixel values.

[0059] Furthermore, the sequence input data is analyzed to identify changes in scene evolution by dividing it into a sequence of non-uniform data segments. The divided sequence input data in the form of non-uniform segments is sent to the non-uniform video encoder system 106. In one embodiment, the non-uniform segments are sent to the non-uniform video encoder system 106 via a network 104. The network 104 can be any combination of communication networks including local area networks (“LANs”) and wide area networks (“WANs”) (e.g., the Internet, etc.).

[0060] Some implementations are based on the recognition that non-uniform partitioning of the sequence input data adds additional flexibility and adaptability to the encoding algorithm. For example, different segments of different sizes can encode different kinds of objects in a scene and / or different parts of the scene with the same semantic meaning. Allowing the partitioning of the input sequence data to be non-uniform enables some implementations to perform semantically meaningful partitioning, such as partitioning into non-uniform segments that indicate changes in scene evolution, such as moving from one room to another in a building. For example, in some implementations, changes in scene evolution are identified by one or a combination of: events detected in the scene, changes in color patterns in the scene, changes in captions describing the scene, changes in the scene's classification results, anomalies detected in the scene, acoustic events detected in the scene, or events associated with a camera that captures the scene's evolution using a sequence of video frames.

[0061] Therefore, the non-uniform video encoder system 106 is configured to encode each segment in a non-uniform segment sequence using an encoder that utilizes a non-uniform unrolling recursive autoencoder architecture to produce a multi-depth code for the sequence input data. To encode the current segment at the current iteration to produce the current code, the non-uniform unrolling recursion combines the current segment with the previous code generated at the previous iteration and encodes the combination using the encoder. The resulting multi-depth code for the sequence input data is then sent as the output from the non-uniform video encoder system 106. The output can also be used for further processing and applications, including performing tasks related to scene perception. For example, a downstream neural network can be used to consume the multi-depth code of the scene and use it to enable task execution. A scene represents the environment in which an agent (such as a robot) performs a task. The concept of environment varies for different applications, but for many practical applications, such as robot control or drone navigation, the size of the environment makes it very large for full consideration. For example, a robot search operation within a multi-story building requires maintaining a complete map of the building to make decisions at each step of control, which is computationally expensive for the search robot's limited resources.

[0062] To this end, the non-uniform video encoder system 106 is configured to provide a system and method for representing large amounts of data into a compact and fixed-size representation suitable for processing by a downstream neural network. Additionally or alternatively, the non-uniform video encoder system 106 is configured to encode temporally and / or spatially expanded incoming sequence data into a fixed-size global representation. Some implementations address this problem by utilizing an unfolded recursive autoencoder. Thus, the autoencoder encodes input data in the form of a video frame sequence 102 into a latent representation by semantically combining the input data at the current time step with the output of the previously encoded encoder at the previous time step, based on a predefined criterion at least associated with the size of the input data, to produce a compressed latent representation of the input data as output. The compressed latent representation of the input data includes multi-depth encoding of the input data. Multi-depth encoding of input data that is inherently time-series data typically refers to a hierarchical or multi-layered representation of the input time-series data. Each layer in this encoding captures different levels of abstraction or features from the time series, starting from low-level features (e.g., simple patterns) and progressing to high-level features (e.g., complex patterns or global trends).

[0063] In the context of autoencoders, a type of neural network used for unsupervised learning and dimensionality reduction, multi-depth coding involves using multiple layers in the encoder portion of the autoencoder to progressively learn increasingly abstract and complex representations of the input time series. In some implementations of autoencoders utilizing multi-depth coding, the encoder includes multiple hidden layers stacked on top of each other, such that each hidden layer learns increasingly abstract and complex features of the input data, thus forming a hierarchical structure of representations. Therefore, the final hidden layer of the encoder provides a compressed latent representation of the input data, which is produced as the output of the non-uniform video encoder system 106. This output is then passed to downstream neural networks for further processing.

[0064] Downstream neural networks are tailored for specific types of applications related to the execution of tasks. For example, tasks may be related to speech processing, specific navigation implementations, GNSS measurements, automation in manufacturing settings, etc. Figure 2 A block diagram is provided describing the operations performed by the non-uniform video encoder system 106 in processing the frame sequence 102 in the manner described above.

[0065] Figure 2 A block diagram 200 is shown illustrating operations performed by a non-uniform video encoder system 106 for processing a frame sequence 102 according to an embodiment of the present disclosure.

[0066] At position 202, a sequence of video frames is received. This sequence corresponds to video of a scene in which an agent (such as a robot) is controlled to perform a task. For example, the task is specifically implemented as robot navigation. This implemented navigation may optionally be extended through the robot's interaction with its environment, which is captured as a scene using various sensors capable of perceiving the environment. Examples of various sensors include RGB cameras, depth cameras, LiDAR units, etc. The video of the scene is captured as a sequence of video frames using these various sensors.

[0067] At position 204, the video frame sequence is transformed into sequential input data. The transformation includes processes such as extracting frames from the video, extracting features from each individually extracted frame, and sequentially organizing the extracted features to form the sequential input data. Therefore, the sequential input data includes features (such as motion features) that expand in time, space, or both as the scene evolves.

[0068] At point 206, the sequence input data is analyzed to identify changes in scene evolution. This is done by dividing the sequence input data into a non-uniform sequence of segments. This division can be performed using video slicing algorithms, such as the sliding window method. As a result, the sequence input data can be analyzed and sent for further processing in a segment-by-segment manner.

[0069] At position 208, each segment is then encoded by an encoder utilizing a non-uniform unfolding recursive autoencoder architecture. Non-uniform unfolding recursion is the operation of generating multi-depth codes for a sequence of input data generated at position 204 and fragmented at position 206. Non-uniform unfolding recursion is performed by encoding the current segment at the current iteration to produce the current code, then combining the current segment with the previous code generated at a previous iteration, and encoding the combination of the current segment and the previous code using the encoder. This is, for example, in... Figure 7 As shown in the figure. This combination of encodings thus produces multi-depth encodings of the sequence input data, where different layers are formed by different combinations of multi-depth encodings of the features of the sequence input data.

[0070] At position 210, the multi-depth code is output by the non-uniform video encoder system 106. This multi-depth code can be used by a downstream neural network to perform tasks. These tasks can include video processing, audio processing, customized navigation, manufacturing control, UAV navigation and control, anomaly detection, etc.

[0071] Operations 202-210 are executed by a processor, which executes computer-readable instructions that define each of operations 202-210 in the form of a computer program, computer code, computer algorithm, etc. These computer-readable instructions may be stored in a non-transitory computer-readable storage medium in the form of a program executable by the processor to perform all the operations shown in block diagram 200. In one embodiment, the processor and memory are part of a non-uniform video encoder system 106 configured to perform non-uniform unrolling recursion, as... Figure 3 As shown.

[0072] Figure 3 This is a block diagram 300 illustrating details of the components of a non-uniform video encoder system 106 according to an example embodiment. The non-uniform video encoder system 106 includes an autoencoder 302, which includes an encoder 304 that generates an encoding 306 of data input to the autoencoder 302 in a latent space. The autoencoder 302 also includes a decoder 308 to decode the encoding 306 generated by the encoder 304. During the inference phase of the autoencoder 302, the decoder module 308 may be absent and / or moved to downstream applications.

[0073] The autoencoder 302 is an artificial neural network used to learn efficient encodings (unsupervised learning) of unlabeled data. The autoencoder 302 learns two functions: an encoding function for encoder 304 that transforms the input data of the autoencoder 302; and a decoding function for decoder 308 that recreates the input data from the encoded representation or encoding 306 produced by encoder 304 in the latent space. The autoencoder 302 learns efficient representations (encodings) of datasets, often used for dimensionality reduction.

[0074] The output of encoder 304 (i.e., the input data encoded 306 in the latent space) can be forced to a fixed size. The encoding 306 in the latent space may not have physical meaning, but due to the principles of autoencoders, this latent space preserves the original information in a way that allows decoder 308 to decode it into the original space of the input data. Therefore, for the non-uniform video encoder system 106, the input data of autoencoder 302 includes segments 102a of a non-uniform segment sequence of the input sequence of video frames 102. Segment 102a is encoded by encoder 304 using non-uniform unrolling recursion. Non-uniform unrolling recursion involves encoding segment 102a at the current time instance t, which, for simplicity, will be referred to as the current segment 102a, to produce encoding 306 at the current time instance t. Thus, encoding 306 becomes current encoding 306. Encoder 304 also produces previous encoding 310 during previous iterations of autoencoder 302 (such as at time instance t-1). Therefore, the current code 306 is generated by combining the current segment 102a with the previous code 310 and encoding the combination using the encoder 304. This encoding and combination process can be iteratively repeated by the autoencoder 302 until a termination condition is met. When the termination condition is met, the iterative operation of the autoencoder 302 terminates 316, and a multi-depth code 306a is generated as the output 106a of the non-uniform video encoder system 106.

[0075] Some implementations are based on the understanding that the principle of autoencoders can be extended to recursive encoding in the latent space. Specifically, encoding 306 in the latent space can be encoded again by the same encoder 304 and later recursively decoded by the decoder 308 of the autoencoder 302. For example, if the current segment 102a is encoded twice by the encoder of the autoencoder architecture, the data encoded twice will have the same dimension in the latent space, regardless of the number of encodings, and can be recursively decoded by executing the decoder of the autoencoder architecture twice.

[0076] In one implementation, the combination of the current segment 102a with the previous encoding 310 of the encoder 304 is performed based on a predefined criterion associated with at least the size of the input data. For example, the predefined criterion specifies a size limit for the input data in the form of a video frame sequence 102, or it specifies the number of time steps that have elapsed since the last combination operation was performed. Thus, the combined input of the encoder 304 is then used to produce an output 106a, which includes multi-depth encoding 306a of the input sequence of video frames 102.

[0077] Some implementations are based on another understanding: the recursive encoding of the autoencoder 302 can be used by downstream neural networks even without subsequent recursive decoding. It should be recognized that one of the advantages of the autoencoder paradigm is that it only performs encoding, making decoding possible. As a result, the recursive encoding retains sufficient information for decoding and can therefore be used by downstream neural networks with and / or without decoding.

[0078] Some implementations are based on the understanding that the same fundamental principles used for recursive autoencoders are effective for autoencoders 302 utilizing unwinding recursion, where the input to encoder 304 includes raw (unencoded) data and data previously encoded by encoder 304. Unwinding recursion is performed by combining the input stream with previously encoded data in the latent space and encoding this combination into the latent space, and repeating this process until a termination condition is met.

[0079] As a result, the output of encoder 304 includes codes 306 of varying depths, and can encode increasingly new incoming data into multi-depth codes of the same fixed size. Furthermore, the multi-depth codes can be submitted to the downstream neural network without recursive decoding, allowing the downstream neural network to perform tasks involving processing lengthy data of unknown lengths by processing the multi-depth codes at a fixed-length output 106a.

[0080] In this implementation, segments of different sizes are encoded during different iterations of the non-uniform unfolding recursion of the autoencoder 302. In this implementation, it is ensured that such non-uniform unfolding recursion does not violate the performance guarantees of the autoencoder architecture shown in block diagram 300. Furthermore, allowing the size of the unencoded data to be varied in different encoding iterations provides each encoding with additional meaning specific to and / or beneficial to the downstream task to be performed.

[0081] For example, in navigation-related downstream applications, the uncoded data in each encoding iteration can represent a space with specific semantic meaning. Examples of such spaces include rooms, streets, towns, etc. For instance, in one implementation where the semantic space is rooms and the downstream application is a robot navigating within a building with multiple rooms, each encoding iteration includes adding uncoded features of the rooms (e.g., represented as a local scene map) to previously encoded information indicating features of multiple rooms in the building. Doing so in this way allows the navigation application to rely on additional semantic meaning at each decoding iteration. However, different rooms can include different numbers of objects and local scene maps of different sizes representing different objects in different rooms. Therefore, some implementations use non-uniform unfolding recursion to capture variations in the data representing different rooms.

[0082] In another example of speech recognition, uncoded data of varying sizes can come from spoken utterances of varying lengths, such as sentences. In another example of video processing, uncoded data of varying sizes can come from scenes of varying durations. In different implementations, the uncoded input data is segmented into distinct, non-uniform semantic segments based on rules that favor the downstream neural network.

[0083] In some implementations, events are detected based on input data, and then the input data is divided into segments of different sizes based on the detected events. These events can be used to detect changes in the evolution of the scene in which the non-uniform video encoder system 106 operates.

[0084] In some implementations, scene evolution is recognized as a change in the shading pattern within the scene. For example, a scene may change from a brightly lit environment to a dimly lit one, instructing an agent (such as a robot) to move from one room in a building to another.

[0085] In some implementations, scene evolution is recognized as changes in the captions describing the scene. For example, language captions can be updated to reflect different speaker preferences.

[0086] In some implementations, scene evolution is recognized as a change in scene classification results. For example, in search and rescue operation type settings, when a trapped person is detected, the scene classification can change from search unsuccessful to search successful.

[0087] In some implementations, scene evolution is recognized as the detection of anomalies within the scene. For example, in an industrial automation environment, improper placement of an object can be identified as an anomaly and further trigger fault detection operations.

[0088] In some implementations, the evolution of a scene is recognized as the occurrence of an acoustic event within the scene. An acoustic event could be, for example, the start of music playback within the scene.

[0089] In some implementations, scene evolution is recognized as an event detected by a camera that captures the scene evolution using a sequence of video frames 102. For example, for a "find the teapot" task, the agent (robot) can use the camera to perceive objects in its field of view to determine if the agent can already see the teapot, and may then construct an abstraction (such as the teapot) that would be located near other objects (such as an oven, stove, plate, dish, etc.) rather than near a toilet or bathtub. Therefore, if the agent can see an object such as a microwave or stove in its current field of view, it may explore the environment more closely, while if it is near an object such as a television, sofa, etc., it may only make a cursory scan. As the agent explores the neighborhood, the scene continues to evolve based on the camera's perception.

[0090] In one implementation, the scene may be captured by one or more sensors of the agent and then modeled in the form of a scene graph representation, and in this case, the autoencoder 302 is a graph encoder that operates on the graph data as input.

[0091] Figure 4 A block diagram of a scene diagram generation pipeline 400 according to an embodiment of the present disclosure is shown.

[0092] Scene graph representation provides a way to sparsely model a scene without sacrificing its semantic content. In a scene graph representation, the nodes of the graph correspond to objects in the scene generated using an object detector (trained to detect objects of interest in the scene) and a relation detector that can capture 3D spatial semantic relationships between each node (e.g., a table is "behind" a chair, etc.). Scene graph representation provides deentanglement of pixel details and abstraction of the scene with higher semantic granularity, thereby improving generalization. A sequence of video frames 102 of a scene processed by a non-uniform video encoder system 106 is transformed into a scene graph representation. For this purpose, the scene graph representation forms a sequence of input data 102a received and processed by the non-uniform video encoder system 106. The video frame sequence 102 is input to a scene graph generator 402, which provides a spatiotemporal scene graph 406 (which may also be interchangeably referred to as scene graph 406 below) with nodes representing one or more objects in the scene as output. The current segment of the current iteration includes a portion of the spatiotemporal scene graph 406. Spatiotemporal scene graph 406 includes nodes representing one or more static objects (such as static node 406A) and one or more dynamic objects (such as dynamic node 406B) in the scene. The appearance and position of each of the static objects in the scene are represented by attributes of a single node in spatiotemporal scene graph 406, and each of the dynamic objects in the scene is represented by attributes of multiple nodes in spatiotemporal scene graph 406, which describe the appearance, position, and motion of each of the dynamic objects at different time instances.

[0093] In some embodiments, the processor 314 of the non-uniform video encoder system 106 is configured to receive a sequence of video frames 102 corresponding to a video of a scene. In one embodiment, the received video frame sequence 102 is preprocessed by a scene graph generator preprocessor 402 to output a preprocessed video frame sequence 402a. The preprocessed video frame sequence 402a includes objects detected in the video frame sequence 102 and depth information of the objects in the video frame sequence 102. In some embodiments, the video frame sequence 102 can be preprocessed using an object detection model for object detection in each video frame sequence of the video frame sequence 102 and a neural network model for depth information estimation.

[0094] The preprocessed video frame sequence 402a can then be input into the spatiotemporal transformer 404. The spatiotemporal transformer 404 transforms each preprocessed video frame sequence in the preprocessed video frame sequence 402a into a spatiotemporal scene graph 406(G) of the video frame sequence 102, in order to capture the spatiotemporal information of the video frame sequence 102.

[0095] The spatiotemporal scene graph 406 comprises graph nodes representing one or more static objects in the scene, such as beds, chairs, tables, etc. The spatiotemporal scene graph 406 also includes one or more dynamic objects in the scene, such as people. The spatiotemporal scene graph 406 includes nodes representing one or more static objects, such as static node 406A, and nodes representing one or more dynamic objects in the scene, such as dynamic node 406B. The appearance and position of each of the static objects in the scene are represented by attributes of a single node in the spatiotemporal scene graph 406, and each of the dynamic objects in the scene is represented by attributes of multiple nodes in the spatiotemporal scene graph 406. The motion of each of the dynamic objects is also represented by motion features 406C. In some example implementations, motion features 406C are extracted from the dynamic graph nodes of the spatiotemporal scene graph 406 using an action recognition model (e.g., an Inflated 3D Network (I3D) action recognition model).

[0096] In the spatiotemporal scene graph 406, each graph node (static or dynamic) has attributes representing the corresponding object. For example, a static graph node has attributes representing the appearance and position of the corresponding static object. Similarly, a dynamic graph node has attributes representing the appearance, position, and motion of the corresponding dynamic object at different time instances. As a result, the spatiotemporal scene graph 406 forms sequential input data 102a. The sequential input data 102a in the form of scene graph 406 is sent to a non-uniform video encoder system 106. The non-uniform video encoder system 106 is configured to encode each segment of the scene graph at the current time instance using encoder 304 to produce an encoding of the scene graph at the current time instance. Furthermore, the scene graph 406 may include previous encodings of previous scene graphs generated at previous iterations of the operation of the non-uniform video encoder system 106 that forms supernodes. Supernodes are connected to at least one node in a portion of the scene graph 406 to produce a combination encoded by encoder 304 of the non-uniform video encoder system 106 in the current iteration. The autoencoder 302 of the non-uniform video encoder system 106 thus becomes a graph encoder. The following is in conjunction with Figure 5 Further explanation of how this graph autoencoder works.

[0097] Figure 5 A block diagram illustrating the operation of a non-uniform video encoder 106 based on scene map input, according to an embodiment of the present disclosure, is shown. (In conjunction with...) Figure 3 To explain the components Figure 5A non-uniform video encoder 106 can be coupled to a robot configured to capture a sequence of video frames from scene 500. The robot 502 is configured to perform tasks, such as navigation, which can be implemented in a specific manner. To this end, the robot 502 includes various sensors to assist in performing these tasks. These sensors include, for example, RGB cameras, depth cameras, audio sensors, motion sensors, LiDAR sensors, temperature sensors, etc.

[0098] In some embodiments, robot 502 may include an input interface configured to receive input data to induce movement of robot 502. In examples, the input interface may receive input data from various sensors, including imaging devices (such as cameras, video cameras, etc.), audio sensors, speech sensors, etc. The input data can be used to transition the robot 502's posture from a starting posture to a target posture to perform a task, such as a navigation task. The input interface may also be configured to accept end posture modifications. End posture modifications include at least one or a combination of a new starting posture and a new target posture for robot 502. In some embodiments, the input interface is configured to receive input data indicative of visual and audio signals experienced by robot 502 during task performance. For example, the input data corresponds to multimodal information, such as audio, video, text, natural language, user input, or authentication. Such input data may include sensor-based video information received or sensed by one or more visual sensors, sensor-based audio information received or sensed by one or more audio sensors, or natural language instructions received or sensed by one or more speech sensors. The input data can be raw measurements received from one or more sensors, or any derivative of measurements coupled to or installed within robot 502, representing audio and / or video information and signals. The input data corresponds to the video frame sequence 102 of scene 500.

[0099] In one implementation, robot 502 is a set of components, such as arms, legs, and end tools, linked by joints. In an example, the joints can be rotary joints, sliding joints, or other types of joints. The set of joints determines the degrees of freedom of the corresponding component. In an example, the arm can have five to six joints allowing five to six degrees of freedom. In an example, the end tool can be a parallel jaw gripper. For example, a parallel jaw gripper has two parallel fingers whose distance can be adjusted relative to each other. Many other end tools can be used alternatively, such as end tools with welding ends. The joints can be adjusted to achieve a desired configuration of the components. The desired configuration can involve a desired position in Euclidean space or a desired value in joint space. The joints can also be commanded in the time domain to achieve a desired (angular) velocity and / or (angular) acceleration. The joints can have embedded sensors that can report the corresponding state of the joint. The reported state can be, for example, an angle value, a current value, a velocity value, a torque value, an acceleration value, or any combination thereof. The set of reported joint states is called a state.

[0100] Robot 502 may have multiple interfaces for connecting robot 502 to other systems and devices, such as connections to a controller for controlling robot 502. For example, robot 502 may be connected via a bus to one or more sensors to receive new starting and target postures via input interfaces. Additionally or alternatively, in some implementations, robot 502 includes a human-machine interface (HMI) that connects a processor to a keyboard and pointing devices, wherein the pointing devices may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, etc. In some embodiments, robot 502 may include one or more motors configured to move joints to change the movement of arms and / or legs according to commands generated according to a control strategy. Additionally, robot 502 includes a controller configured to execute control commands for controlling robot 502 to perform tasks. For example, as part of a task description, the controller is configured to operate the motors to change the placement of arms and / or legs according to a control strategy for commanding robot 502 to navigate and reach an object or location of interest.

[0101] It can be noted that, without categorizing "physical," "real," or "real-world," the reference to a robot can refer to a physical agent or physical robot, or a robot simulator designed to faithfully simulate the behavior of a physical agent or physical robot. A robot simulator is a program composed of a set of algorithms based on mathematical formulas to simulate the kinematics and dynamics of a real-world robot. In a preferred embodiment, the robot simulator also simulates a controller. The robot simulator can generate data for 2D or 3D visualization of robot 502.

[0102] Robot 502 may also include a processor configured to execute stored instructions, and a memory storing instructions executable by the processor. The processor may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations.

[0103] The memory may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The processor can be connected via a bus to one or more input interfaces and other devices. In this embodiment, the memory is specifically implemented within the controller and may additionally store the non-uniform video encoder system 106, including the self-encoder 302. The memory may additionally store program modules or code for implementing the navigation system. The code may be used to implement the functionality of a neural network configured to generate commands for controlling the robot to perform navigation based on the output received from the non-uniform video encoder system 106. The output of the non-uniform video encoder system 106 is in the form of a multi-depth code 106a corresponding to the sequence input data of scene 500 captured by the robot 502.

[0104] Robot 502 may also include a storage device suitable for storing various modules of stored executable instructions for the processor. The storage device may also store a self-exploration program for generating training data instructing the robot 502 on the space of the environment 500 in which it may need to navigate. The storage device may be implemented using a hard disk drive, optical disk drive, thumb drive, drive array, or any combination thereof. The processor of robot 502 is configured to determine control laws for controlling multiple motors to move the arm and / or foot according to a control strategy, and to execute a self-exploration program that explores the environment by controlling the motors according to the learned control strategy.

[0105] Robot 502 can be configured to perform tasks, such as navigation tasks that navigate robot 502 from its initial state to a target state (such as a room in a building) by following a trajectory. The trajectory can be decomposed into various sub-trajectories, representing various interactions of robot 502.

[0106] For example, robot 502 could be tasked with searching for objects of interest in a natural 3D environment 500, or environment 500 could be a concrete implementation of a virtual representation of a natural 3D environment. Robot 502 could be configured to learn policies (or inductive biases) to develop efficient and effective policies for solving them. Furthermore, these policies could be generalized to new environments that might differ from the training data used to train the robot 502's controller. Thus, the environment 500 could be represented using the scene's RGB and depth maps (from the robot 502's view) as input to a learning model to learn navigation policies (via reinforcement learning or imitation learning), for example, to determine where to move next to search for objects. However, vision-guided object-target navigation presents a double challenge: (i) accurately detecting objects of interest within the robot's field of view; and (ii) reasoning about the robot 502's current position in space. However, such input (with raw pixels capturing the scene) presents a challenge in learning which scene regions need to be used to generate an effective navigation policy, potentially requiring a huge training ensemble and millions of training epochs.

[0107] Based on this implementation, the non-uniform video encoder system 106 connected to robot 502 extracts relevant semantic details from the scene (such as environment 500, hereinafter referred to interchangeably as scene 500) and then trains robot 502 to use only a subset of image regions, resulting in faster training. Figure 4 The scene graph generation pipeline 400 shown constructs a scene graph representation for sparse modeling of scene 502. For this purpose, sequential input data 102a is generated for each consecutive movement of robot 502. Robot 502 captures an RGB image 506 and a depth map 504 of scene 500 at the current segment of video frame sequence 102. The RGB image 506 is fed to a mask-based region-based convolutional neural network (RCNN) module 508 to generate a local scene map 510. The depth map 504 is used to generate a global point cloud 512 of scene 500.

[0108] To this end, robot 502 constructs a 3D local scene graph 310 using objects in scene 500 as graph nodes and spatial relationships between nodes as graph edges. Then, the local graph 510 is registered with the 3D global scene graph 514 by calculating the spatial proximity of the local graph nodes and edges to the global graph 514 constructed so far. Two nodes are considered identical if their semantic labels match, their 3D point cloud segmentation masks overlap (Mask RCNN), and their approximate spatial neighborhoods are similar. Nodes in the local graph 510 that meet the above criteria are merged with nodes in the global scene graph 514, while nodes that do not meet the criteria are inserted into the global graph 514, with edges connecting them to their approximate 3D neighborhoods. Therefore, this process allows for the avoidance of redundancy in graph construction.

[0109] For example, given a sequence of time-evolving local scene graphs, such as local graph 510, one for each specific implementation video frame, by... This indicates that the partial graph from the video frame at time step t is composed of... Given, having vertices ,side and ,in It is a neural feature associated with vertex v.

[0110] The non-uniform video encoder system 106 is configured to encode local map sequences Registration to global scene diagram of time evolution In (such as global graph 514), (ii) if the global graph meets the criteria for compression, the global graph is compressed into supernodes using (graph) autoencoder 302, which embeds the entire global graph into supernode Euclidean graph embedding (EGE) and associates each supernode with an attribute that identifies it as a special node, (iii) the supernode features are incorporated into the subsequent evolution of the local scene graph and the global scene graph, and the graph (with supernodes) is recursively encoded into supernodes, repeating the process, and (iv) supernodes are used to avoid future computations along previously visited spatial regions, thereby improving the computational efficiency and storage requirements of the non-uniform video encoder system 106.

[0111] Local scene graph construction (510)

[0112] Robot 502 is equipped with an RGB camera and a depth camera, and it always has access to the agent's position and pose information. Let I represent an RGB image frame, such as RGB image 506, and D be its corresponding depth map, such as depth map 504, and let... This pertains to the camera pose of the global frame. To construct the local scene map (i.e., local graph 510), the Mask R-CNN pre-trained model, Mask R-CNN module 508, takes I as input and produces tuples. , where b is the detected bounding box, X is its feature vector, l is the object label, and conf represents the detection confidence. Only for a certain threshold are considered. have Those confidence tests. These tuples form a local scene graph of a frame at a given time step. Node V.

[0113] To construct a graph Edge E is defined using the spatial proximity criterion. Specifically, for two nodes u and v, assume... and These represent the positions of nodes u and v, respectively. For example, It can be the 3D location of the centroid of the bounding box corresponding to node w (e.g., for the small chair node), or it can also abstractly represent the point cloud corresponding to the large wall node. Given two such locations... and ,if This creates an edge between two nodes u and v in the local scene graph. Where dist is the appropriate distance and a given threshold. For example, if the 3D centroid location of a node is used, then dist can simply be the Euclidean distance; however, if it is a point cloud, then the chamfer distance is used. Calculating the chamfer distance can be expensive when nodes have large point sets (e.g., wall nodes or floor nodes). Furthermore, using the centroid of L in such large nodes may be unreasonable. To provide a computationally cheaper non-uniform video encoder system 106, a point cloud of a pre-specified number of points can be calculated, and then only these points can be used to calculate the chamfer distance. Thus, using the nodes and edges defined above, a local graph 510 is generated.

[0114] Register the local graph to the global graph (514)

[0115] As the robot agent 502 moves in 3D space, the RGBD frames at time steps t-1 and t are respectively... and The likelihood of significant overlap is high, so object detection in these frames could be different views of the same set of objects. These frames can be registered (using the available camera pose p) and the two local scene maps can be merged to form a more compact 3D scene map. For this purpose, the global scene map can be initialized from the beginning of the navigation trajectory using the local scene map. Such as global graph 514. Assume... This is the global graph constructed so far, and It is a local plot at time step t. Then, if If the camera pose is at frame t or segment t, then the figure The position of each node V can be transformed into a global frame, such as: ,in It is a 4×1 homogeneous vector (or a 4×r matrix if we use the centroid of u, or if we use a point cloud with r points), and p is a 3×4 projection matrix that includes the camera parameters.

[0116] Since the local graph exists in the coordinate system of the global graph, if the two graphs overlap, they can be merged. To achieve this, for the nodes in local graph 510... And if If a node is a node in global graph 514, then the standard is defined as: Standard = (i) & (ii) & (iii), where (4) (i) (5) (ii) (6) (iii) (7) in, Define the set of neighbors of node u. It is the class label of node u, and Let be the 3D location of node u. Approximate similarity is used using neighborhoods, where the approximate neighborhood is defined as the non-empty intersection of node pairs. That is, assuming It is the set of all node pairs u that have their neighbor nodes, and if If is the set of vertices in the global graph, then if If there exists a non-zero intersection of neighborhoods, then the criterion will be satisfied.

[0117] If two nodes and If they come from local graph 510 and global graph 514 respectively, then will with Merge features.

[0118] (8)

[0119] For a certain (That is, applying soft updates and merging of features from two nodes). Furthermore, nodes... The neighbor list is used Non-merging (i.e., non-redundant) neighbor expansion. Specifically, for nodes that do not meet the above criteria in (4). Then the node will be added to the graph as a new node. Its edge is connected to an edge that cannot be connected to. Other new nodes or edges are connected to the nodes in the merge process. Merged The nodes in the text. In the following text, Indicates by With local view The global graph at time step t obtained by merging.

[0120] The generation of local graph 510 and global graph 514 can be used as Figure 4 The scene graph generation pipeline 400 shown is part of the process. The local graph 510 and global graph 514 at time step t form the sequence input data 102a of the current segment in the current iteration of the operation of the non-uniform video encoder system 106. The sequence input data is then processed by the autoencoder 302.

[0121] Supernode generation using autoencoder 302

[0122] Some implementations are based on the understanding that processing a global scene graph is computationally expensive because the size of such a graph grows rapidly as more objects are detected in the scene.

[0123] To this end, the non-uniform video encoder system 106 is configured to utilize the autoencoder 302 to provide efficient processing of the global graph 514 by encoding the global graph 514 corresponding to each segment in the non-uniform segment sequence of the sequence input data 102a by an encoder 304 utilizing a non-uniform unfolding recursive autoencoder 302 to produce a multi-depth encoding of the sequence input data 102a. The encoder 304 generates an encoding 306 of the sequence input data 102a in the form of the global scene graph 514. The encoding 306 includes an embedding of the global graph 514 in the form of a Euclidean vector. This embedding is associated with a supernode as a feature vector. A supernode is a special node that is an abstraction of the embedded scene graph. Mathematically, a supernode is represented as... v s And by operator S: Generate, among which .

[0124] In one implementation, S is a graph neural network trained using end-to-end backpropagation of downstream tasks. In another implementation, S is a pre-trained model of supernode graph embedding (SuGE), which performs the SuGE algorithm and is trained separately for a self-supervised task, rather than using end-to-end training of downstream tasks. For this purpose, encoder 304 includes methods for embedding data in a global graph 514 (…). supernodes The SuGE algorithm generates encoding 306 in the form of [the following]: Therefore, the autoencoder 302 becomes [the process of] generating scene graphs [from the following]. As input and to generate supernodes The output is a graph autoencoder. Supernodes. The graphic embedding generated by the self-encoder 302 (by... (Parameterization).

[0125] Therefore, the encoder 304 of the self-encoder 302 is characterized by its encoding function E: Decoding function D of decoder 308: Some implementations are based on the understanding that the graph has the characteristics of each node and a structure with potentially irregular neighborhoods—captured by edges via an adjacency matrix.

[0126] To this end, the SuGE model provides a two-stage end-to-end encoding / decoding method for the autoencoder 302: (i) In the first stage, a graph convolutional network is used with an adjacency matrix, and the encoder f is used to encode the node features of each node v. Encoding as latent features Therefore, for the entire set of nodes, the set of potential values ​​is given as: And (ii) the set encoder g is used to generate latent feature vectors. .

[0127] Assumption and It is a decoder, and makes and D= ,and If is the set of all learnable parameters in the SuGE model, then the parameters of the learned model are defined by the following formula:

[0128] Where, assume f and It is a module that operates on the graph node features, while in (12), f is a graph convolutional network (GCN) that takes node features and adjacency matrix as input to generate latent k-dimensional features for each node. , where k < d. Since we use a graph convolutional network for f, it is expected that each latent node feature also encodes information from its neighbors. Next, these latent features are encoded into a single feature embedding using the encoder g, which is a set encoder that treats the latent feature matrix Z (each as a column of Z) as a set (ignoring edge connections) and encodes it into the vector in (13). The vector is the supernode graph embedding (SuGE) of the graph.

[0129] To ensure that this autoencoder 302 model correctly encodes all useful information in the graph (i.e., ensure that y indeed encodes all information about the graph ), the loss in (10) is proposed. The key challenge in the encoding in (13) is that both the graph structure and the node embeddings are mixed in the latent space of y and need to be recovered. However, when decoding y into a set as in (14), it may not be clear how this can be done because the order in which the encoding was done may be lost when decoding a vector into a set. To this end, y is first decoded into a matrix Z with an arbitrary order of its columns. When a vector is decoded into a set, the decoder must also need to know how many elements will be in the decoded set. For example, if is an LSTM, it needs to know how many times the recurrence of <000040!>should be performed. To this end, is provided with an estimate of the number of nodes , i.e., n decoded using the function <000040!>in (17). Next, an alignment between the terms decoded with and the terms encoded with Z is obtained, which is described in (11) by solving an optimal transport problem using the Wasserstein distance , where corresponds to the permutation matrix γ in the space of all permutations that capture the alignment. Using these inputs, the loss in (10) is implemented. Specifically, in the first term in (10), the decoded feature matrix [[ID=!4]] is minimized with respect to the original feature X. In the second term, the decoded adjacency matrix produced using (15) is used with a binary compatibility loss (e.g., binary cross - entropy) using the alignment with respect to the original adjacency alignment (recall that we have assumed that the latent feature z also encodes the neighborhood details of the nodes, and we assume that the neighborhood can be revealed by taking the correlation between the latent embeddings between the nodes in (15) using the sigmoid).

[0130] In the third term of (10), the number of nodes in the graph is correctly embedded in the SuGE embedding y. When using a large set of graphs sharing the same encoder and decoder as described above, the autoencoder 302 model learns to produce vector SuGE embeddings for any global graph that can be faithfully decoded to its original graph.

[0131] In this implementation, an autoencoder 302, including a SuGE model, is trained using data associated with navigation trajectories from several specific implementations. This autoencoder can work similarly with any other sequence of local scene graphs (e.g., video scene graphs), each with a temporal evolution of the scene graph. The SuGE model is recursively invoked so that the graph is sometimes encoded using the SuGE encoder. The final embedding is then used for decoding using the SuGE decoder to produce supernode features and the encoded global graph.

[0132] Figure 6 The diagram illustrates the use of the SuGE model implemented by the aforementioned autoencoder 302 model to perform non-uniform unfolding recursion to global graph 514 ( The sequence input data in the form of ) is encoded into a multi-depth encoding corresponding to the sequence input data. An example of a recursive graph generation method in a supernode.

[0133] Figure 6 It shows the use of Figure 5 The method 600 describes the generation of recurrence graphs using the SuGE model. Combined with... Figure 3 and Figure 5 To explain the elements Figure 6 Method 600 is agnostic to downstream tasks. Method 600 includes one or more operations. These operations are associated with data from a hierarchical decomposed scene graph. The operations of Method 600 are described in the following description.

[0134] The operation of method 600 includes, in step 1, starting method 600 at the current time instance t. In step 2, the local scene graph of time instance t is... Global scene graph assigned to instances t at the same time Partial scene diagram It can be Figure 5 The partial scene diagram shown is 510. Global scene diagram. It can be Figure 5 The global scene diagram shown is 514.

[0135] In step 3, the iterative processing of steps 4-10 is initiated. The time instance is incremented in step 4, and in step 5, the local scene graph of the previous time instance is accessed. (Such as from the memory of the non-uniform video encoder system 106). Reading a local scene map. This includes running an object detector on RGB frame 506 of scene 500 and creating graph edges using a depth map or depth map 504. At this point, in step 6, if the previously created global scene graph... If a graph (such as global graph 514) satisfies some reduction properties (e.g., the number of nodes is greater than a threshold), then the forward pass of the SuGE model for generating the supernode graph embedding is invoked by the autoencoder 302.

[0136] Furthermore, using the SuGE model, supernodes are generated in step 7. and its characteristics Steps 8 and 9 involve defining a new global graph. The vertices and edges. Next, in step 10, the new global graph... , where nodes By supernode With the vertex set of the local graph from step t The union of the given set is given, and the edges are formed by... The edges and All nodes and supernodes The union of the new set of edges between them is obtained. Once the supernode is created... Therefore, it is necessary to merge it with the local graph.

[0137] Figure 7 A schematic diagram 700 illustrating the merging of a supernode with a local graph according to an embodiment of the present disclosure is shown. While method 600 uses a recursive scene graph (RSG) algorithm corresponding to method 600 to set up the basis for constructing the supernode, it is necessary to merge the local graph from previously visited locations with the RSG. In one example, at the current time step... When robot 502 reaches the supernode created at the previous time step t-1 When the number of nodes, represented as n, in the graph grows to exceed a threshold, the criteria used to create the supernode are checked. At that time, assuming the agent calls the supernode, for example, robot 502 maintains a list of the spatial locations of all supernodes, and if there is such a node within a predefined radius, the supernode closest to its current location is selected. Furthermore, in one implementation, a SuGE decoder is used to reproduce the... The associated scene diagram. This scene diagram is represented as... The current local view can be represented as: Furthermore, feature X is captured within the nodes of the corresponding graph. This is to merge the two graphs. and Consider some predefined criteria. In one example, these include four cases: (i) there is no overlap between the decoded graph and the local graph, i.e., (ii) There is some overlap between the decoded image and the local image. (iii) The local graph is a subgraph of the decoded graph. (iv) The decoded graph is a subgraph of the local graph. .

[0138] In some implementations, the merging rule for each of these four conditions is described as follows: When in time Local scene diagram and When there is no overlap, that is, Compared with the current scene diagram Merge in global coordinate space, as previously combined. Figure 5 As described above, to generate a new global graph .

[0139] ,when Some nodes in the decoding nodes and local graph When nodes overlap, they are merged. Nodes in Overlapping nodes in the data, and using a new feature set. Update supernode No additional steps are needed in this case because the current local graph... Already The explanation is in the middle.

[0140] In this case, It is considered a new local scene graph, and method 600 is executed.

[0141] In some implementations, supernodes are stored as neurally embedded (neural) linked lists, while also having hash access to the nearest node, where the hash function is the spatial location of the agent.

[0142] The process of merging and executing method 600 is in... Figure 7As shown in the diagram. For time step t-1, if the number of global graph nodes is greater than a threshold or K time steps have elapsed, a graph autoencoder module, such as an unfolded recursive autoencoder 302, is invoked. This module receives the current global graph 514a and produces an encoding of the form of feature embedding 518a for supernode 520a, which summarizes the global graph 514a at the current time step (time t-1). Then, using any of the merging conditions (i)-(iv), the supernode 520a is integrated or merged into the local scene graph 516a at time step t via fully connected edges to nodes in the local graph. The global graph construction process then continues as described above, and at time t+K-1, for the global graph 514b, another feature 518b is constructed by the same graph encoder of autoencoder 302, resulting in a supernode 520b integrated with the local graph 516b at time t+K. This process continues to recursively construct supernodes.

[0143] Supernodes are used to store scene data in a computationally efficient manner. The supernodes are generated by an autoencoder 302 using multi-depth encoding as sequence input data 102a. This multi-depth encoding is then used when performing downstream tasks.

[0144] Figure 8 A schematic diagram 800 illustrates a use case of a non-uniform video encoder system 106 for performing downstream task 804. Sequential input data 102a is provided to the non-uniform video encoder system 106, which uses an autoencoder 302 to generate multi-depth codes 106a. The multi-depth codes 106a are sent to a downstream neural network 802 to perform task 804. For this purpose, the downstream neural network 802 outputs commands to control a robot 502 to perform task 804. This task could be an automatic speech recognition (ASR) task, as will be discussed later. Figure 9 As shown in the image.

[0145] Figure 9A schematic diagram 900 illustrates a use case including an Automatic Speech Recognition (ASR) task 904 performed by a downstream neural network 902a using a non-uniform video encoder system 106. The non-uniform video encoder system 106 receives speech data 902 as input data 102. For example, the speech data 902 includes speech utterances with multiple sentences, and the event can be the end of a sentence. The end of a sentence can be detected by using a stop action. Upon detection of the event, the speech data 902 can be segmented into input data segments and passed to the non-uniform video encoder system 106. The non-uniform video encoder system 106 can then generate multi-depth codes for the speech data by summarizing the multiple sentences, and then use these multi-depth codes to perform the ASR task 904 using the downstream neural network 902a. In the example, the downstream neural network 902a can be a caption decoder to provide captions for different speech utterances in different scenes of the input data 902. In the caption decoding task, the end of a scene of a segment of captured audio and video frames is detected based on one or a combination of the end of the current caption and the start of the next caption.

[0146] In another example, ASR task 904 could be a speech-to-text task. In this way, the non-uniform video encoder system 106 can be used for efficient data processing in various applications, such as efficient scene navigation, efficient speech processing, and efficient GNSS measurement data processing. In implementations, the applications are as follows: Figure 8 The navigation task shown.

[0147] Figure 10 A schematic diagram 1000 illustrates a use case of a non-uniform video encoder system 106 for performing a downstream navigation task 1004. Sequential input data 102a is provided to the non-uniform video encoder system 106, which uses an autoencoder 302 to generate multi-depth codes 106a. The multi-depth codes 106a are sent to a downstream neural network 1002 for performing task navigation 1004. For this purpose, the downstream neural network 1004 outputs commands to control a robot 502 to navigate in an environment. The environment may be a building, and the robot 502 may navigate different rooms, floors, areas, etc., within the building.

[0148] The non-uniform video encoder system 106 can be implemented as a computing device in various ways, such as a controller, a processor that executes stored instructions, a dedicated computing device, etc.

[0149] In one implementation, the non-uniform video encoder system 106 includes a transformer neural network model for extracting graphical features from nodes of a scene graph based on sequence input data 102a.

[0150] In one implementation, when only a single node exists in the global graph and supernodes are computed at each time step, the non-uniform video encoder system 106 includes a recurrent neural network (RNN). The latent output of the RNN forms supernode embeddings that can be decoded as inputs, as in a sequence-to-sequence model.

[0151] In one implementation, in addition to spatial proximity, other relationships between nodes are used to generate local and global scene graphs. Such relationships may include semantic relationships such as "in front," "next to," etc., or actions between object nodes, such as "a boy is playing with a baseball bat," where the relationship between the boy and the bat is "playing." Any such equivalent relationships may be used in the non-uniform video encoder system 106 without departing from the scope of this disclosure.

[0152] The non-uniform video encoder system 106 can be used to perform a variety of downstream tasks, such as graphics compression for various tasks, including (i) navigation for efficient resolution of specific implementations of tasks related to scene navigation, (ii) audio tasks, such as locating objects that produce specific sounds, and (iii) data abstraction tasks.

[0153] In one implementation, the multi-depth encoding generated at the output 106a of the non-uniform video encoder system 106 is used to generate a recursive scene graph (RSG) as a representation of the scene in the robot's memory, which can be used to determine the robot's next move. Given the structure of the RSG, the size of the representation is fixed, and simultaneously, the robot can recursively decode and navigate the scene to, for example, efficiently search for targets.

[0154] In one implementation, a non-uniform video encoder system 106 is used to encode a video scene graph.

[0155] Therefore, the non-uniform video encoder system 106 can be implemented using computing devices, such as... Figure 11 As shown.

[0156] Figure 11This is a schematic diagram illustrating a computing device 1100 for implementing the non-uniform video encoder system 106 of this disclosure. The computing device 1100 includes a power supply 1101, a processor 1103, a memory 1105, and a storage device 1107, all connected to a bus 1109. Additionally, a high-speed interface 1111, a low-speed interface 1113, a high-speed expansion port 1115, and a low-speed connection port 1117 can be connected to the bus 1109. Furthermore, a low-speed expansion port 1119 is connected to the bus 1109. Additionally, an input interface 1121 can be connected via the bus 1109 to an external receiver 1123 and an output interface 1125. A receiver 1127 can be connected via the bus 1109 to external transmitters 1129 and 1131. External memory 1133, external sensors 1135, a machine 1137, and an environment 1139 can also be connected to the bus 1109. Furthermore, one or more external input / output devices 1141 can be connected to the bus 1109. The network interface controller (NIC) 1143 may be adapted to be connected to the network 1145 via the bus 1109, wherein data or other data may be presented on a third-party display device, a third-party imaging device, and / or a third-party printing device outside the computing device 1100.

[0157] Memory 1105 may store instructions executable by computing device 1100 and any data usable by the methods and systems of this disclosure. Memory 1105 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Memory 1105 may be one or more volatile memory cells and / or one or more non-volatile memory cells. Memory 1105 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0158] Storage device 1107 may be adapted to store supplemental data and / or software modules used by computer device 1100. Storage device 1107 may include hard disk drives, optical disk drives, thumb drives, drive arrays, or any combination thereof. Furthermore, storage device 1107 may contain computer-readable media such as floppy disk devices, hard disk devices, optical disk devices, magnetic tape devices, flash memory or other similar solid-state storage devices, or device arrays, including devices in storage area networks or other configurations. Instructions may be stored in an information carrier. When executed by one or more processing devices (e.g., processor 1103), the instructions perform one or more methods, such as those described above.

[0159] The computing device 1100 may be optionally linked via bus 1109 to a display interface or user interface (HMI) 1147 suitable for connecting the computing device 1100 to a display device 1149 and a keyboard 1151, wherein the display device 1149 may include a computer monitor, camera, television, projector, or mobile device, etc. In some implementations, the computing device 1100 may include a printer interface for connecting to a printing device, wherein the printing device may include a liquid inkjet printer, a solid ink printer, a large-scale commercial printer, a thermal printer, a UV printer, or a dye-sublimation printer, etc.

[0160] High-speed interface 1111 manages bandwidth-intensive operations of computing device 1100, while low-speed interface 1113 manages lower bandwidth-intensive operations. This functional allocation is merely an example. In some implementations, high-speed interface 1111 may be connected to memory 1105, user interface (HMI) 1149, keyboard 1151, and display 1149 (e.g., via a graphics processor or accelerator), and to high-speed expansion port 1115, which can accept various expansion cards via bus 1109. In some implementations, low-speed interface 1113 is connected to storage devices.

[0161] Device 1107 and low-speed expansion port 1117 are connected via bus 1109. Low-speed expansion port 1117, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), can be connected to one or more input / output devices 1141. Computing device 1100 can be connected to server 1153 and rack server 1155. Computing device 1100 can be implemented in several different forms. For example, computing device 1100 can be implemented as part of rack server 1155.

[0162] The embodiments described above can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can execute on any suitable processor or set of processors, whether provided on a single computer or distributed among multiple computers. Such a processor can be implemented as an integrated circuit using one or more processors within an integrated circuit assembly. However, the processor can be implemented using circuitry of any suitable format.

[0163] Furthermore, embodiments of this disclosure can be specifically implemented as a method, examples of which have been provided. Actions performed as part of the method can be ordered in any suitable manner. Therefore, embodiments in which actions are performed in a different order than those shown can be constructed, which may include performing some actions simultaneously, even if they are shown as sequential actions in the illustrative embodiments.

[0164] The use of ordinal terms such as “first” or “second” to modify a claim element in a claim does not imply any priority, precedence, or order of one claim element relative to another, or the chronological order of the actions of the method, but is merely a label to distinguish one claim element with a specific name from another element with the same name (but using ordinal terms), thus differentiating the claim elements.

[0165] Although this disclosure has been described by way of examples of preferred embodiments, it should be understood that various other adjustments and modifications may be made within the spirit and scope of this disclosure.

[0166] Therefore, the purpose of the appended claims is to cover all such variations and modifications that fall within the true spirit and scope of this disclosure.

Claims

1. A non-uniform video encoder system, the non-uniform video encoder system comprising: At least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the non-uniform video encoder system to: The video frame sequence of the received scene; Transform the video frame sequence into a sequence of input data that indicates the evolution of the scene in time, space, or both spatiotemporal; The sequence input data is divided into a non-uniform segment sequence that indicates changes in the evolution of the scene; Each segment in the non-uniform segment sequence is encoded by an encoder using a non-uniform unfolding recursive autoencoder architecture to generate a multi-depth code for the sequence input data, wherein, in order to encode the current segment at the current iteration to generate the current code, the non-uniform unfolding recursion combines the current segment with the previous code generated at the previous iteration, and the encoder encodes the combination. as well as The multi-depth encoding of the output sequence input data.

2. The non-uniform video encoder system according to claim 1, wherein, Changes in the evolution of the scene are identified by one or a combination of the following: events detected in the scene, changes in the coloring patterns in the scene, changes in the captions describing the scene, changes in the classification results of the scene, anomalies detected in the scene, acoustic events detected in the scene, and events associated with the camera that captured the evolution of the scene using the video frame sequence.

3. The non-uniform video encoder system according to claim 1, wherein, The multi-depth encoding of the sequence input data forms a spatiotemporal scene graph having nodes representing one or more objects in the scene, wherein the current segment of the current iteration includes a portion of the scene graph, wherein the previous encoding generated at the previous iteration forms a supernode, and wherein the processor is configured to connect the supernode to at least one node in the portion of the scene graph to produce the combination encoded by the encoder at the current iteration.

4. The non-uniform video encoder system according to claim 3, wherein, The spatiotemporal scene graph includes nodes representing one or more static objects and one or more dynamic objects in the scene, wherein the appearance and position of each static object in the scene are represented by attributes of a single node of the spatiotemporal scene graph, and wherein each dynamic object in the scene is represented by attributes of multiple nodes of the spatiotemporal scene graph describing the appearance, position, and motion of each dynamic object at different time instances.

5. The non-uniform video encoder system according to claim 1, wherein, The processor is configured to: The multi-depth encoding of the sequence input data is submitted to the downstream neural network to perform the task.

6. The non-uniform video encoder system according to claim 5, wherein, The scenario includes an audio scenario, which includes speech speech data having multiple sentences, and wherein the downstream neural network is configured to perform a speech processing task in response to submitting multi-depth encodings of the sequence input data of the audio scenario to the downstream neural network.

7. The non-uniform video encoder system according to claim 1, wherein, The processor is configured to: The multi-depth encoding of the sequence input data is submitted to the downstream neural network to perform the navigation task.

8. A robot, the robot comprising: The non-uniform video encoder system according to claim 1; as well as A navigation system comprising a neural network configured to generate navigation commands based on the multi-depth encoding of the sequence input data.

9. The non-uniform video encoder system according to claim 8, wherein, The scenario includes observing objects in rooms of a building by moving the robot within the building, and detecting the end of the scenario when the robot leaves the room.

10. The non-uniform video encoder system according to claim 9, wherein, The processor is configured to execute a scene decoder, which is configured to generate a navigation plan that includes computer-executable instructions to bring the robot to a target object in a scene previously encoded by the autoencoder.

11. The non-uniform video encoder system according to claim 1, wherein, The processor is configured to execute the Supernode Graph Embedding (SuGE) algorithm to perform the non-uniform unfolding recursion, thereby encoding the sequence input data into supernodes corresponding to the multi-depth encoding of the sequence input data.

12. The non-uniform video encoder system according to claim 11, wherein, The SuGE algorithm includes one or more operations, which are executed by the processor to enable the non-uniform video encoder system to: At the current iteration, obtain local graph data associated with the scene; At the previous iteration, global graph data associated with the scene is obtained; In response to the global graph satisfying the graph reduction criterion, generate supernodes for the global graph; The updated global graph data is generated by merging the local graph data and the generated supernodes. as well as The updated global graph data is used as the multi-depth encoding of the sequence input data and stored in the memory.

13. The non-uniform video encoder system according to claim 1, wherein, The autoencoder is a graph autoencoder.

14. A controller for controlling a robot to perform a task, the controller comprising: A memory configured to store instructions; as well as A processor configured to execute stored instructions to perform the steps of a method, including: The video frame sequence of the received scene; Transform the video frame sequence into a sequence of input data that indicates the evolution of the scene in time, space, or both spatiotemporal; The sequence input data is analyzed by dividing it into non-uniform fragment sequences to identify changes in the evolution of the scene; Each segment in a non-uniform segment sequence is encoded by an encoder utilizing a non-uniform unfolding recursive autoencoder architecture to generate a multi-depth code for the sequence input data. Specifically, to encode the current segment at the current iteration to generate the current code, the non-uniform unfolding recursion combines the current segment with a previous code generated at a previous iteration, and the encoder encodes the combination. The multi-depth encoding of the output sequence input data.

15. The controller according to claim 14, wherein, Changes in the evolution of the scene are identified by one or a combination of the following: events detected in the scene, changes in the coloring patterns in the scene, changes in the captions describing the scene, changes in the classification results of the scene, anomalies detected in the scene, acoustic events detected in the scene, and events associated with the camera that captured the evolution of the scene using the video frame sequence.

16. The controller according to claim 14, wherein, The sequence input data includes a spatiotemporal scene graph having nodes representing one or more objects in the scene, wherein the current segment of the current iteration includes a portion of the scene graph, wherein the previous encoding generated at the previous iteration forms a supernode, and wherein the processor is configured to connect the supernode to at least one node in the portion of the scene graph to produce the combination encoded by the encoder at the current iteration.

17. The controller according to claim 16, wherein, The spatiotemporal scene graph includes nodes representing one or more static objects and one or more dynamic objects in the scene, wherein the appearance and position of each static object in the scene are represented by attributes of a single node of the spatiotemporal scene graph, and wherein each dynamic object in the scene is represented by attributes of multiple nodes of the spatiotemporal scene graph describing the appearance, position, and motion of each dynamic object at different time instances.

18. The controller according to claim 14, wherein, The processor is configured to: The multi-depth encoding of the sequence input data is submitted to the downstream neural network to perform the task.

19. The controller according to claim 18, wherein, The downstream neural network is configured to generate navigation commands for controlling the robot to perform navigation tasks.

20. A non-transitory computer-readable storage medium having a program implemented thereon that can be executed by a processor to perform a method, the method comprising: The video frame sequence of the received scene; Transform the video frame sequence into a sequence of input data that indicates the evolution of the scene in time, space, or both spatiotemporal; The sequence input data is analyzed by dividing it into non-uniform fragment sequences to identify changes in the evolution of the scene; Each segment in the non-uniform segment sequence is encoded by an encoder using a non-uniform unfolding recursive autoencoder architecture to generate a multi-depth code for the sequence input data, wherein, in order to encode the current segment at the current iteration to generate the current code, the non-uniform unfolding recursion combines the current segment with the previous code generated at the previous iteration, and the encoder encodes the combination. as well as The multi-depth encoding of the output sequence input data.

Images