Systems and methods for accelerated video-based training of machine learning models

Abstract

The systems and methods may include a computing system that receives a bitstream of a video sequence and one or more indications of one or more image frames of the video sequence, parses the bitstream to determine timestamps, positions within the bitstream, and types of the image frames of the video sequence, and uses the one or more indications and the timestamps, positions within the bitstream, and types of the image frames of the video sequence to determine one or more segments of the bitstream for decoding to extract the one or more image frames. For an image frame of the one or more image frames, a corresponding segment represents a corresponding reference chain of image frames of the video sequence. The computing system may decode the one or more segments of the bitstream and use the one or more image frames to train an ML model.

Description

[Technical Field]

[0001] [Cross-reference to related patent applications] This application claims priority to U.S. Provisional Application No. 63 / 377,954, filed September 30, 2022, and U.S. Provisional Application No. 63 / 378,012, filed September 30, 2022, each of which is incorporated by reference herein in its entirety for all purposes.

[0002] FIELD OF THE DISCLOSURE This disclosure relates generally to video training of machine learning (ML) models. In particular, this disclosure relates to systems and methods for accelerated training of ML models with video data. [Background technology]

[0003] Due to rapid advances in computer technology, autonomous navigation techniques used for autonomous vehicles and robots (collectively, egos) have become widespread. These advances enable safer and more reliable autonomous navigation for egos. Egos often must navigate through complex and dynamic environments and terrain that may include vehicles, traffic, pedestrians, cyclists, and a variety of other static or dynamic obstacles. Understanding an ego's surroundings is necessary for making informed and appropriate decisions to avoid collisions. Summary of the Invention

[0004] The systems, devices, and methods described herein provide accelerated training of machine learning (ML) models. In particular, for ML models trained on video data, the systems, devices, and methods described herein enable fast decoding of the video data and efficient use of computational and memory resources. For ML or artificial intelligence (AI) models used to predict or detect ego surroundings, such as occupancy networks, training such models is very time-consuming. The systems, devices, and methods described herein significantly accelerate the training and / or validation of such models.

[0005] In one embodiment, a method includes receiving, by a processor, a bitstream of a video sequence and one or more indications showing one or more image frames of the video sequence for training a machine learning (ML) model; determining, by the processor, timestamps, positions in the bitstream, and types of the image frames of the video sequence by parsing the bitstream; determining, by the processor, using the one or more indications and the timestamps, positions in the bitstream, and types of the image frames of the video sequence, for one image frame of the one or more image frames, one or more segments of the bitstream for decoding to extract one or more image frames, such that a corresponding segment represents a corresponding reference chain of the image frames of the video sequence; decoding, by the processor, the one or more segments of the bitstream; and using, by the processor, the one or more image frames to train the machine learning model.

[0006] The method may further include allocating, by the processor, a memory area in the processor's memory; storing, by the processor, the bitstream in the allocated memory area; and storing, by the processor, one or more image frames in the allocated memory area after decoding one or more segments.

[0007] For each image frame of the one or more image frames, a corresponding reference chain of image frames may start with an intra-frame (I-frame) of the video sequence and end with an image frame of the one or more image frames.

[0008] The step of determining the timestamp, position in the bitstream, and type of image frames of the video sequence may include the step of generating one or more data structures that store (i) for each image frame of the video sequence, a corresponding timestamp and a corresponding offset indicating a corresponding position of the compressed data for the image frame in the bitstream, and (ii) for each image frame of a particular type in the video sequence, a corresponding indication of the particular type.

[0009] The processor may be a graphical processing unit (GPU). In some implementations, one or more segments of the bitstream may be decoded by a hardware decoder built into the processor.

[0010] The one or more representations may include one or more time values, and the step of determining one or more segments of the bitstream includes, for each time value of the one or more time values, determining a first timestamp among the timestamps of an image frame of the video sequence that is closest to the time value, such that the first timestamp corresponds to a first image frame in the video sequence; determining a second timestamp of an I-frame of the video sequence, the second timestamp being determined as the I-frame timestamp that is closest to the first timestamp and is less than or equal to the first timestamp; using the second timestamp to determine a start position of the I-frame in the bitstream among the image frame positions; and using the first timestamp to determine an end position of the first image frame in the bitstream among the image frame positions; and determining a segment of the bitstream to extend between the start position of the I-frame and the end position of the first image frame.

[0011] The bitstream can be a first bitstream of a first compressed video sequence captured by a first camera, and the method can further include receiving, by a processor, a second bitstream of a second video sequence captured by a second camera; determining, by the processor, timestamps, positions within the second bitstream, and types of image frames of the second video sequence by parsing the second bitstream; determining, by the processor, one or more segments of the second bitstream for decoding to extract one or more second image frames of the second video sequence using the one or more indications and the timestamps, positions within the second bitstream, and types of image frames of the second video sequence, such that with respect to one of the one or more second image frames, a corresponding segment of the second bitstream represents a corresponding reference chain of image frames of the second video sequence; decoding, by the processor, the one or more segments of the second bitstream; and using, by the processor, the one or more second image frames to train an ML model. The first camera and the second camera do not need to be synchronized with each other.

[0012] At least two of the one or more segments of the first bitstream and the one or more segments of the second bitstream may be decoded in parallel by at least two hardware decoders integrated into the processor.

[0013] In another embodiment, a computing device may include a memory and a processing circuit. The processing circuit may be configured to receive a bitstream of a video sequence and one or more indications of one or more image frames of the video sequence, parse the bitstream to determine timestamps, positions in the bitstream, and types of the image frames of the video sequence, use the one or more indications and the timestamps, positions in the bitstream, and types of the image frames of the video sequence to determine one or more segments of the bitstream for decoding to extract one or more image frames, such that a corresponding segment for one of the one or more image frames represents a corresponding reference chain of the image frames of the video sequence, decode the one or more segments of the bitstream, and use the one or more image frames to train the machine learning (ML) model.

[0014] The processing circuitry may be further configured to allocate a memory area in the memory, store the bitstream in the allocated memory area, and, after decoding the one or more segments, store one or more image frames in the allocated memory area.

[0015] For each image frame of the one or more image frames, a corresponding reference chain of image frames may start with an intra-frame (I-frame) of the video sequence and end with an image frame of the one or more image frames.

[0016] Upon determining the timestamps, positions within the bitstream, and types of image frames of the video sequence, the processing circuitry may be configured to generate one or more data structures that may store (i) for each image frame of the video sequence, a corresponding timestamp and a corresponding offset that indicate a corresponding position of the compressed data for the image frame within the bitstream, and (ii) for each image frame of a particular type in the video sequence, a corresponding indication of the particular type.

[0017] The computing device may be a graphical processing unit (GPU), and one or more segments of the bitstream may be decoded by a hardware decoder built into the GPU.

[0018] The one or more representations may include one or more time values, and in determining the one or more segments of the bitstream, the processing circuit may be configured to: for each time value of the one or more time values, determine a first timestamp among timestamps of image frames of the video sequence that is closest to the time value, the first timestamp corresponding to the first image frame in the video sequence; determine a second timestamp of an I-frame of the video sequence, the second timestamp being determined as an I-frame timestamp that is closest to the first timestamp and is less than or equal to the first timestamp; use the second timestamp to determine a starting position of the I-frame in the bitstream among the image frame positions; use the first timestamp to determine an ending position of the first image frame in the bitstream among the image frame positions; and determine a segment of the bitstream to extend between the starting position of the I-frame and the ending position of the first image frame.

[0019] The bitstream can be a first bitstream of a first compressed video sequence captured by a first camera, and the processing circuitry can be further configured to receive a second bitstream of a second video sequence captured by a second camera, parse the second bitstream to determine timestamps, positions within the second bitstream, and types of image frames of the second video sequence, use the one or more indications and the timestamps, positions within the second bitstream, and types of image frames of the second video sequence to determine one or more segments of the second bitstream for decoding to extract one or more second image frames of the second video sequence, where, for an image frame of the one or more second image frames, a corresponding segment of the second bitstream represents a corresponding reference chain of image frames of the second video sequence, decode the one or more segments of the second bitstream, and use the one or more second image frames to train the ML model. The first camera and the second camera do not need to be synchronized with each other.

[0020] At least two of the one or more segments of the first bitstream and the one or more segments of the second bitstream may be decoded in parallel by at least two hardware decoders incorporated in the computing device.

[0021] In yet another embodiment, a non-transitory computer-readable medium can store computer code instructions that, when executed by a processor, cause the processor to receive a bitstream of a video sequence and one or more indications of one or more image frames of the video sequence to train a machine learning (ML) model, determine timestamps, positions in the bitstream, and types of the image frames of the video sequence by parsing the bitstream, determine one or more segments of the bitstream for decoding to extract the one or more image frames using the one or more indications and the timestamps, positions in the bitstream, and types of the image frames of the video sequence, where, for one image frame of the one or more image frames, the corresponding segment represents a corresponding reference chain of image frames of the video sequence, decode the one or more segments of the bitstream, and use the one or more image frames to train the machine learning model. [Brief explanation of the drawings]

[0022] Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying drawings, which are schematic and not intended to be drawn to scale, and unless indicated as representing background art, the figures represent aspects of the present disclosure.

[0023] [Figure 1A] 1 illustrates components of an AI-enabled visual data analysis system, according to one embodiment.

[0024] [Figure 1B] 1 illustrates various sensors associated with an ego according to one embodiment.

[0025] [Figure 1C] 1 illustrates components of a vehicle, according to one embodiment.

[0026] [Figure 2]1 illustrates a block diagram of a video training system according to one embodiment.

[0027] [Figure 3] FIG. 1 illustrates a flowchart diagram of a method for accelerated training of machine learning (ML) models with video data, according to one embodiment.

[0028] [Figure 4] 1 shows a diagram depicting a set of selected image frames in a video sequence and corresponding image frames to be decoded, according to one embodiment.

[0029] [Figure 5] 5 shows a diagram of a bitstream corresponding to the video sequence of FIG. 4, compressed data corresponding to selected image frames, and compressed data corresponding to frames to be decoded, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0030] Reference will now be made to the exemplary embodiments illustrated in the drawings, and specific language will be used herein to describe the embodiments. It will nevertheless be understood that no limitation of the scope of the claims or this disclosure is intended thereby. Alterations and further modifications of the features of the invention shown herein, and further applications of the principles of the subject matter shown herein that will occur to those skilled in the art in possession of this disclosure, are to be considered within the scope of the subject matter disclosed herein. Other embodiments may be used and / or other changes may be made without departing from the spirit or scope of the present disclosure. The exemplary embodiments described in the detailed description are not meant to limit the presented subject matter.

[0031] Training an ML model using video data is typically very time-consuming. Furthermore, processing (e.g., decoding) a video sequence consumes significant processing power and memory. For ML or AI models that are adapted to be trained using vast amounts of video data, training (or validating) the model can be very time-consuming and demanding in terms of processing, memory, and bandwidth resources. For example, training an ML or AI model to predict or detect an ego's surroundings involves using millions or even billions of video frames for training data. The video frames are typically stored as compressed video data. Decoding and processing such vast amounts of video data to train an ML model can take thousands of hours. The systems, devices, and methods described herein provide accelerated training and more efficient use of computational and memory resources. In particular, the systems, devices, and methods described herein enable accelerated training and / or validation of an occupancy network configured to predict or detect an ego's three-dimensional surroundings.

[0032] FIG. 1A is a non-limiting example of system components capable of implementing the methods and systems discussed herein. For example, an analytics server may train an AI model and use the trained AI model to generate occupancy datasets and / or maps for one or more egos. FIG. 1A illustrates components of an AI-enabled visual data analytics system 100. System 100 may include analytics server 110a, system database 110b, administrator computing device 120, egos 140a-b (collectively ego 140), ego computing devices 141a-c (collectively ego computing devices 141), and server 160. System 100 is not limited to the components described herein and may include additional or other components not shown for the sake of brevity, which components should be considered within the scope of the embodiments described herein.

[0033] The above components may be connected via a network 130. Examples of network 130 may include, but are not limited to, a private or public LAN, a WLAN, a MAN, a WAN, and the Internet. Network 130 may include wired and / or wireless communications according to one or more standards and / or over one or more transport media.

[0034] Communications over network 130 may occur according to various communication protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), IEEE communications protocols, etc. In one example, network 130 may include wireless communications according to the Bluetooth® set of specifications or another standard or proprietary wireless communications protocol. In another example, network 130 may also include communications over cellular networks, including, for example, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), or Enhanced Data for Global Evolution (EDGE) networks.

[0035] System 100 illustrates an example of a system architecture and components that may be used to train and execute one or more AI models, such as AI model 110c. Specifically, as depicted in FIG. 1A and described herein, analytics server 110a may train AI model 110c using data retrieved from ego 140 (e.g., by using data streams 172 and 174) using methods described herein. Once AI model 110c is trained, each ego 140 may access the trained AI model 110c and execute it. For example, vehicle 141a with ego computing device 140a may send its camera feed to trained AI model 110c and determine the occupancy status of its surroundings (e.g., data stream 174). Furthermore, data captured and / or predicted by AI model 110c for ego 140 (during inference) may be used to improve AI model 110c. Thus, system 100 describes a continuous loop that can periodically improve the accuracy of AI model 110c. Furthermore, the system 100 loops in such a way that the data received by the ego 140 can be used in the learning phase as well as the inference phase.

[0036] Analytics server 110a may be configured to collect, process, and analyze navigation data (e.g., images captured while navigating) and various sensor data collected from ego 140. The collected data may then be processed and prepared into a training dataset. The training dataset may then be used to train one or more AI models, such as AI model 110c. Analytics server 110a may also be configured to collect visual data from ego 140. Using AI model 110c (trained using the methods and systems discussed herein), analytics server 110a may generate a dataset and / or an occupancy map for ego 140. Analytics server 110a may display the occupancy map on ego 140 and / or transmit the occupancy map / dataset to ego computing device 141, administrator computing device 120, and / or server 160.

[0037] Although FIG. 1A shows AI model 110c as a component of system database 110b, AI model 110c may be stored in a different or separate component, such as cloud storage or any other data repository accessible to analytics server 110a.

[0038] The analytics server 110a may also be configured to display an electronic platform showing various training attributes for training the AI ​​model 110c. The electronic platform may be displayed on the administrator computing device 120 to allow an analyst to monitor the training of the AI ​​model 110c. An example of an electronic platform generated and hosted by the analytics server 110a may be a web-based application or website configured to display the training dataset collected from the ego 140 and / or the training status / metrics of the AI ​​model 110c.

[0039] Analysis server 110a may be any computing device equipped with a processor and non-transitory machine-readable storage capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices may include workstation computers, laptop computers, server computers, etc. Although system 100 includes a single analysis server 110a, system 100 may include any number of computing devices operating in a distributed computing environment, such as a cloud environment.

[0040] Ego 140 may represent various electronic data sources that transmit data associated with a previous or current navigation session to analytics server 110a. Ego 140 may be any device configured for navigation, such as vehicle 140a and / or truck 140c. Ego 140 is not limited to being a vehicle and may include robotic devices as well. For example, ego 140 may include robot 140b, which may represent a general-purpose, bipedal, autonomous humanoid robot capable of navigating various terrains. Robot 140b may be provided with software that enables balance, navigation, perception, or interaction with the physical world. Robot 140b may also include various cameras configured to transmit visual data to analytics server 110a.

[0041] Although referred to herein as “ego,” ego 140 may or may not be an autonomous device configured for automatic navigation. For example, in some embodiments, ego 140 may be controlled by a human operator or by a remote processor. ego 140 may include various sensors, such as those depicted in FIG. 1B . The sensors may be configured to collect data as ego 140 navigates various terrains (e.g., roads). Analytics server 110a may collect data provided by ego 140. For example, analytics server 110a may obtain navigation session and / or road / terrain data (e.g., images of ego 140 navigating roads) from various sensors, such that the collected data is ultimately used by AI model 110c for training purposes.

[0042] As used herein, a navigation session corresponds to a trip in which ego 140 travels a route, regardless of whether the trip was autonomous or controlled by a human. In some embodiments, the navigation session may be for data collection and model training purposes. However, in some other embodiments, ego 140 may refer to a vehicle purchased by a consumer, and the purpose of the trip may be classified as daily use. A navigation session may begin when ego 140 travels more than a threshold distance (e.g., 0.1 miles, 100 feet) from a non-moving location or exceeds a threshold speed (e.g., greater than 0 mph, greater than 1 mph, greater than 5 mph). A navigation session may end when ego 140 is returned to a non-moving location and / or turned off (e.g., when the driver exits the vehicle).

[0043] Ego 140 may correspond to a group of egos monitored by analytics server 110a to train AI model 110c. For example, drivers of vehicles 140a may authorize analytics server 110a to monitor data associated with their respective vehicles. As a result, analytics server 110a may collect sensor / camera data using various methods discussed herein and generate a training dataset for training AI model 110c accordingly. Analytics server 110a may then apply the trained AI model 110c to analyze data associated with ego 140 and predict an occupancy map for ego 140. Furthermore, additional / ongoing data associated with ego 140 may be processed and added to the training dataset, allowing analytics server 110a to recalibrate AI model 110c accordingly. Thus, system 100 performs a loop in which navigation data received from ego 140 can be used to train AI model 110c. Ego 140 may include a processor that executes the trained AI model 110c for navigation purposes. During navigation, ego 140 can collect additional data about their navigation session and can use the additional data to calibrate AI model 110c. That is, ego 140 corresponds to an ego that can be used to train, run / use, and recalibrate AI model 110c. In a non-limiting example, ego 140 corresponds to a vehicle purchased by a customer that can navigate autonomously using AI model 110c and simultaneously improve AI model 110c.

[0044] Egos 140 may be equipped with various technologies that enable them to gather data from their surroundings and (potentially) navigate autonomously. For example, ego 140 may be equipped with an inference chip for running self-driving software.

[0045] Various sensors for each ego 140 may monitor and transmit collected data associated with different navigation sessions to analytics server 110a. FIGS. 1B-1C show block diagrams of sensors incorporated within ego 140, according to one embodiment. The number and location of each sensor discussed with respect to FIGS. 1B-1C may depend on the type of ego discussed in FIG. 1A. For example, robot 140b may include different sensors than vehicle 140a or truck 140c. For example, robot 140b may not include airbag activation sensor 170q. Additionally, the sensors for vehicle 140a and truck 140c may be in different locations than those shown in FIG. 1C.

[0046] As discussed herein, various sensors incorporated within each ego 140 may be configured to measure various data associated with each navigation session. Analytics server 110a may periodically collect the data monitored and collected by these sensors, which is processed according to methods described herein and used to train and / or execute AI model 110c to generate an occupancy map.

[0047] Ego 140 may include user interface 170a. User interface 170a may refer to the user interface of an ego computing device (e.g., ego computing device 141 of FIG. 1A). User interface 170a may be implemented as a display screen integrated with or coupled to a vehicle's interior, a head-up display, a touchscreen, etc. User interface 170a may include input devices such as a touchscreen, knobs, buttons, a keyboard, a mouse, a gesture sensor, a steering wheel, etc. In various embodiments, user interface 170a may be adapted to provide user input (e.g., as types of signals and / or sensor information) to other devices or sensors of ego 140 (e.g., the sensors shown in FIG. 1B), such as controller 170c.

[0048] User interface 170a may also be implemented with one or more logic devices that may be adapted to execute instructions, such as software instructions, that implement any of the various processes and / or methods described herein. For example, user interface 170a may be adapted to form a communication link, send and / or receive communications (e.g., sensor signals, control signals, sensor information, user input, and / or other information), or perform various other processes and / or methods. In another example, a driver may use user interface 170a to control the temperature of ego 140 or activate its features (e.g., autonomous driving or steering system 170o). Accordingly, user interface 170a may monitor and collect driving session data in conjunction with other sensors described herein. User interface 170a may also be configured to display various data generated / predicted by analytics server 110a and / or AI model 110c.

[0049] Orientation sensor 170b may be implemented as one or more of a compass, float, accelerometer, and / or other digital or analog device capable of measuring the orientation of ego 140 (e.g., the magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations, such as gravity and / or magnetic north). Orientation sensor 170b may be adapted to provide orientation measurements at ego 140. In other embodiments, orientation sensor 170b may be adapted to provide roll, pitch, and / or yaw rate of ego 140 using a time series of orientation measurements. Orientation sensor 170b may be positioned and / or adapted to provide orientation measurements relative to a particular coordinate frame of ego 140.

[0050] Controller 170c may be implemented as any suitable logic device (e.g., a processing device, microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), memory storage device, memory reader, or other device or combination of devices) that can be adapted to execute, store, and / or receive appropriate instructions, such as software instructions that implement control loops for controlling various operations of ego 140. Such software instructions may also process sensor signals, determine sensor information, provide user feedback (e.g., via user interface 170a), interrogate devices regarding operating parameters, select operating parameters for devices, or implement methods for performing any of the various operations described herein.

[0051] Communications module 170e may be implemented as any wired and / or wireless interface configured to communicate sensor data, configuration data, parameters, and / or other data and / or signals to any feature shown in FIG. 1A (e.g., analytics server 110a). As described herein, in some embodiments, communications module 170e may be implemented in a distributed manner, such that portions of communications module 170e are implemented within one or more elements and sensors shown in FIG. 1B. In some embodiments, communications module 170e may delay communication of sensor data. For example, when ego 140 does not have network connectivity, communications module 170e may store sensor data in temporary data storage and transmit the sensor data when ego 140 is identified as having adequate network connectivity.

[0052] Speed ​​sensor 170d may be implemented as an electronic pitot tube, a metering gear or wheel, a water speed sensor, a wind speed sensor, a wind speed sensor (e.g., direction and magnitude), and / or other device capable of measuring or determining the linear velocity of ego 140 (e.g., within the surrounding medium and / or aligned with the longitudinal axis of ego 140) and providing such measurement as a sensor signal that can be communicated to various devices.

[0053] Gyroscope / accelerometer 170f may be implemented as an electronic sextant, a semiconductor device, an integrated chip, an accelerometer sensor, or other system or device capable of measuring angular velocity / acceleration and / or linear acceleration (e.g., direction and magnitude) of ego 140 and providing such measurements as sensor signals that can be communicated to other devices, such as analytics server 110a. Gyroscope / accelerometer 170f may be positioned and / or adapted to make such measurements relative to a particular coordinate frame of ego 140. In various embodiments, gyroscope / accelerometer 170f may be mounted in a common housing and / or module with other elements shown in FIG. 1B to ensure a common frame of reference or known transformations between frames of reference.

[0054] Global Navigation Satellite System (GNSS) 170h may be implemented as a global positioning satellite receiver and / or another device capable of determining the absolute and / or relative position of ego 140 based on, for example, radio signals received from space-born and / or terrestrial sources and providing such measurements as sensor signals that can be communicated to various devices. In some embodiments, GNSS 170h may be adapted to determine the velocity, speed, and / or yaw rate of ego 140 (e.g., using a time series of position measurements), such as the absolute velocity and / or yaw component of the angular velocity of ego 140.

[0055] Temperature sensor 170i may be implemented as a thermistor, an electrical sensor, an electrical thermometer, and / or other device capable of measuring a temperature associated with ego 140 and providing such measurement as a sensor signal. Temperature sensor 170i may be configured to measure an environmental temperature associated with ego 140, such as a cockpit or dash temperature, which may be used to estimate the temperature of one or more elements of ego 140.

[0056] Humidity sensor 170j may be implemented as a relative humidity sensor, an electrical sensor, an electrical relative humidity sensor, and / or another device capable of measuring the relative humidity associated with ego 140 and providing such measurement as a sensor signal.

[0057] Steering sensor 170g may be adapted to physically adjust the orientation of ego 140 according to one or more control signals provided by a logic device, such as controller 170c, and / or user input. Steering sensor 170g may include one or more actuators and control surfaces of ego 140 (e.g., rudders or other types of steering or trim mechanisms) and may be adapted to physically adjust the control surfaces to various positive and / or negative steering angles / positions. Steering sensor 170g may also be adapted to sense the current steering angles / positions of such steering mechanisms and provide such measurements.

[0058] Propulsion system 170k may be implemented as a propeller, turbine, or other thrust-based propulsion system, a mechanical wheeled and / or tracked propulsion system, a wind / sail-based propulsion system, and / or other type of propulsion system that may be used to provide motive power to ego 140. Propulsion system 170k may also monitor the direction of motive power and / or thrust of ego 140 relative to a coordinate frame of reference of ego 140. In some embodiments, propulsion system 170k may be coupled to and / or integrated with steering sensor 170g.

[0059] Occupant restraint sensor 170l may monitor the seat belt detection and lock / unlock assembly, as well as other occupant restraint subsystems. Occupant restraint sensor 170l may include various environmental and / or status sensors, actuators, and / or other devices that facilitate operation of safety mechanisms associated with operation of ego 140. For example, occupant restraint sensor 170l may be configured to receive movement and / or status data from other sensors shown in FIG. 1B. Occupant restraint sensor 170l may determine whether a safety mechanism (e.g., a seat belt) is engaged.

[0060] Camera 170m may refer to one or more cameras integrated into ego 140, as depicted in FIG. 1C, or may include multiple cameras integrated into (or retrofitted to) ego 140. Camera 170m may be an inward-facing or outward-facing camera of ego 140. For example, as depicted in FIG. 1C, ego 140 may include one or more inward-facing cameras that can monitor and collect video of occupants of ego 140. Ego 140 may include eight outward-facing cameras. For example, ego 140 may include front camera 170m-1, forward-looking side camera 170m-2, forward-looking side camera 170m-3, rearward-looking side camera 170m-4 on each front fender, camera 170m-5 on each side (e.g., integrated into the B-pillar), and rear camera 170m-6.

[0061] 1B, radar 170n and ultrasonic sensor 170p may be configured to monitor the distance of ego 140 to other objects, such as other vehicles or immovable objects (e.g., trees or garage doors). Ego 140 may also include an automatic driving or steering system 170o configured to autonomously navigate ego 140 using data collected via various sensors (e.g., radar 170n, speed sensor 170d, and / or ultrasonic sensor 170p).

[0062] Thus, automated driving or steering system 170o may analyze various data collected by one or more sensors described herein to identify driving data. For example, automated driving or steering system 170o may calculate the risk of a forward collision based on ego 140's speed and its distance to another vehicle on the road. Autonomous driving or steering system 170o may also determine whether the driver is touching the steering wheel. Autonomous driving or steering system 170o may transmit the analyzed data to various features discussed herein, such as an analytics server.

[0063] Airbag deployment sensor 170q may predict or detect a crash and cause one or more airbags to deploy or inflate. Airbag deployment sensor 170q may transmit data regarding the deployment of the airbags, including data associated with the event that caused the deployment.

[0064] 1A , administrator computing device 120 may represent a computing device operated by a system administrator. Administrator computing device 120 may be configured to display data retrieved or generated by analytics server 110a (e.g., various analytics metrics and risk scores), allowing the system administrator to monitor various models utilized by analytics server 110a, review feedback, and / or facilitate training of AI models 110c maintained by analytics server 110a.

[0065] Ego 140 may be any device configured to navigate various routes, such as vehicle 140a or robot 140b. As discussed with respect to FIGS. 1B-1C, ego 140 may include various telemetry sensors. Ego 140 may also include ego computing device 141. Specifically, each ego may have its own ego computing device 141. For example, truck 140c may have ego computing device 141c. For simplicity, ego computing devices are collectively referred to as ego computing device 141. Ego computing device 141 may control the presentation of content on ego 140's infotainment system, process commands related to the infotainment system, aggregate sensor data, manage communication of data to electronic data sources, receive updates, and / or send messages. In one configuration, ego computing device 141 communicates with an electronic control unit. In another configuration, ego computing device 141 is an electronic control unit. Ego computing device 141 may include a processor and non-transitory machine-readable storage media capable of performing the various tasks and processes described herein. For example, AI model 110c described herein may be stored and executed (or directly accessed) by ego computing device 141. Non-limiting examples of ego computing device 141 may include a vehicle multimedia and / or display system.

[0066] In one example of a method for accelerating the training of AI model 110c and / or other ML models with video data, analytics server 110a can include multiple graphical processing units (GPUs) configured to train AI model 110c in parallel. For example, analytics server 110a can include a supercomputer. Each GPU can receive video data (e.g., one or more bitstreams) and a representation of a video frame (or image frame) to be extracted from the video data and used to train AI model 110c. The GPU can decode only a portion of the bitstream necessary to decode a selected image frame and train AI model 110c using the selected image frame in decoded form.

[0067] In some implementations, each GPU may be configured or designed to perform the training steps independently without using external resources. In particular, the GPU may receive video data, decode relevant portions or segments to extract selected image frames, extract features from the selected image frames, and use the extracted features to train the AI ​​model 110c without using external memory or processing resources. In other words, all processing and data handling, from receiving the video data to training the AI ​​model 110c, may be performed internally and independently within the GPU.

[0068] Each GPU can include one or more hardware video decoders built in to accelerate video decoding. A GPU can include multiple video decoders to parallelize video decoding. Parallelization can be implemented in various ways, for example, per video segment, per bitstream, or per training session.

[0069] In some implementations, the GPU may have sufficient internal memory, e.g., cache memory, to store data necessary to perform a training step. The GPU may allocate memory regions within its memory to store data associated with the training step and use the allocated memory regions for data storage throughout the training step.

[0070] 2 shows a block diagram of a computing environment 200 for training an ML model, according to one embodiment. The computing environment 200 may include a training system 202 for training the ML model and a data storage system 204 for storing training data and / or validation data. The training system 202 may include multiple training nodes (or processing nodes) 206. Each training node 206 may include a respective data loader (or data loading device) 208 and a respective graphical processing unit (GPU) 210. Each GPU 210 may include memory, such as a cache memory 212, processing circuitry 214, and one or more video decoders 216.

[0071] Data storage system 204 can include or be a distributed storage system. For example, data storage system 204 can have an infrastructure that can split data across multiple physical servers, such as a supercomputer. Data storage system 204 can include one or more storage clusters of storage units, along with mechanisms and infrastructure for parallel and accelerated access of data from multiple nodes or storage units of the storage cluster. For example, data storage system 204 can include sufficient data links and bandwidth to deliver data to training nodes 206 in parallel or simultaneously.

[0072] Data storage system 204 may include sufficient memory capacity to store millions or even billions of video frames, e.g., in compressed format. For example, data storage system 204 may have memory capacity to store multiple petabytes of data, e.g., 10, 20, or 30 petabytes. Data storage system 204 may allow thousands of video sequences to move in and out of data storage system 204 at any given time instance. The relatively large size and bandwidth of storage system 204 enables parallel training of one or more ML models, as discussed below.

[0073] Training system 202 may be implemented as one or more physical servers, such as server 110a. For example, training system 202 may be implemented as one or more supercomputers. Each supercomputer may include thousands of processing or training nodes 206. Training nodes 206 may be configured or designed to support parallel training of one or more ML models, such as AI model 110c. Each training node 206 may be communicatively coupled to storage system 204 and may access training data and / or validation data stored therein.

[0074] Each training node 206 may include a respective data loader 208 and a respective GPU 210 that are communicatively coupled to each other. The data loader 208 may be (or may include) a processor or central processing unit (CPU) for handling data requests or data transfers between a corresponding GPU 210, e.g., a GPU 210 within the same training node 206, and the data storage system 204. For example, the GPU 210 may request one or more video sequences captured by one or more of the cameras 170m described in connection with FIG. 1C . For example, the front or forward-looking cameras 170m-1, 170m-2, and 170m-3, the rear-looking side camera 170m-4, the side camera 170m-5, and the rear camera 170m-6 may simultaneously capture video sequences and send the video sequences to the data storage system 204 for storage in the data storage system 204. In some implementations, data storage system 204 can store video sequences captured simultaneously by multiple cameras, such as camera 170m of ego 140, as bundles or combinations of video sequences that can be distributed together to training node 206. For example, data storage system 204 can maintain additional data indicating which video sequences were captured simultaneously by camera 170m or represent the same scene from different camera angles. Data storage system 204 may, for example, maintain data indicating, for each stored video sequence, an ego identifier, a camera identifier, and a time instance associated with the video sequence.

[0075] The training node 206 can simultaneously train one or more ML models, e.g., in parallel, using video data captured by the ego 140's cameras 170m and stored in the data storage system 204. In some implementations, video data can be captured by multiple ego 170's cameras 140m. In the training node 206, a corresponding data loader 208 can request video data for one or more video sequences captured simultaneously during a time interval by one or more ego 140's cameras 170m from the data storage system 204 and send the received video data to a corresponding GPU 210 for use in performing a training step (or validation step) when training the ML model. The video data can be in a compressed format. For example, the video sequences can be encoded by an encoder built into or implemented in the ego 140. Each data loader 208 can have sufficient processing power and bandwidth to deliver video data to a corresponding GPU 210 so as to keep the GPU 210 busy. In other words, GPU 210 may be configured or designed, for example, in terms of processing power and bandwidth, to request video data for a bundle of compressed video sequences from data storage system 204 and deliver the video to GPU 210 in a period of time that is equal to or less than the average time consumed by GPU 210 to process the bundle of video sequences.

[0076] Each GPU 210 may include a corresponding internal memory 212, such as a cache memory, for storing executable instructions for performing processes described herein, received video data of one or more video sequences, decoded video frames, features extracted from the decoded video frames, parameters, or data for the trained ML model, and / or other data used to train the ML model. The memory 212 may be large enough to store all data necessary to perform a single training step. As used herein, a training step may include receiving and decoding video data of one or more video sequences (e.g., a bundle of video sequences simultaneously captured by one or more cameras 170m of ego 140), extracting features from the decoded video data, and using the extracted features to update parameters of the ML model being trained or validated.

[0077] Each GPU 210 may include processing circuitry 214 for performing the processes or methods described herein. Processing circuitry 214 may include one or more microprocessors, multi-core processors, digital signal processors (DSPs), one or more logic circuits, or a combination thereof. Processing circuitry 214 may execute computer code instructions stored, for example, in memory 212, to perform the processes or methods described herein.

[0078] The GPU 210 may include one or more video decoders 216 for decoding video data received from the data storage system 204. The one or more video decoders 216 may include hardware video decoders integrated into the GPU 110 to accelerate video decoding. The one or more video decoders 216 may be part of the processing circuitry 214 or may include separate electronic circuitry communicatively coupled to the processing circuitry 214.

[0079] Each GPU 210 may be configured or designed to handle or perform the training steps without the use of external resources. The GPUs 210 may include sufficient memory capacity and processing power to perform the training steps. The processes performed by a training node 208 or corresponding GPU 210 are described in further detail below in connection with Figures 3-6.

[0080] Referring now to FIG. 3, a flowchart diagram of a method 300 for accelerated training of a machine learning (ML) model using video data is shown, according to one embodiment. Briefly, the method 300 may include receiving a bitstream of a video sequence and one or more indications of one or more image frames of the video sequence to train the machine learning (ML) model (STEP 302), and parsing the bitstream to determine timestamps, positions within the bitstream, and types of the image frames of the video sequence (STEP 304). The method 300 may also include using the one or more indications and the timestamps, positions within the bitstream, and types of the image frames of the video sequence to determine one or more segments of the bitstream for decoding to extract the one or more image frames (STEP 306). For an image frame of the one or more image frames, the corresponding segment may represent a corresponding reference chain of image frames of the video sequence. The method 300 may include decoding one or more segments of the bitstream (step 308) and using one or more image frames to train an ML model (step 310).

[0081] Method 300 may be implemented, performed, or executed entirely by GPU 210. GPU 210 may perform steps 302-310 without using external memory or processing resources. Performing method 300 entirely by a single GPU 210 may accelerate the training of ML models. In particular, performing method 300 entirely within a single GPU 210 may reduce processing time by avoiding data exchange between GPU 210 and any external resources. For example, decoding video data or storing decoded video data external to GPU 210 may result in delays associated with exchanging compressed and / or decoded video data between GPU 210 and any external resources.

[0082] Method 300 may include GPU 210 receiving a bitstream of a video sequence and one or more representations showing one or more image frames of the video sequence to train a machine learning (ML) model (STEP 302). In some implementations, GPU 210 or processing circuitry 214 may allocate memory space in memory 212 for a training step to be performed. For example, before receiving the bitstream and one or more representations, processing circuitry 214 may allocate memory space to store data for the next training step, such as compressed video data received from storage system 204, decoded video or image frames, features extracted from video frames, and / or other data. In some implementations, processing circuitry 214 may allocate memory space at the beginning of each training step, or may allocate memory space at the beginning of a training session and use the allocated memory space for successive training steps. In some implementations, processing circuitry 214 may overwrite segments of allocated memory space (or memory 212) that store data that is no longer needed to efficiently use memory 212.

[0083] The GPU 210 can receive one or more bitstreams of one or more video sequences from the storage system 204 via the loader 208. For example, the GPU 210 can receive multiple bitstreams of multiple compressed video sequences captured simultaneously, e.g., by cameras 170m of the ego 140. The compressed video sequences can be encoded at the ego 140 and can be unsynchronized. For example, each camera 170m can have a separate timeline on which image frames are captured and a separate encoder for encoding the captured image frames. Image capture time instances at different cameras 170m can be time-aligned. Also, encoders associated with different cameras 170m can be unsynchronized. Thus, image frames captured at the same time instance by different cameras 170m (or at substantially the same time instance, taking into account differences in the timelines for capturing image frames by different cameras 170m) can have different timestamps when encoded by separate encoders in separate bitstreams.

[0084] The GPU 210 can receive one or more indicators indicating image frames selected or to be selected from one or more video sequences to use for training the ML model. The one or more indicators can be specified by a user of the training system 202 and received by the GPU 210 as input, for example, via the data loader 208. The one or more indicators can include one or more time values. Each time value can indicate a separate image frame within each received bitstream. For example, if eight bitstreams of eight video sequences captured by eight different cameras 170m of the ego 140 are received by the GPU 210, each time value can indicate eight image frames, e.g., an image frame within each video sequence. The time values ​​can indicate, but are not necessarily exactly equal to, the timestamp of the image frame selected or to be selected. The GPU 210 can store the received bitstreams and one or more indicators in an allocated memory area.

[0085] Method 300 may include a step (STEP 304) in which GPU 210 parses the bitstream to determine timestamps, positions within the bitstream, and types of image frames of the video sequence. Each video bitstream may include multiple headers distributed throughout the bitstream. The video bitstream may include a separate header for each compressed image frame in the bitstream. Each image frame header may immediately precede the compressed data for the image frame and may include data indicating information about the image frame and the corresponding compressed data. The header may include the image frame timestamp, the image frame type, the size of the compressed image frame data, the position of the compressed image frame data within the bitstream, and / or other data.

[0086] The type of image frame can include an intra-frame (I-frame) type or a predicted frame (P-frame). A P-frame is encoded using data from another previously encoded image frame. To decode a P-frame, a decoder decodes any other image frames on which the P-frame depends before decoding the P-frame. An I-frame is also called a reference frame and can be decoded independently of any other image frame. For an image frame, the corresponding timestamp represents the presentation time of the image frame, for example, relative to the first image frame in the video sequence. The location of the compressed image frame data within the bitstream can be, for example, an offset value in bytes, indicating where the compressed image frame data begins within the bitstream.

[0087] The processing circuitry 214 can parse each received bitstream to identify or determine the header of each compressed image frame within the bitstream. The processing circuitry 214 can read the header within each bitstream to determine the timestamp, location of compressed image frame data, and type of each image frame within each bitstream. Parsing a bitstream is significantly less expensive in terms of processing power and time compared to decoding a single bitstream or a portion thereof. The timestamp, location within the bitstream, and type of image frame determined by parsing the bitstream can significantly reduce the amount of video data that needs to be decoded to extract selected image frames, thus significantly accelerating the training process.

[0088] Upon determining the timestamps, positions within the bitstream, and types of the image frames of the video sequence, the GPU 210 or the processing circuitry 214 may generate one or more data structures for storing the timestamps, positions within the bitstream, and types of the image frames of the video sequence. The one or more data structures may include a table, a data file, and / or some other type of data structure. For example, the processing circuitry 214 may generate a separate table similar to Table 1 below for each bitstream. The first leftmost column of Table 1 may include the timestamps of all image frames in the video sequence, e.g., in ascending order; the second column may include the position (or offset) of the compressed frame data for each image frame; and the rightmost column may include the type of each image frame. Each row of Table 1 corresponds to a separate image frame in the video sequence. [Table 1]

[0089] In some implementations, the one or more data structures may include a first data structure for an I-frame and a second data structure for all frames in the video sequence. Each of the first and second data structures may be a table or a data file. An example of the first data structure may be Table 2 below. Each row of Table 2 represents a distinct I-frame in the video sequence. The first (e.g., left-most) column may contain the timestamps of the I-frames (e.g., in ascending order), and the second column may contain the corresponding positions or offsets (e.g., in bytes). The data in Table 2 allows for quick determination of the location of compressed data for any I-frame in the bitstream. [Table 2]

[0090] An example of a first data structure may be Table 3 below. Each row of Table 3 represents a distinct image frame in a video sequence. The first (e.g., left-most) column may contain the timestamps of the image frames (e.g., in ascending order), and the second column may contain the corresponding positions or offsets (e.g., in bytes). The data in Table 3 allows for the determination of the location of the compressed data for any image frame in the bitstream. [Table 3]

[0091] The data in any of the above tables may be stored in a data file. Compared to Table 1, Table 2 and Table 3 may not include an indication of the image frame type. Instead, I-frames may be identified from Table 2, which is specific to I-frames. In some implementations, other data structures may be generated (e.g., instead of or in combination with any of Tables 1, 2, and / or 3). Once generated, GPU 210 or processing circuitry 214 may store one or more data structures in memory 212. The one or more data structures may indicate (i) for each image frame of the video sequence, a corresponding timestamp and a corresponding offset indicating a corresponding location of the compressed data for the image frame within the bitstream, and (ii) a particular type of image frame within the video sequence.

[0092] The method 300 may include a step (STEP 306) in which the GPU 210 determines one or more segments of the bitstream for decoding to extract one or more image frames using the one or more indications and the timestamps, positions within the bitstream, and types of the image frames in the video sequence. The one or more indications may include one or more time values ​​for use in identifying or determining image frames selected for use in training the ML model. Each time value may indicate one or more corresponding image frames or corresponding timestamps within the one or more received bitstreams. If multiple bitstreams corresponding to multiple cameras 170m are received, each indicator or time value may indicate a separate image frame or corresponding timestamp within each received bitstream.

[0093] For each time value of the one or more time values, GPU 210 or processing circuitry 214 may determine a corresponding timestamp that is closest to the time value in each received bitstream. For example, processing circuitry 214 may use Table 3 (or Table 1) for a given bitstream to determine the timestamp of the bitstream that is closest to the time value. Processing circuitry 214 may determine a separate timestamp that is closest to the time value for each bitstream by using a corresponding data structure (e.g., Table 3 or Table 1). Each determined timestamp indicates a respective image frame in the corresponding bitstream that is indicated or selected via the time value. For example, if eight bitstreams corresponding to eight cameras 170m are received, processing circuitry 214 may determine, for each time value (or indicator), eight corresponding timestamps that indicate eight selected image frames, with one selected image frame from each bitstream. If the GPU 210 receives three indicators and eight bitstreams corresponding to eight cameras 170m, the processing circuit 214 can determine a total of 24 timestamps corresponding to the 24 selected image frames, with three timestamps corresponding to the three selected image frames being determined for each bitstream.

[0094] Processing circuit 214 may determine a separate second timestamp in each received bitstream for each time value (or each indicator). For each bitstream, the second timestamp indicates the corresponding I-frame of the bitstream. For each time value, processing circuit 214 may determine the second timestamp in a given bitstream as the timestamp of the I-frame of the bitstream that is closest to the time value (or the timestamp of the selected frame of the bitstream indicated by the time value) that is less than or equal to the time value (or less than or equal to the timestamp of the selected frame of the bitstream indicated by the time value). For example, processing circuit 214 may use Table 2 (or Table 1) for the bitstream to determine the timestamp of the I-frame of the bitstream that is closest to and less than or equal to the given time value (or indicator).

[0095] For a given time value (or indicator) and a given bitstream, if the timestamp of the selected image frame and the timestamp of the corresponding I-frame are equal, it means that the selected image frame (or the image frame indicated by the time value) is an I-frame. However, if the two timestamps are different, processing circuitry 214 can determine that the selected image frame (or the image frame indicated by the time value) is not an I-frame.

[0096] For each time value (or indicator) and each received bitstream, processing circuitry 214 can use the I-frame timestamp and one or more data structures to determine the location or offset (e.g., starting position) of the I-frame within the bitstream. For example, processing circuitry 214 can use Table 2 (or Table 1) to determine the location or offset corresponding to the determined I-frame timestamp. For example, if the determined I-frame timestamp is T 3,I If , the corresponding offset is O 3,I is.

[0097] For each time value (or indicator) and each received bitstream, the processing circuitry 214 can use the timestamp of the corresponding selected image frame to determine the end position of the corresponding selected image frame. The processing circuitry 214 can determine the end position of the selected image frame in the bitstream as the start position of the next (or subsequent) image frame in the bitstream. The processing circuitry 214 can determine the end position of the selected image frame in the bitstream as the start position of the selected image frame plus the size of the compressed data of the bitstream. The processing circuitry 214 can determine the size of each image frame in the bitstream by parsing the bitstream or frame headers and recording the size in one or more data structures.

[0098] For each time value (or indicator) and each received bitstream, processing circuitry 214 can determine a corresponding segment of the bitstream extending between the start of the corresponding I-frame and the end of the corresponding selected image frame. The determined segment represents the minimum amount of compressed video data that must be decoded to decode the selected image frame. If two or more segments corresponding to different time values ​​(or different indicators) but from the same bitstream overlap, processing circuitry 214 can consider the longest segment for decoding and omit shorter segments.

[0099] It should be noted that although the indicator is described above as a time value, other implementations are possible. For example, the indicator received by GPU 210 may be an image frame index.

[0100] Referring now to FIG. 4, a diagram illustrating a set of selected image frames in a video sequence 400 and corresponding image frames to be decoded is shown, according to one embodiment. Processing circuitry 214 can use three different indicators or time values ​​to determine image frames 402, 406, and 412 as the selected image frames or image frames indicated to be selected by the indicators or time values ​​(e.g., as described above). Image frame 402 is an I-frame, and image frames 406 and 412 are P-frames. With respect to selected image frame 402, processing circuitry 214 determines that only image frame 402 should be decoded because it is an I-frame and does not depend on other image frames. With respect to selected image frame 406, which is a P-frame, processing circuitry 214 determines that the closest preceding I-frame is image frame 404, and that both frames 404 and 406 should be decoded to obtain selected image frame 406 in decoded form. For the selected image frame 412, which is a P-frame, the processing circuitry 214 determines that the closest preceding I-frame is image frame 408, and that frames 408, 410, and 412 should be decoded to obtain the selected image frame 412 in decoded form.

[0101] 5 shows a diagram of a bitstream 500 corresponding to video sequence 400 of FIG. 4, compressed data corresponding to selected image frames, and compressed data corresponding to frames to be decoded, according to one embodiment. Compressed data segment 502 represents the compressed data for image frame 402. Compressed data segment 504 represents the compressed data for image frames 404 and 406. Compressed data segment 506 represents the compressed data for image frames 408, 410, and 412. Instead of decoding the entire bitstream 500, processing circuitry 214 can provide only compressed data segments 502, 504, and 506 to video decoder 216 for decoding.

[0102] For each selected image frame (or each image frame indicated for selection), the corresponding compressed data segment to be decoded can be considered to represent a corresponding reference chain of image frames. The reference chain begins with the nearest I-frame preceding the selected frame and ends with the selected image frame. The reference chain represents a chain of interdependent image frames with dependencies (frame references) starting from the selected image frame and proceeding all the way back to the first encountered I-frame. For example, in the reference chain formed by image frames 408, 410, and 412, image frame 412 references image frame 410, which in turn references image frame 410, which is an I-frame.

[0103] Referring again to FIG. 3 , method 300 may include a step in which GPU 210 decodes one or more segments of the bitstream (step 308). Processing circuit 214 may provide or supply the determined compressed data segments to video decoder 216 for decoding. By decoding only the compressed data needed to decode the selected image frames, GPU 210 significantly reduces the processing time and processing power consumed to decode the selected image frames (e.g., compared to decoding the entire bitstream). Processing circuit 214 may store the decoded video data in memory 212. For efficient use of memory resources in GPU 210, video processing circuit 214 may overwrite decoded video data that is no longer needed. For example, when decoding compressed data segment 506, once image frame 410 is decoded, processing circuit 214 may determine that the data for decoded image frame 410 is no longer needed and delete decoded image frame 410 to free up memory space. Once the compressed data segment corresponding to a selected image frame has been decoded, only the decoded data for the selected image frame can be retained in memory 212 or the allocated memory area, and other decoded image frames (non-selected image frames) can be deleted to free up memory space.

[0104] As mentioned above, the GPU 210 can include multiple video decoders that can operate in parallel. The GPU 210 can perform parallel video decoding in various ways. For example, the processing circuit 214 can assign different compressed data segments (regardless of the corresponding bitstream) to different video decoders 216 to keep all video decoders 216 busy and speed up video decoding of the segments. In some implementations, the processing circuit 214 can assign different bitstreams (or compressed data segments thereof) to different video decoders 216. In some implementations, the GPU 210 can simultaneously receive video bitstreams in multiple sessions (e.g., bitstreams captured by different egos 140 or captured at different time intervals). The processing circuit 214 can assign different video decoders 216 to decode video data of different sessions.

[0105] Method 300 may include GPU 210 using one or more image frames to train an ML model (step 310). Once the selected image frames are decoded, processing circuitry 214 may extract one or more features from each selected image frame and provide the extracted features to a training module configured to train the ML model. In response, the training module may modify or update one or more parameters of the ML model.

[0106] Although method 300 is described above as being performed or executed by GPU 210, in general, method 300 may be performed or executed by any computing system that includes memory and one or more processors. Also, other types of processors may be used in place of GPU 210.

[0107] The various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure or the claims.

[0108] Computer software-implemented embodiments may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A code segment or machine-executable instruction may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0109] The actual software code or specialized control hardware used to implement these systems and methods is not a limitation of the claimed features or this disclosure. Accordingly, the operation and behavior of the systems and methods are described without reference to specific software code, with the understanding that software and control hardware can be designed to implement the systems and methods based on the description herein.

[0110] If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory, computer-readable, or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. Non-transitory processor-readable storage media may be any available medium that can be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer or processor. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), Blu-ray discs, and floppy disks, with a "disc" typically reproducing data magnetically and a "disc" reproducing data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media. Furthermore, the operations of a method or algorithm may reside as one or any combination or set of code and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0111] The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the embodiments described herein and variations thereof. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the spirit or scope of the subject matter disclosed herein. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0112] While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various disclosed aspects and embodiments are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

1. receiving, by a processor, a bitstream of a video sequence and one or more representations of one or more image frames of the video sequence for training a machine learning model; parsing the bitstream with the processor to determine timestamps, positions within the bitstream, and types of image frames of the video sequence; determining, by the processor, one or more segments of the bitstream for decoding to extract the one or more image frames using the one or more representations and the timestamps, positions within the bitstream, and types of the image frames of the video sequence, wherein, for an image frame of the one or more image frames, a corresponding segment represents a corresponding reference chain of image frames of the video sequence; decoding, by the processor, the one or more segments of the bitstream; using, by the processor, the one or more image frames to train the machine learning model; A method comprising:

2. allocating, by the processor, a memory region within a memory of the processor; storing, by the processor, the bitstream in the allocated memory area; storing, by the processor, the one or more image frames in the allocated memory area after decoding the one or more segments; The method of claim 1 further comprising:

3. 2. The method of claim 1, wherein for each image frame of the one or more image frames, the corresponding reference chain for the image frame starts with an intra-frame (I-frame) of the video sequence and ends with the image frame of the one or more image frames.

4. 2. The method of claim 1, wherein determining the timestamps, positions within the bitstream, and types of the image frames of the video sequence includes generating one or more data structures, the one or more data structures indicating, for each image frame of the video sequence, a corresponding timestamp and a corresponding offset that represent a corresponding position of the compressed data for the image frame within the bitstream, as well as a particular type of image frame within the video sequence.

5. The method of claim 1 , wherein the processor is a graphical processing unit.

6. The method of claim 1 , wherein the one or more segments of the bitstream are decoded by a hardware decoder integrated into the processor.

7. The one or more representations include one or more time values, and determining the one or more segments of the bitstream comprises, for each time value of the one or more time values: determining a first timestamp among the timestamps of the image frames of the video sequence that is closest to the time value, the first timestamp corresponding to a first image frame in the video sequence; determining a second timestamp of an I-frame of the video sequence, the second timestamp being determined as an I-frame timestamp closest to the first timestamp that is less than or equal to the first timestamp; using the second timestamp to determine the location of the image frame within the bitstream where the I-frame begins; using the first timestamp to determine one of the positions of the image frame where the first image frame ends within the bitstream; determining a segment of the bitstream to extend between the starting position of the I-frame and the ending position of the first image frame; The method of claim 1 , comprising:

8. The bitstream is a first bitstream of a first compressed video sequence captured by a first camera, and the method includes: receiving, by the processor, a second bitstream of a second video sequence captured by a second camera; parsing, by the processor, the second bitstream to determine timestamps, positions within the second bitstream, and types of image frames of the second video sequence; determining, by the processor, one or more segments of the second bitstream for decoding to extract one or more second image frames of the second video sequence using the one or more indications and the timestamp, position within the second bitstream, and type of the image frames of the second video sequence, wherein, for an image frame of the one or more second image frames, a corresponding segment of the second bitstream represents a corresponding reference chain of image frames of the second video sequence; decoding, by the processor, the one or more segments of the second bitstream; using, by the processor, the one or more second image frames to train the ML model; The method of claim 1 further comprising:

9. The method of claim 8 , wherein the first camera and the second camera are not synchronized with each other.

10. 10. The method of claim 9, wherein at least two of the one or more segments of the first bitstream and the one or more segments of the second bitstream are decoded in parallel by at least two hardware decoders integrated into the processor.

11. Memory and A processing circuit, receiving a bitstream of a video sequence and one or more representations of one or more image frames of the video sequence to train a machine learning model; Parsing the bitstream to determine timestamps, positions within the bitstream, and types of image frames of the video sequence; determining one or more segments of the bitstream for decoding to extract the one or more image frames using the one or more representations and the timestamps, positions within the bitstream, and types of image frames of the video sequence, wherein for an image frame of the one or more image frames, a corresponding segment represents a corresponding reference chain of image frames of the video sequence; decoding the one or more segments of the bitstream; using the one or more image frames to train the machine learning model; a processing circuit configured to A computing device comprising:

12. The processing circuitry Allocating a memory region within the memory; storing the bitstream in the allocated memory area; storing the one or more image frames in the allocated memory area after decoding the one or more segments; The computing device of claim 11 further configured to:

13. 12. The computing device of claim 11, wherein for each image frame of the one or more image frames, the corresponding reference chain for the image frame begins with an intra-frame (I-frame) of the video sequence and ends with the image frame of the one or more image frames.

14. Upon determining the timestamps, positions within the bitstream, and types of the image frames of the video sequence, the processing circuitry is configured to generate one or more data structures, the one or more data structures comprising: for each image frame of the video sequence, a corresponding timestamp and a corresponding offset indicating a corresponding location of the compressed data for that image frame within the bitstream; and a particular type of image frame in said video sequence; 12. The computing device of claim 11, wherein:

15. The computing device of claim 11 , wherein the computing device is a graphical processing unit (GPU).

16. 16. The computing device of claim 15, wherein the one or more segments of the bitstream are decoded by a hardware decoder built into the GPU.

17. The one or more representations include one or more time values, and in determining the one or more segments of the bitstream, the processing circuitry, for each time value of the one or more time values: determining a first timestamp among the timestamps of the image frames of the video sequence that is closest to the time value, the first timestamp corresponding to a first image frame in the video sequence; determining a second timestamp of an I-frame of the video sequence, the second timestamp being determined as an I-frame timestamp closest to the first timestamp that is less than or equal to the first timestamp; using the second timestamp to determine the location of the I-frame in the bitstream among the locations of the image frames; using the first timestamp to determine the location of the image frame within the bitstream where the first image frame ends; determining a segment of the bitstream to extend between the starting position of the I-frame and the ending position of the first image frame; The computing device of claim 11 configured to:

18. the bitstream is a first bitstream of a first compressed video sequence captured by a first camera, and the processing circuitry receiving a second bitstream of a second video sequence captured by a second camera; Parsing the second bitstream to determine timestamps, positions within the second bitstream, and types of image frames of the second video sequence; determining one or more segments of the second bitstream for decoding to extract one or more second image frames of the second video sequence using the one or more indications and the timestamps, positions within the second bitstream, and types of the image frames of the second video sequence, wherein for an image frame of the one or more second image frames, a corresponding segment of the second bitstream represents a corresponding reference chain of image frames of the second video sequence; decoding the one or more segments of the second bitstream; using the one or more second image frames to train the ML model. The computing device of claim 11 further configured to:

19. The computing device of claim 18 , wherein the first camera and the second camera are not synchronized with each other.

20. 20. The computing device of claim 19, wherein at least two of the one or more segments of the first bitstream and the one or more segments of the second bitstream are decoded in parallel by at least two hardware decoders integrated into the computing device.

21. A non-transitory computer-readable medium storing computer code instructions that, when executed by a processor, cause the processor to: receiving a bitstream of a video sequence and one or more representations of one or more image frames of the video sequence to train a machine learning model; parsing the bitstream to determine timestamps, positions within the bitstream, and types of image frames of the video sequence; determining one or more segments of the bitstream for decoding to extract the one or more image frames using the one or more indications and the timestamps, positions within the bitstream, and types of image frames of the video sequence, wherein for an image frame of the one or more image frames, a corresponding segment represents a corresponding reference chain of image frames of the video sequence; Decoding the one or more segments of the bitstream; using the one or more image frames to train the machine learning model; Non-transitory computer-readable medium.

Images