Online adaptation of segmentation machine learning systems

Abstract

Techniques and systems are provided for performing online adaptation of machine learning models. For example, a process may include obtaining features extracted from an image by a machine learning model during inference; and determining, by the machine learning model, keypoint estimates in the image and / or boundary region estimates associated with objects in the image based on the features during inference. The process may also include generating a dummy tag based on the plurality of keypoint estimates and / or the boundary region estimate. The process may include determining at least one self-supervised loss based on the plurality of keypoint estimates and / or the boundary region estimate. The process may also include adapting a parameter of the machine learning model based on the at least one self-supervised loss. The process may include generating a segmentation mask for the image (or another image) using the machine learning model with the adapted parameters.

Description

Technical Field

[0001] The present disclosure as a whole relates to processing image data to perform segmentation (e.g., semantic segmentation, instance segmentation, etc.) For example, aspects of the present disclosure include systems and techniques for providing online adaptation (also referred to as test-time adaptation or inference-based adaptation) of a segmentation machine learning system (e.g., a segmentation neural network). Background Art

[0002] Devices or systems (e.g., autonomous vehicles, such as autonomous and semi-autonomous vehicles, drones or unmanned aerial vehicles (UAVs), mobile robots; mobile devices, such as mobile phones, extended reality (XR) devices, and other suitable devices or systems) increasingly include multiple sensors for collecting information about the environment, and processing systems for processing the information for various purposes, such as for route planning, navigation, collision avoidance, etc. An example of such a system is an advanced driver assistance system (ADAS) for an autonomous or semi-autonomous vehicle.

[0003] The device or system may perform segmentation on sensor data (e.g., one or more images) to generate a segmentation output (e.g., a segmentation mask or segmentation map). Based on the segmentation, objects may be identified and labeled with corresponding classifications of specific objects (e.g., people, cars, background, etc.) within the image or video. Labeling may be performed on a per-pixel basis. The segmentation mask may be a representation of the label of the image or view. The segmentation output may then be used to perform one or more operations, such as image processing (e.g., blurring a portion of an image). Summary of the invention

[0004] The following presents a simplified summary of the invention related to one or more aspects disclosed herein. Therefore, the following summary of the invention should neither be considered as an exhaustive overview related to all conceived aspects, nor should it be considered to identify key or decisive elements related to all conceived aspects or to delineate the scope associated with any particular aspect. Accordingly, the following overview presents certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form before the detailed description presented below.

[0005] Described herein are systems and techniques for providing online adaptation of a segmentation machine learning system (e.g., a segmentation neural network). According to at least one example, a processor-implemented method (e.g., for performing online adaptation of one or more machine learning models) is provided. The method includes: obtaining features extracted by a machine learning model from a first image during inference; determining, by the machine learning model, at least one of a plurality of keypoint estimates in the first image or boundary region estimates associated with an object in the first image based on the features during inference; generating one or more pseudo labels based on at least one of the plurality of keypoint estimates or the boundary region estimates; determining at least one self-supervised loss based on at least one of the plurality of keypoint estimates or the boundary region estimates; self-adapting one or more parameters of the machine learning model based on the at least one self-supervised loss; and generating a segmentation mask for the first image or a second image using the machine learning model with the adapted one or more parameters.

[0006] In another example, a device for performing online adaptation of one or more machine learning models is provided. The device includes at least one memory; and at least one processor (e.g., configured in a circuit), the at least one processor coupled to the at least one memory and configured to: obtain features extracted by a machine learning model from a first image during inference; use the machine learning model to determine at least one of a plurality of keypoint estimates in the first image or boundary region estimates associated with an object in the first image based on the features during inference; generate one or more pseudo labels based on at least one of the plurality of keypoint estimates or the boundary region estimates; determine at least one self-supervised loss based on at least one of the plurality of keypoint estimates or the boundary region estimates; self-adapt one or more parameters of the machine learning model based on the at least one self-supervised loss; and use the machine learning model with the adapted one or more parameters to generate a segmentation mask for the first image or the second image.

[0007] In another example, a non-transitory computer-readable medium for performing online adaptation of one or more machine learning models is provided. The computer-readable medium has instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to: obtain features extracted by a machine learning model from a first image during inference; use the machine learning model to determine at least one of a plurality of key point estimates in the first image or boundary region estimates associated with an object in the first image based on the features during inference; generate one or more pseudo labels based on at least one of the plurality of key point estimates or the boundary region estimates; determine at least one self-supervised loss based on at least one of the plurality of key point estimates or the boundary region estimates; adapt one or more parameters of the machine learning model based on the at least one self-supervised loss; and use the machine learning model with the adapted one or more parameters to generate a segmentation mask for the first image or the second image.

[0008] In another example, a device for performing online adaptation of one or more machine learning models is provided. The device includes: a component for obtaining a feature extracted by a machine learning model from a first image during inference; a component for determining, by the machine learning model, at least one of a plurality of key point estimates in the first image or a boundary region estimate associated with an object in the first image based on the feature during inference; a component for generating one or more pseudo labels based on at least one of the plurality of key point estimates or the boundary region estimate; a component for determining at least one self-supervised loss based on at least one of the plurality of key point estimates or the boundary region estimate; a component for self-adapting one or more parameters of the machine learning model based on the at least one self-supervised loss; and a component for generating a segmentation mask for the first image or the second image using the machine learning model with the adapted one or more parameters.

[0009] In some aspects, one or more of the devices described herein are, are part of, and / or include a vehicle or a computing device or component of a vehicle (e.g., an autonomous vehicle), a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a camera, a wearable device (e.g., a network-connected watch, etc.), a personal computer, a laptop computer, a server computer, or other device. In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the devices described herein may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0011] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0013] Figure 1A and Figure 1B is a block diagram illustrating a vehicle suitable for implementing various aspects according to the present disclosure.

[0014] Figure 1C is a block diagram illustrating components of a vehicle suitable for implementing various aspects according to the present disclosure;

[0015] Figure 1D illustrates example implementations of a system on a chip (SOC) according to some examples;

[0016] Figure 2A is a component block diagram illustrating components of an example vehicle management system in accordance with various aspects;

[0017] Figure 2B is a component block diagram illustrating components of another example vehicle management system according to various aspects;

[0018] FIG. 3A to FIG. 4 is a diagram illustrating an example of a neural network according to some examples;

[0019] Figure 5 is a diagram illustrating an example of a segmentation machine learning system according to aspects of the present disclosure;

[0020] Figure 6 is a diagram illustrating an example of a segmentation machine learning system that may perform online (or test-time or inference-based) adaptation according to aspects of the present disclosure;

[0021] Figure 7 is a flow chart illustrating an example of a process for processing one or more images according to aspects of the present disclosure; and

[0022] Figure 8 An example computing device architecture is illustrated that illustrates an example computing device that can implement the various techniques described herein. DETAILED DESCRIPTION

[0023] Certain aspects of the present disclosure are provided below. Some of these aspects may be applied independently, and some of them may be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects may be practiced without these specific details. Each of the drawings and descriptions is not intended to be restrictive.

[0024] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0025] In some cases, such as before performing one or more operations on an image or video (e.g., autonomous or semi-autonomous driving operations, applying effects to an image, etc.), an image or video frame may be processed to identify one or more objects present within the image or video frame. For example, adding a virtual background to a video conference may include identifying an object in the foreground (e.g., one or more people, buildings, vehicles, etc.) and modifying all portions of the video frame except for pixels belonging to the object. In some cases, objects in an image may be identified by using one or more neural networks or other machine learning (ML) models to assign segmentation categories or labels (e.g., human categories, car categories, background categories, etc.) to each pixel in the frame, and then grouping adjacent pixels that share a segmentation category to form an object of the segmentation category (e.g., person, car, background, etc.). This technique may be referred to as pixel-by-pixel segmentation. The categories or labels assigned to each pixel may be referred to as pixel-by-pixel categories or labels, which may be collectively referred to as segmentation masks (also referred to herein as segmentation maps).

[0026] An example of a type of segmentation is semantic segmentation. Semantic segmentation treats multiple objects of the same class as a single entity or instance (e.g., all detected people within an image are considered to be of the "person" class). Another type of segmentation is instance segmentation. Instance segmentation treats multiple objects of the same class as different entities or instances (e.g., a first person detected in an image is a second instance of the "person" class, and a second person detected in the image is a second instance of the "person" class).

[0027] In some cases, pixel-by-pixel segmentation may involve inputting an image into an ML model such as a convolutional neural network (CNN). The ML model may process the input image and output a segmentation mask or segmentation map of the image. As previously described, the segmentation mask may include segmentation class information (e.g., pixel-by-pixel class or label) for each pixel in the frame. In some cases, the segmentation mask may be configured to maintain information only for pixels corresponding to one or more classes (e.g., for pixels classified as people), thereby isolating selected classified pixels from other classified pixels (e.g., thereby isolating human pixels from background pixels).

[0028] The segmentation network may perform poorly in some cases, such as when the test image exhibits a large domain shift relative to the training dataset used to train the segmentation network (e.g., indoors compared to outdoors, or daytime compared to nighttime). For example, for human instance segmentation, such a domain shift may be due to severe occlusion of the human subject in the test image. However, despite the segmentation network's poor segmentation performance in such cases, the segmentation network tends to estimate the keypoints of the object well.

[0029] ML models (such as neural networks) may be trained offline using training techniques such as supervised learning, unsupervised learning, semi-supervised learning, and / or other types of training techniques. During such offline training, parameters (e.g., weights, biases, etc.) of the ML model are tuned based on one or more loss functions. In some cases, the parameters of the ML model may be adapted or fine-tuned during inference or test time (which is referred to as online adaptation or fine-tuning, test time adaptation or fine-tuning, or inference-based adaptation or fine-tuning).

[0030] Many online adaptive methods for ML models trained to generate segmentation masks use pixel pseudo-labels. For example, a mask head of a neural network trained to generate a segmentation mask can generate a segmentation mask, wherein each pixel of the segmentation mask is labeled with a category determined by the segmentation mask. The labeled pixels can be used as pseudo-labels. The pseudo-labels can then be used to perform online adaptation of the parameters of the ML model. However, such pixel pseudo-labels are inherently unreliable. In addition, ML models (e.g., deep neural networks (DNNs)) can misclassify pixels with high confidence. In contrast, higher-level attributes such as boundary regions (e.g., bounding boxes, which can be indicated as Bboxes) and key points are more reliable and can therefore provide better candidates for pseudo-labels. In addition, many segmentation ML models have a common feature extractor and may include multiple parallel heads (e.g., key point heads and object detection heads), in which case the generation of boundary regions (e.g., bounding boxes) and key points will be available with negligible overhead.

[0031] Described herein are systems, apparatus, electronic devices, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") that provide solutions for performing online adaptation of segmentation machine learning systems (e.g., segmentation neural networks). Online adaptation may also be referred to herein as test-time adaptation (TTA) or inference-based adaptation or fine-tuning of machine learning systems (e.g., neural networks). For example, the present systems and techniques may perform online adaptation of an object segmentation ML model (e.g., a human segmentation network) using boundary regions (e.g., an object pose delineated by a boundary region, such as a human pose) and / or keypoints.

[0032] Various aspects of the application will be described with reference to the accompanying drawings.

[0033] The systems and techniques described herein may be implemented by any type of system or device. An illustrative example of a system that can be used to implement the systems and techniques described herein is a vehicle (e.g., an autonomous or semi-autonomous vehicle) or a system or component of a vehicle (e.g., an ADAS or other system or component). Other examples of systems or devices that can be used to perform the techniques described herein may include mobile devices (e.g., mobile phones or so-called "smart phones" or other mobile devices), XR devices (e.g., VR devices, AR devices, MR devices, etc.), cameras, wearable devices (e.g., network-connected watches, etc.), and / or other types of systems or devices.

[0034] Figure 1A and Figure 1B is a diagram illustrating an example vehicle 100 in which the systems and techniques described herein may be implemented. Figure 1A and Figure 1B , the vehicle 100 may include a control unit 140 and a plurality of sensors 102-138, including a satellite geo-positioning system receiver (e.g., sensor) 108, occupancy sensors 112, 116, 118, 126, 128, tire pressure sensors 114, 120, cameras 122, 136, microphones 124, 134, impact sensors 130, radar 132, and lidar 138. The plurality of sensors 102-138 disposed in or on the vehicle may be used for various purposes, such as autonomous and semi-autonomous navigation and control, collision avoidance, location determination, etc., and provide sensor data about objects and people in or on the vehicle 100. The sensors 102-138 may include one or more sensors from a wide variety of sensors capable of detecting various information useful for navigation and collision avoidance. Each of the sensors 102-138 may communicate with the control unit 140 and each other in wired or wireless communication. Specifically, the sensors may include one or more cameras 122, 136 or other optical or photoelectric sensors. The sensors may also include other types of object detection and ranging sensors, such as radar 132, lidar 138, IR sensors, and ultrasonic sensors. The sensors may also include tire pressure sensors 114, 120, humidity sensors, temperature sensors, satellite geolocation sensors 108, accelerometers, vibration sensors, gyroscopes, gravity meters, impact sensors 130, force gauges, pressure gauges, strain sensors, fluid sensors, chemical sensors, gas content analyzers, pH sensors, radiation sensors, Geiger counters, neutron detectors, biomaterial sensors, microphones 124, 134, occupancy sensors 112, 116, 118, 126, 128, proximity sensors, and other sensors.

[0035] The vehicle control unit 140 may be configured with processor-executable instructions to perform various aspects using information received from various sensors, particularly the cameras 122, 136, the radar 132, and the lidar 138. In some aspects, the control unit 140 may supplement the processing of camera images using distance and relative positioning information (e.g., relative azimuth) available from the radar 132 and / or lidar 138 sensors. The control unit 140 may be further configured to use information about other vehicles determined using various aspects to control the steering, braking, and speed of the vehicle 100 when operating in an autonomous or semi-autonomous mode.

[0036] Figure 1C is a component block diagram of a system 150 illustrating components and supporting systems suitable for implementing various aspects. Figure 1A , Figure 1B and Figure 1C , the vehicle 100 may include a control unit 140, which may include various circuits and devices for controlling the operation of the vehicle 100. Figure 1C In the example illustrated in , the control unit 140 includes a processor 164, a memory 166, an input module 168, an output module 170, and a radio module 172. The control unit 140 may be coupled to and configured to control a driving control component 154, a navigation component 156, and one or more sensors 158 of the vehicle 100.

[0037] The control unit 140 may include a processor 164 that may be configured with processor-executable instructions to control maneuvering, navigation, and / or other operations, including various aspects of operation, of the vehicle 100. The processor 164 may be coupled to a memory 166. The control unit 140 may include an input module 168, an output module 170, and a radio module 172.

[0038] The radio module 172 may be configured for wireless communication. The radio module 172 may exchange signals 182 (e.g., command signals for controlling maneuvers, signals from navigational aids, etc.) with the network node 180, and may provide the signals 182 to the processor 164 and / or the navigation component 156. In some aspects, the radio module 172 may enable the vehicle 100 to communicate with the wireless communication device 190 via the wireless communication link 92. The wireless communication link 92 may be a bidirectional or unidirectional communication link, and may use one or more communication protocols.

[0039] Input module 168 may receive sensor data from one or more vehicle sensors 158 and electronic signals from other components, including driving control component 154 and navigation component 156. Output module 170 may be used to communicate with or activate various components of vehicle 100, including driving control component 154, navigation component 156, and sensor 158.

[0040] The control unit 140 may be coupled to a driving control assembly 154 to control physical elements of the vehicle 100 related to maneuvering and navigation of the vehicle, such as the engine, motor, throttle, steering elements, other control elements, braking or deceleration elements, etc. The driving control assembly 154 may also include components for controlling other devices of the vehicle, including environmental controls (e.g., air conditioning and heating), exterior and / or interior lighting, interior and / or exterior information displays (which may include display screens or other devices for displaying information), safety devices (e.g., tactile devices, audible alarms, etc.), and other similar devices.

[0041] The control unit 140 may be coupled to the navigation component 156 and may receive data from the navigation component 156. The control unit 140 may be configured to use such data to determine the current location and orientation of the vehicle 100 and the appropriate route towards the destination. In various aspects, the navigation component 156 may include or be coupled to a GNSS receiver system (e.g., one or more global positioning system (GPS) receivers) that enables the vehicle 100 to use a global navigation satellite system (GNSS) signal to determine the current location of the vehicle. Alternatively or in addition, the navigation component 156 may include a radio navigation receiver for receiving navigation beacons or other signals from a radio node (such as a Wi-Fi access point, a cellular network site, a radio station, a remote computing device, other vehicles, etc.). By controlling the driving control component 154, the processor 164 can control the navigation and maneuvering of the vehicle 100. The processor 164 and / or the navigation component 156 may be configured to communicate with a server 184 on a network 186 (e.g., the Internet) using wireless signals 182 exchanged via a network node 180 over a cellular data network to receive commands for controlling maneuvers, receiving data useful in navigation, providing real-time location reports, and evaluating other data.

[0042] Control unit 140 may be coupled to one or more sensors 158. Sensors 158 may include sensors 102-138 as described, and may be configured to provide various data to processor 164.

[0043] Although the control unit 140 is described as including separate components, in some aspects, some or all of the components (e.g., the processor 164, the memory 166, the input module 168, the output module 170, and the radio module 172) may be integrated into a single device or module, such as a system-on-chip (SOC) processing device. Such an SOC processing device may be configured for use in a vehicle and configured (such as configured with processor-executable instructions executed in the processor 164) to perform various aspects of operation when installed in the vehicle.

[0044] Figure 1D An example implementation of a system on chip (SOC) 105 is illustrated, which may include a central processing unit (CPU) 110 or a multi-core CPU configured to perform one or more of the functions described herein. In some cases, the SOC 105 may be based on the ARM instruction set. In some cases, the CPU 110 may be similar to the processor 164. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, and other information may be stored in a memory block associated with a neural processing unit (NPU) 125, in a memory block associated with the CPU 110, in a memory block associated with a graphics processing unit (GPU) 115, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 185, and / or may be distributed across multiple blocks. Instructions executed at the CPU 110 may be loaded from a program memory associated with the CPU 110, or may be loaded from a memory block 185.

[0045] The SOC 105 may also include additional processing blocks tailored for specific functions, such as a GPU 115, a DSP 106, a connectivity block 135 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 145 that may, for example, detect and recognize gestures. In one specific implementation, the NPU is implemented in the CPU 110, the DSP 106, and / or the GPU 115. The SOC 105 may also include a sensor processor 155, an image signal processor (ISP) 175, and / or a navigation module 195, which may include a global positioning system. In some cases, the navigation module 195 may be similar to the navigation component 156, and the sensor processor 155 may accept inputs from, for example, one or more sensors 158. In some cases, the connectivity block 135 may be similar to the radio module 172.

[0046] Figure 2AAn example of a method that can be used in a vehicle (such as Figure 1A Examples of vehicle applications, subsystems, computing elements or units within a vehicle management system 200 utilized within a vehicle 100 of FIG. Figures 1A to 2A In some aspects, various vehicle applications, computing elements, or units within the vehicle management system 200 may be implemented within a system of interconnected computing devices (i.e., subsystems) that communicate data and commands to each other. In other aspects, the vehicle management system 200 may be implemented as multiple vehicle applications, such as separate threads, processes, algorithms, or computing elements, executed within a single computing device. However, the use of the term vehicle application when describing various aspects is not intended to imply or require that the corresponding functionality be implemented within a single autonomous (or semi-autonomous) vehicle management system computing device, although this is a potential implementation aspect. Instead, the use of the term vehicle application is intended to include subsystems with independent processors, computing elements (e.g., threads, algorithms, subroutines, etc.) running in one or more computing devices, and combinations of subsystems and computing elements.

[0047] In various aspects, the vehicle applications executed in the vehicle management system 200 may include, but are not limited to, a radar-aware vehicle application 202, a camera-aware vehicle application 204, a positioning engine vehicle application 206, a map fusion and arbitration vehicle application 208, a route vehicle planning application 210, a sensor fusion and road world model (RWM) management vehicle application 212, a motion planning and control vehicle application 214, and a behavior planning and prediction vehicle application 216. The vehicle applications 202-216 are merely examples of some of the vehicle applications in the example configuration of the vehicle management system 200. In other configurations consistent with various aspects, other vehicle applications may be included, such as additional vehicle applications for other perception sensors (e.g., LIDAR perception layer, etc.), additional vehicle applications for planning and / or control, additional vehicle applications for modeling, etc., and / or some of the vehicle applications 202-216 may be excluded from the vehicle management system 200. Each of the vehicle applications 202-216 may exchange data, calculation results, and commands.

[0048] The vehicle management system 200 may receive and process data from sensors (e.g., radar, lidar, camera, inertial measurement unit (IMU), etc.), navigation systems (e.g., GPS receiver, IMU, etc.), vehicle networks (e.g., controller area network (CAN) bus), and databases in memory (e.g., digital map data). The vehicle management system 200 may output vehicle control commands or signals to a drive-by-wire (DBW) system / control unit 220, which is a system, subsystem, or computing device that interfaces directly with the vehicle's steering, throttle, and brake controls. Figure 2A The configurations of the vehicle management system 200 and DBW system / control unit 220 illustrated in FIG. 1 are merely example configurations and other configurations of vehicle management systems and other vehicle components may be used in various aspects. As an example, Figure 2A The configuration of the vehicle management system 200 and DBW system / control unit 220 illustrated in FIG. 2 may be used in a vehicle configured for autonomous or semi-autonomous operation, while different configurations may be used in a non-autonomous vehicle.

[0049] The radar-aware vehicle application 202 may receive data from one or more detection and ranging sensors, such as radar (e.g., 132) and / or lidar (e.g., 138), and process the data to identify and determine the location of other vehicles and objects in the vicinity of the vehicle 100. The radar-aware vehicle application 202 may include the use of neural network processing and artificial intelligence methods to identify objects and vehicles and pass such information to the sensor fusion and RWM management vehicle application 212.

[0050] The camera-aware vehicle application 204 may receive data from one or more cameras, such as the cameras (e.g., 122, 136), and process the data to identify and determine the location of other vehicles and objects in the vicinity of the vehicle 100. The camera-aware vehicle application 204 may include the use of neural network processing and artificial intelligence methods to identify objects and vehicles and pass such information to the sensor fusion and RWM management vehicle application 212.

[0051] The positioning engine vehicle application 206 may receive data from various sensors and process the data to determine the location of the vehicle 100. The various sensors may include, but are not limited to, GPS sensors, IMUs, and / or other sensors connected via a CAN bus. The positioning engine vehicle application 206 may also utilize input from one or more cameras, such as cameras (e.g., 122, 136) and / or any other available sensors, such as radar, lidar, etc.

[0052] The map fusion and arbitration vehicle application 208 can access data within a high-definition (HD) map database, and receive output received from the positioning engine vehicle application 206, and process the data to further determine the location of the vehicle 100 within the map, such as the location within the traffic lane, the location within the street map, etc. The HD map database can be stored in a memory (e.g., memory 166). For example, the map fusion and arbitration vehicle application 208 can convert the latitude and longitude information from the GPS into a location within a ground road map contained in the HD map database. The GPS positioning position includes errors, so the map fusion and arbitration vehicle application 208 can be used to determine the best guess position of the vehicle 100 within the road based on the arbitration between the GPS coordinates and the HD map data. For example, although the GPS coordinates can place the vehicle 100 near the middle of a two-lane road in the HD map, the map fusion and arbitration vehicle application 208 can determine that the vehicle 100 is most likely aligned with the driving lane consistent with the direction of travel based on the direction of travel. The map fusion and arbitration vehicle application 208 can pass the map-based location information to the sensor fusion and RWM management vehicle application 212.

[0053] The route planning vehicle application 210 can utilize the HD map and input from the operator or dispatcher to plan a route that the vehicle 100 will follow to a specific destination. The route planning vehicle application 210 can pass the map-based location information to the sensor fusion and RWM management vehicle application 212. However, other vehicle applications (such as the sensor fusion and RWM management vehicle application 212, etc.) do not require the use of a priori maps. For example, other stacks can operate and / or control vehicles based solely on sensory data without a provided map, building concepts of lanes, boundaries, and local maps as the sensory data is received.

[0054] The sensor fusion and RWM management vehicle application 212 may receive data and output generated by one or more of the radar-aware vehicle application 202, the camera-aware vehicle application 204, the map fusion and arbitration vehicle application 208, and the route planning vehicle application 210, and use some or all of such inputs to estimate or refine the position and state of the vehicle 100 relative to the road, other vehicles on the road, and other objects in the vicinity of the vehicle 100. For example, the sensor fusion and RWM management vehicle application 212 may combine image data from the camera-aware vehicle application 204 with arbitrated map position information from the map fusion and arbitration vehicle application 208 to refine the determined positioning of the vehicle within the traffic lane. As another example, the sensor fusion and RWM management vehicle application 212 may combine object recognition and image data from the camera-aware vehicle application 204 with object detection and ranging data from the radar-aware vehicle application 202 to determine and refine the relative positioning of other vehicles and objects in the vicinity of the vehicle. As another example, the sensor fusion and RWM management vehicle application 212 may receive information about the location and direction of travel of other vehicles from vehicle-to-vehicle (V2V) communications (such as via a CAN bus) and combine this information with information from the radar awareness vehicle application 202 and the camera awareness vehicle application 204 to refine the location and motion of the other vehicles. The sensor fusion and RWM management vehicle application 212 may output the refined location and state information of the vehicle 100 and the refined location and state information of other vehicles and objects in the vicinity of the vehicle to the motion planning and control vehicle application 214 and / or the behavior planning and prediction vehicle application 216.

[0055] As a further example, the sensor fusion and RWM management vehicle application 212 can use dynamic traffic control instructions to guide the vehicle 100 to change speed, lane, direction of travel, or (other) navigation elements, and combine this information with other received information to determine the refined position and state information. The sensor fusion and RWM management vehicle application 212 can output the refined position and state information of the vehicle 100 and the refined position and state information of other vehicles and objects in the vicinity of the vehicle 100 to the motion planning and control vehicle application 214, the behavior planning and prediction vehicle application 216 and / or devices remote from the vehicle 100, such as data servers, other vehicles, etc. via wireless communication (such as through a C-V2X connection, other wireless connections, etc.).

[0056] As yet a further example, the sensor fusion and RWM management vehicle application 212 may monitor perception data from various sensors (such as perception data from the radar perception vehicle application 202, the camera perception vehicle application 204, other perception vehicle applications, etc.) and / or data from one or more sensors themselves to analyze conditions in the vehicle sensor data. The sensor fusion and RWM management vehicle application 212 may be configured to detect sensor data conditions, such as sensor measurements being at, above, or below a threshold, certain types of sensor measurements occurring, etc., and may output the sensor data as part of the refined position and state information of the vehicle 100 that is provided to the behavior planning and prediction vehicle application 216 and / or to devices (such as data servers, other vehicles, etc.) remote from the vehicle 100 via wireless communications (such as through a C-V2X connection, other wireless connections, etc.).

[0057] The refined location and status information may include vehicle descriptors associated with the vehicle 100 and the vehicle owner and / or operator, such as: vehicle specifications (e.g., size, weight, color, type of onboard sensors, etc.); vehicle location, speed, acceleration, direction of travel, attitude, orientation, destination, fuel / power level, and other status information; vehicle emergency status (e.g., the vehicle is an emergency vehicle or a private individual is in an emergency); vehicle restrictions (e.g., heavy / wide loads, turning restrictions, high occupancy vehicles (HOVs); and vehicle status information. OV) authorization, etc.); vehicle capabilities (e.g., all-wheel drive, four-wheel drive, snow tires, chains, supported connection types, onboard sensor operating status, onboard sensor resolution level, etc.); equipment issues (e.g., low tire pressure, weak brakes, sensor failure, etc.); owner / operator travel preferences (e.g., preferred lane, road, route and / or destination, preference to avoid tolls or highways, preference for fastest route, etc.); permission to provide sensor data to a data proxy server (e.g., 184); and / or owner / operator identification information.

[0058] The behavior planning and prediction vehicle application 216 of the autonomous vehicle system 200 can use the refined position and state information of the vehicle 100 and the position and state information of other vehicles and objects output from the sensor fusion and RWM management vehicle application 212 to predict the future behavior of other vehicles and / or objects. For example, the behavior planning and prediction vehicle application 216 can use such information to predict the future relative position of other vehicles based on the own vehicle position and speed and the position and speed of other vehicles in the vicinity of the vehicle. Such predictions can take into account information from HD maps and route planning to anticipate changes in relative vehicle positions as the host vehicle and other vehicles travel along the road. The behavior planning and prediction vehicle application 216 can output other vehicle and object behavior and position predictions to the motion planning and control vehicle application 214.

[0059] In addition, the behavior planning and prediction vehicle application 216 can use the object behavior in conjunction with the position prediction to plan and generate control signals for controlling the movement of the vehicle 100. For example, based on the route planning information, the refined position in the road information, and the relative position and movement of other vehicles, the behavior planning and prediction vehicle application 216 can determine that the vehicle 100 needs to change lanes and accelerate, such as to maintain or achieve a minimum spacing with other vehicles, and / or to prepare for a turn or exit. Therefore, the behavior planning and prediction vehicle application 216 can calculate or otherwise determine the steering angle of the wheel and the change to the throttle setting, which will be commanded to the motion planning and control vehicle application 214 and the DBW system / control unit 220 along with various such parameters necessary to achieve such lane changes and acceleration. One such parameter can be the calculated steering wheel command angle.

[0060] The motion planning and control vehicle application 214 may receive data and information output from the sensor fusion and RWM management vehicle application 212, and other vehicle and object behavior and position predictions from the behavior planning and prediction vehicle application 216, and use the information to plan and generate control signals for controlling the motion of the vehicle 100, and verify that such control signals meet the safety requirements of the vehicle 100. For example, based on the route planning information, the refined position in the road information, and the relative position and motion of other vehicles, the motion planning and control vehicle application 214 may verify and pass various control commands or instructions to the DBW system / control unit 220.

[0061] The DBW system / control unit 220 may receive these commands or instructions from the motion planning and control vehicle application 214 and translate such information into mechanical control signals for controlling the wheel angles, brakes, and throttle of the vehicle 100. For example, the DBW system / control unit 220 may respond to a calculated steering wheel command angle by sending a corresponding control signal to a steering wheel controller.

[0062] In various aspects, the vehicle management system 200 may include functionality to perform safety checks or oversight of various commands, plans, or other decisions of various vehicle applications that may affect the safety of the vehicle and its occupants. Such safety checks or oversight functionality may be implemented within a dedicated vehicle application or distributed among various vehicle applications and included as part of the functionality. In some aspects, various safety parameters may be stored in a memory, and the safety check or oversight functionality may compare the determined values ​​(e.g., relative spacing to nearby vehicles, distance to the centerline of the road, etc.) to the corresponding safety parameters and issue a warning or command if the safety parameters are violated or about to be violated. For example, the safety or supervisory functionality in the behavior planning and prediction vehicle application 216 (or in a separate vehicle application) may determine the current or future spacing (e.g., based on the world model refined by the sensor fusion and RWM management vehicle application 212) between another vehicle (as refined by the sensor fusion and RWM management vehicle application 212) and the vehicle 100, compare the spacing to a safe spacing parameter stored in memory, and issue a command to the motion planning and control vehicle application 214 to accelerate, decelerate, or turn if the current or predicted spacing violates the safe spacing parameter. As another example, the safety or supervisory functionality in the motion planning and control vehicle application 214 (or in a separate vehicle application) may compare the determined or commanded steering wheel command angle to a safe wheel angle limit or parameter, and issue an override command and / or alarm in response to the commanded angle exceeding the safe wheel angle limit.

[0063] Some safety parameters stored in memory, such as maximum vehicle speed, may be static (e.g., do not change over time). Other safety parameters stored in memory may be dynamic in that they are continuously or periodically determined or updated based on vehicle state information and / or environmental conditions. Non-limiting examples of safety parameters include: maximum safe speed, maximum brake pressure, maximum acceleration, and safe wheel angle limits, all of which may vary depending on road and weather conditions.

[0064] Figure 2B Examples of vehicle applications, subsystems, computing elements, or units within the vehicle management system 250 that may be utilized within the vehicle 100 are illustrated. Figures 1A to 2B In some aspects, the vehicle applications 202, 204, 206, 208, 210, 212, and 216 of the vehicle management system 200 may be similar to those of reference Figure 2A The vehicle management system 250 may be similar to the vehicle management system 200, except that the vehicle management system 250 may pass various data or instructions to the vehicle safety and collision avoidance system 252 instead of the DBW system / control unit 220. For example, Figure 2B The configuration of the vehicle management system 250 and the vehicle safety and collision avoidance system 252 illustrated in FIG. 2 may be used in a non-autonomous vehicle.

[0065] In various aspects, the behavior planning and prediction vehicle application 216 and / or the sensor fusion and RWM management vehicle application 212 may output data to the vehicle safety and collision avoidance system 252. For example, the sensor fusion and RWM management vehicle application 212 may output sensor data as part of the refined position and state information of the vehicle 100 provided to the vehicle safety and collision avoidance system 252. The vehicle safety and collision avoidance system 252 may use the refined position and state information of the vehicle 100 to make safety determinations about the vehicle 100 and / or the occupants of the vehicle 100. As another example, the behavior planning and prediction vehicle application 216 may output behavior models and / or predictions related to the movement of other vehicles to the vehicle safety and collision avoidance system 252. The vehicle safety and collision avoidance system 252 may use the behavior models and / or predictions related to the movement of other vehicles to make safety determinations about the vehicle 100 and / or the occupants of the vehicle 100.

[0066] In various aspects, the vehicle safety and collision avoidance system 252 may include functionality to perform safety checks or oversight on various commands, plans or other decisions of various vehicle applications and human driver actions that may affect the safety of the vehicle and its occupants. In some aspects, various safety parameters may be stored in memory, and the vehicle safety and collision avoidance system 252 may compare the determined values ​​(e.g., relative spacing to nearby vehicles, distance to the centerline of the road, etc.) with the corresponding safety parameters, and issue a warning or command if the safety parameters are violated or will be violated. For example, the vehicle safety and collision avoidance system 252 may determine the current or future spacing (e.g., based on a world model refined by the sensor fusion and RWM management vehicle application 212) between another vehicle (as refined by the sensor fusion and RWM management vehicle application 212) and the vehicle, compare the spacing with the safety spacing parameters stored in the memory, and issue a command to the driver to accelerate, decelerate, or turn if the current or predicted spacing violates the safety spacing parameters. As another example, the vehicle safety and collision avoidance system 252 may compare the human driver's steering wheel angle changes to a safe wheel angle limit or parameter and issue an override command and / or alert in response to the steering wheel angle exceeding the safe wheel angle limit.

[0067] As indicated above, segmentation may be performed on image data (e.g., an independent image or a video frame of a video) to generate a segmentation mask for the image data. In some cases, segmentation may be performed using one or more machine learning techniques, such as using one or more neural networks. A neural network is an example of a machine learning system, and the neural network may include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processed by the hidden nodes of one or more hidden layers, and output is generated by the output nodes of the output layer. A deep learning network typically includes a plurality of hidden layers. Each layer of the neural network may include a feature map or an activation map, which may include an artificial neuron (or node). The feature map may include filters, kernels, etc. A node may include one or more weights for indicating the importance of a node in one or more layers in the layer. In some cases, a deep learning network may have a series of many hidden layers, wherein the early layers are used to determine simple and low-level characteristics of the input, and the later layers establish a hierarchical structure of more complex and abstract characteristics.

[0068] Deep learning architectures can learn hierarchies of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0069] Deep learning architectures perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to recognize cars, trucks, and airplanes.

[0070] Neural networks can be designed to have a variety of connection patterns. In a feedforward network, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in consecutive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The loop architecture can help identify patterns that span more than one block of input data that is sequentially delivered to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. A network with many feedback connections can be helpful when the recognition of high-level concepts can assist in discerning specific low-level features of the input. The connections between the layers of a neural network can be fully connected, or locally connected. The following is a description of the loop architecture. FIG. 3A to FIG. 4 Describes various examples of neural network architectures.

[0071] Neural networks can be designed to have a variety of connection patterns. In a feedforward network, information is passed from a lower layer to a higher layer, where each neuron in a given layer communicates with a neuron in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The loop architecture can help identify patterns that span more than one input data block in the input data blocks that are sequentially delivered to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of a high-level concept can assist in discerning specific low-level features of the input, a network with many feedback connections may be helpful.

[0072] The connections between neural network layers can be fully connected or partially connected. Figure 3A An example of a fully connected neural network 302 is illustrated. In the fully connected neural network 302, a neuron in a first layer may communicate its output to each neuron in a second layer, such that each neuron in the second layer will receive input from each neuron in the first layer. Figure 3B An example of a locally connected neural network 304 is illustrated. In the locally connected neural network 304, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 304 may be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strengths may have different values ​​(e.g., 310, 312, 314, and 316). The locally connected connectivity patterns may produce spatially different receptive fields in higher layers because higher layer neurons in a given region may receive inputs that are tuned through training to characteristics of a limited portion of the total input to the network.

[0073] An example of a locally connected neural network is a convolutional neural network. Figure 3C An example of a convolutional neural network 306 is illustrated. The convolutional neural network 306 can be configured such that the connection strengths associated with the inputs of each neuron in the second layer are shared (e.g., 308). Convolutional neural networks may be well suited for problems where the spatial location of the inputs is meaningful. According to aspects of the present disclosure, the convolutional neural network 306 can be used to perform one or more aspects of video compression and / or decompression.

[0074] One type of convolutional neural network is a deep convolutional network (DCN). Figure 3D A detailed example of a DCN 300 is illustrated, which is designed to recognize visual features from an image 326 input from an image capture device 330 (such as a vehicle-mounted camera). The DCN 300 of the current example can be trained to identify traffic signs and numbers set on traffic signs. Of course, the DCN 300 can be trained for other tasks, such as identifying lane markings or identifying traffic lights.

[0075] The DCN 300 may be trained using supervised learning. During training, an image (such as an image 326 of a speed limit sign) may be presented to the DCN 300, and then a forward pass may be calculated to produce an output 322. The DCN 300 may include a feature extraction section and a classification section. Upon receiving the image 326, a convolution layer 332 may apply a convolution kernel (not shown) to the image 326 to generate a first set of feature maps 318. As an example, the convolution kernel of the convolution layer 332 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 318, four different convolution kernels are applied to the image 326 at the convolution layer 332. A convolution kernel may also be referred to as a filter or a convolution filter.

[0076] The first set of feature maps 318 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 320. The max pooling layer reduces the size of the first set of feature maps 318. That is, the size of the second set of feature maps 320 (such as 14×14) is smaller than the size of the first set of feature maps 318 (such as 28×28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 320 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more sets of subsequent feature maps (not shown).

[0077] exist Figure 3D In the example of , the second set of feature maps 320 is convolved to generate a first feature vector 324. In addition, the first feature vector 324 is further convolved to generate a second feature vector 328. Each feature of the second feature vector 328 may include a number corresponding to a possible feature of the image 326, such as "sign", "60", and "100". A softmax function (not shown) may convert the numbers in the second feature vector 328 into probabilities. Therefore, the output 322 of the DCN 300 is the probability that the image 326 includes one or more features.

[0078] In this example, the probability of "logo" and "60" in output 322 is higher than the probability of other numbers (such as "30", "40", "50", "70", "80", "90", and "100") in output 322. Before training, output 322 generated by DCN 300 may be incorrect. Therefore, the error between output 322 and the target output can be calculated. The target output is the true value of image 326 (e.g., "logo" and "60"). The weights of DCN 300 can then be adjusted so that the output 322 of DCN 300 is closer to the target output.

[0079] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease when the weights are adjusted. At the top layer, the gradient may correspond directly to the value of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradients of the higher layers. The weights may then be adjusted to reduce the error. This way of adjusting weights may be referred to as "backward propagation" because it involves a "backward pass" through the neural network.

[0080] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient is close to the true error gradient. This approximation may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, new images may be presented to the DCN, and a forward pass through the network may produce an output 322 that may be considered an inference or prediction of the DCN.

[0081] Deep belief network (DBN) is a probabilistic model including multiple layers of hidden nodes. DBN can be used to extract hierarchical representations of training data sets. DBN can be obtained by stacking the layers of restricted Boltzmann machine (RBM). RBM is a type of artificial neural network that can learn probability distributions through a set of inputs. Because RBM can learn probability distributions without information about the categories to which each input should be classified, RBM is generally used for unsupervised learning. Using an unsupervised and supervised hybrid paradigm, the bottom RBM of DBN can be trained in an unsupervised manner and can be used as a feature extractor, while the top RBM can be trained in a supervised manner (on the joint distribution of the input and target class from the previous layer) and can be used as a classifier.

[0082] A deep convolutional network (DCN) is a network that is a convolutional network configured with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the weights of the network by using a gradient descent method.

[0083] The DCN may be a feed-forward network. In addition, as described above, the connections from a neuron in the first layer of the DCN to a set of neurons in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of the DCN can be used for fast processing. For example, the computational burden of the DCN may be much smaller than that of a similarly sized neural network that includes loops or feedback connections.

[0084] The processing of each layer of the convolutional network can be thought of as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, the convolutional network trained on the input can be thought of as three-dimensional, with two spatial dimensions along the axis of the image and the third dimension capturing color information. The output of the convolutional connection can be thought of as forming a feature map in a subsequent layer, where each element in the feature map (e.g., feature map 320) receives input from a range of neurons in the previous layer (e.g., feature map 318) and from each of the multiple channels. The values ​​in the feature map can be further processed with nonlinearities (such as rectification, max(0,x)). The values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction.

[0085] Figure 4 4 is a block diagram illustrating an example of a deep convolutional network 450. The deep convolutional network 450 may include multiple different types of layers based on connectivity and weight sharing. Figure 4 As shown, the deep convolution network 450 includes convolution blocks 454A and 454B. Each of the convolution blocks 454A and 454B may be configured with a convolution layer (CONV) 456, a normalization layer (LNorm) 458, and a maximum pooling layer (MAX POOL) 460.

[0086] The convolution layer 456 may include one or more convolution filters that may be applied to the input data 452 to generate a feature map. Although only two convolution blocks 454A, 454B are shown, the present disclosure is not limited thereto, and any number of convolution blocks (e.g., convolution blocks 454A, 454B) may be included in the deep convolution network 450 according to design preferences. The normalization layer 458 may normalize the output of the convolution filter. For example, the normalization layer 458 may provide whitening or lateral suppression. The maximum pooling layer 460 may provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0087] For example, the parallel filter bank of the deep convolutional network may be loaded onto the CPU 110 or GPU 115 of the SOC 105 to achieve high performance and low power consumption. In an alternative aspect, the parallel filter bank may be loaded onto the DSP 106 or ISP 175 of the SOC 105. In addition, the deep convolutional network 450 may access other processing blocks that may be present on the SOC 105, such as the sensor processor 155 and the navigation module 195 dedicated to sensors and navigation, respectively.

[0088] The deep convolutional network 450 may also include one or more fully connected layers, such as layer 462A (labeled as "FC1") and layer 462B (labeled as "FC2"). The deep convolutional network 450 may also include a logistic regression (LR) layer 464. Between each layer 456, 458, 460, 462A, 462B, 464 of the deep convolutional network 450 are weights (not shown) to be updated. The output of each of these layers (e.g., 456, 458, 460, 462A, 462B, 464) may serve as an input to a subsequent layer in these layers (e.g., 456, 458, 460, 462A, 462B, 464) of the deep convolutional network 450 to learn a hierarchical feature representation from the input data 452 (e.g., image, audio, video, sensor data, and / or other input data) assigned to the first convolutional block 454A in the convolutional blocks. The output of the deep convolutional network 450 is a classification score 466 for the input data 452. The classification score 466 may be a set of probabilities, where each probability is a probability that the input data includes a feature from the set of features.

[0089] As described above, ML models or systems (such as neural networks) are trained offline using training techniques (such as supervised learning) to tune the parameters (e.g., weights, biases, etc.) of the ML model. In some cases, the parameters of the ML model may be adapted or fine-tuned during inference or test time by performing online adaptation (or test time adaptation (TTA) or inference-based adaptation). Many segmentation ML models (e.g., segmentation neural networks, such as deep neural networks) trained to generate segmentation masks utilize pixel pseudo-labels for online adaptation. However, such pixel pseudo-labels are inherently unreliable, and the segmentation ML model may misclassify pixels with high confidence.

[0090] Other attributes that may be provided by a segmentation ML model, such as bounding regions (e.g., a bounding box that may delineate or define the location and / or pose of an object in an image) and keypoints, may be more reliable and may provide better candidates for pseudo-labels that may be used in online adaptation of the ML model. Furthermore, because segmentation ML models typically have a common feature extractor and multiple parallel heads (e.g., a keypoint head and an object detection head), the generation of bounding regions (e.g., bounding boxes) and keypoints may be performed without significant overhead.

[0091] As previously mentioned, described herein are systems and techniques that provide a solution for online adaptation of a segmentation machine learning system (e.g., a segmentation neural network) using boundary regions (or object poses depicted by boundary regions, such as human poses) and / or keypoints.

[0092] Figure 5is a diagram illustrating an example of a segmentation machine learning system 500. The segmentation machine learning system 500 may include a deep neural network that includes various neural network layers. As shown, the segmentation machine learning system 500 includes a feature extraction engine 504, a keypoint detection engine 506, an object detection engine 508, and a segmentation mask engine 510. The feature extraction engine 504, the keypoint detection engine 506, the object detection engine 508, and the segmentation mask engine 510 may include separate machine learning models or systems (e.g., different neural network models with different sets of layers), may be parts of a single machine learning model or system (e.g., parts of a single neural network model), or some of the components themselves may be combined into a single entity.

[0093] The feature extraction engine 504 may obtain an input image 502. For example, the feature extraction engine 504 may receive the input image 502 from a camera (e.g., camera 122, camera 136, etc.) of a device including the system 500 or another device, may retrieve the input image 502 from a memory of the device including the system 500 or another device or system, or otherwise obtain the input image 502. The input image 502 may be an independent image (e.g., a photograph) or may be a video frame of a video including a plurality of video frames. The feature extraction engine 504 is configured to process the input image 502 to extract or determine features from the input image 502.

[0094] In some aspects, the feature extraction engine 504 may include an ML model, such as a neural network. For example, the feature extraction engine 504 may include various hidden layers (e.g., convolutional layers, normalization layers, etc.) that generate feature maps based on inputs from earlier layers (e.g., input image 502, feature maps output from earlier hidden layers, etc.). The feature maps (or more generally "features") represent the input image 502 at different abstraction levels. Illustrative examples of neural networks that can be used to implement the feature extraction engine 504 include Figure 3A Fully connected neural network 302, Figure 3B Locally connected neural network 304, Figure 3C Convolutional Neural Network 306, Figure 3D Deep Convolutional Network (DCN) 300 and other types of ML models.

[0095] The feature extraction engine 504 may then output the extracted features (e.g., features or one or more feature maps from the last (or other) layer of the feature extraction engine 504). According to various aspects, the feature extraction engine 504 may output the extracted features to a keypoint detection engine 506, an object detection engine 508, and a segmentation mask engine 510. In some cases, the keypoint detection engine 506 may be referred to as a keypoint head, the object detection engine 508 may be referred to as a detection head, and / or the segmentation mask engine 510 may be referred to as a segmentation head.

[0096] The keypoint detection engine 506 is configured to process features from the feature extraction engine 504 to generate keypoint estimates for one or more objects in the input image 502 (e.g., as depicted in image 512). Figure 5 As shown, the input image 502 includes two people. The key point detection engine 506 may generate key point estimates for a specific number of key points for each of the two people. In some cases, the key point detection engine 506 may generate key point estimates for, for example, 17 key points for each person, including one key point for each eye (two key points for eyes in total), one key point for each ear (two key points for ears in total), one key point for the nose, one key point for each wrist (two key points for wrists in total), one key point for each elbow (two key points for elbows in total), one key point for each shoulder (two key points for shoulders in total), one key point for each hip (two key points for hips in total), one key point for each knee (two key points for knees in total), and one key point for each ankle (two key points for ankles in total). In other cases, the key point detection engine 506 may generate key point estimates for additional or fewer key points.

[0097] In some aspects, the key point estimate may include coordinates identifying the location of each key point. In some cases, the key point estimate may include one or more heat maps for the key points, such as one heat map per key point (e.g., 17 heat maps in the example of 17 key points from above). In some examples, a bounding region (e.g., a bounding box or other region) around a person may be determined or identified, and a heat map may be generated from the bounding region. For example, the bounding region may be divided into a grid including a plurality of cells that make up the heat map. In some aspects, the bounding region (e.g., a bounding box) may be output from the object detection engine 508. In other aspects, the key point detection engine 506 and the object detection engine 508 may be combined, in which case the key point detection engine 506 may generate the bounding region.

[0098] As described above, a respective heat map having a grid of cells may be generated for each of the key points (e.g., a first heat map for a first key point, a second heat map for a second key point, etc.). For a heat map associated with a particular key point, each cell of the heat map grid may include a probability (or other measure of likelihood) that the cell includes the key point associated with the heat map (e.g., how likely it is that a particular cell includes a feature associated with a user's left eye).

[0099] The object detection engine 508 (and / or the keypoint detection engine 506, as discussed above) is configured to process the features from the feature extraction engine 504 to detect one or more objects in the input image 502, such as two persons in the input image. Figure 5 As shown in image 514 of , object detection engine 508 may output bounding regions (e.g., bounding boxes) representing the location and / or posture of each object (e.g., a bounding box for the first person in input image 502 and a bounding box for the second person in input image 502).

[0100] The segmentation mask engine 510 is configured to process the features from the feature extraction engine 504 to generate a segmentation mask for the input image 502. For example, Figure 5 As shown in image 516 of , the segmentation mask engine 510 may output a segmentation mask that classifies pixels of the image into different categories (eg, semantic segmentation) and / or different instances (eg, instance segmentation).

[0101] In some aspects, the outputs of keypoint detection engine 506, object detection engine 508, and segmentation mask engine 510 may be used for one or more operations (e.g., by Figure 5 One or more additional engines not shown in the figure). In some cases, in an XR (e.g., AR, VR, etc.) system, one or more segmentation masks can be used to extract one or more people from the background of the scene and insert these people on the virtual scene. In such cases, the key points can be used by a posture engine that models the body posture of each of these people (e.g., to determine the actions of each of these people). Having the key point detection engine 506, the object detection engine 508, and the segmentation mask engine 510 operate based on features from the common backbone feature extraction engine 504 can provide various advantages. An example of an advantage is that a single data set (e.g., images with key point and / or mask annotations) may be small and insufficient to train a backbone, while having a common backbone allows all data sets to be pulled together. Another example of an advantage of having a common backbone is that it can save computation because feature maps are computed once and used by different engines.

[0102] Figure 6is a diagram illustrating an example of a segmentation machine learning system 500 that performs online (or test time or inference-based) adaptation. As shown, the segmentation machine learning system 500 may additionally include a keypoint pseudo-label generation engine 620, a bounding box pseudo-label generation engine 622, a keypoint loss engine 624, and a bounding box loss engine 626.

[0103] In some cases, the segmentation machine learning system 500 may include the keypoint pseudo-label generation engine 620 and the keypoint loss engine 624, and the segmentation machine learning system 500 may not include the bounding box pseudo-label generation engine 622 and the bounding box loss engine 626. In other cases, the segmentation machine learning system 500 may include the bounding box pseudo-label generation engine 622 and the bounding box loss engine 626, and the segmentation machine learning system 500 may not include the keypoint pseudo-label generation engine 620 and the keypoint loss engine 624. In some cases, the keypoint pseudo-label generation engine 620 and / or the bounding box pseudo-label generation engine 622 may include a function (e.g., a neural network, a software program, or other function) configured (e.g., trained, etc.) to generate pseudo-labels, or a separate function.

[0104] The key point pseudo label generation engine 620 may generate pseudo labels (referred to as key point pseudo labels) from the key point estimates output by the key point detection engine 506. In some cases, the key point pseudo label generation engine 620 may utilize single key point thresholding (e.g., accepting key point scores above a score threshold) or object-based key point thresholding (e.g., calculating scores jointly for a set of key points of an object). In some examples, object-based key point thresholding may include person-based key point thresholding, such as using a score jointly calculated for all key points of a person category. In some aspects of single key point thresholding, each cell of the heat map grid may be compared with a score threshold (which may be referred to as a single key point score threshold) (such as a value of 0.7, 0.8, 0.9, or other values). If a cell of the heat map grid is greater than (or in some cases equal to) a single key point score threshold, the cell of the heat map may be used to generate a key point pseudo label. For example, a value of 1 may be added to a cell of the heat map and used as a key point pseudo label. Any grid determined to be less than (or in some cases equal to) may be ignored and not used as a key point pseudo label. In some cases, the keypoint detection engine 506 may predict whether each keypoint is visible or occluded, which may lead to system improvements (eg, generation of more accurate pseudo-labels).

[0105] The bounding box pseudo-label generation engine 622 may generate pseudo-labels (referred to as bounding box pseudo-labels) from bounding box estimates output by the object detection engine 508 (and / or the keypoint detection engine 506). For example, an object detection loss used to train the object detection engine 508 during offline training may compare the network's estimates for each bounding box (e.g., four numbers for the height and width of the top left and bottom right corners of the bounding box in a given image) with the true values ​​(or labels) for those same four numbers, and measure the difference between them (e.g., the L2 norm of the difference between the true value and the estimated corner points). In order to use the estimated bounding boxes as true values ​​during online adaptation, the system 500 may select from the object detection engine 508 the bounding boxes for which the system 500 has high confidence (e.g., the confidence or object score estimated by the object detection engine 508 for each of the detected bounding box estimates is greater than or equal to the bounding box confidence threshold), and the system 500 may pass the coordinates of the corner point estimates for these bounding boxes as true values ​​for online adaptation. Such a solution may inform the system 500 which objects are real and therefore need to have their confidence or object scores increased, and inform the system 500 which objects were detected incorrectly so that the absence of these objects from the pseudo-labels reduces their object scores in the next run.

[0106] The keypoint loss engine 624 may use the keypoint pseudo-labels in the loss function to determine a keypoint-based loss, which may be considered a self-supervised loss. Similarly, the bounding box loss engine 626 may use the bounding box pseudo-labels in the loss function to determine a bounding box-based loss (e.g., a self-supervised loss). The keypoint loss engine 624 and the bounding box loss engine 626 may use the same loss function or may use different loss functions. Illustrative examples of loss functions that may be used by the keypoint loss engine 624 and / or the bounding box loss engine 626 include mean absolute error (MAE) loss (also referred to as L1 loss), mean square error (MSE) loss (also referred to as L2 loss), cross entropy loss, any combination thereof, and / or other loss functions. In some cases, the same loss function used to train the segmentation machine learning system 500 during offline training may be used for online adaptation. In such cases, the original true value (or label) used in the loss function may be replaced with a pseudo-label. In other cases, the loss function used for online adaptation may be different from the loss function used to train the segmentation machine learning system 500 during offline training. In either of these cases, the pseudo-labels may be compared to the output of the keypoint detection engine 506 to determine a keypoint-based loss.

[0107] The keypoint-based loss and / or the bounding box-based loss may then be used to perform back-propagation to adapt or fine-tune previously trained parameters (e.g., weights, biases, etc. tuned during offline training) of the feature extraction engine 504. In some cases, the keypoint-based loss and / or the bounding box-based loss may be used to adapt or fine-tune previously trained parameters of the keypoint detection engine 506, the object detection engine 508, and / or the segmentation mask engine 510.

[0108] Using the updated or fine-tuned parameters (e.g., based on the keypoint-based loss and / or the bounding box-based loss), the feature extraction engine 504 may process the input image 502 again to generate an adaptive feature (e.g., an adaptive feature map) representing the input image 502. The segmentation mask engine 510 may process the adapted features to generate an adapted segmentation mask for the input image 502. The adapted segmentation mask may have a higher quality (e.g., less segmentation error) than a segmentation mask generated by the segmentation mask engine 510 using previous features output by the feature extraction engine 504 before the online adaptation. In some cases, additionally or alternatively, the feature extraction engine 504 may process one or more new input images to generate corresponding feature maps representing the one or more new input images.

[0109] In some aspects, the online adaptation (eg, test-time adaptation) performed by the systems and techniques described herein may use what may be referred to as m={m b ,m m ,m k}, and a single unlabeled test image. These systems and techniques may also allow test-time adaptation of network weights using back-propagation without access to the source dataset. These assumptions may make the test-time adaptation setting very realistic, but at the same time very challenging. In some examples, illustrative steps of the test-time adaptation method are shown in Algorithm 1 below:

[0110]

[0111]

[0112] Algorithm 1 includes multiple rounds (e.g., 3 rounds) of weight adaptation, each round of weight adaptation includes (expressed as The key point estimate is converted to (expressed as The pseudo-label is then determined (expressed as The resulting self-supervised loss is back-propagated to adjust the backbone weights. The system and technique can work on any pseudo-label conversion method. In some examples, the pseudo-label conversion method may include a minimum probability of 0.05 for a keypoint for an object (e.g., a person, a vehicle, etc.) bounding box with a score above a threshold score (e.g., a score above 0.5 or other value). In some cases, the pseudo-label conversion method can generate a pseudo-label by selecting the location with the highest value in a heat map (e.g., a 56×56 heat map). Multi-class cross entropy loss can be used as a keypoint loss The adapted segmentation mask may then be generated by running a mask head (eg, a machine learning system such as a neural network trained to generate segmentation masks) on the adapted feature map.

[0113] In some cases, the systems and techniques described herein can perform a training time regularization method based on a training time generalization method. In some examples, these systems and techniques can use an object (e.g., person, vehicle, etc.) instance segmentation / keypoint estimation network (e.g., m={m b ,m m ,m k}). In some cases, the network may include a masked recursive convolutional neural network (mask RCNN). These systems and techniques may also have access to a labeled source dataset. In some cases, these systems and techniques may not have access to any test images or knowledge about the target domain. In some cases, these systems and techniques may not allow for test-time adaptation of network weights. Such features may allow for a training-time generalization setting that is both realistic and challenging. In some examples, illustrative steps of a training-time generalization method are shown below in Algorithm 2:

[0114]

[0115]

[0116] As shown in Algorithm 2, training a temporal generalization method may include splitting the keypoint head into sub-networks (e.g., ) for feature extraction and regression respectively. The mask head can be modified to accommodate the additional keypoint features. The original loss can then be used to train the training-time generalization model on the labeled source dataset At test time, segmentation masks can be generated by running the training-time generalized mask head on the aggregated feature maps.

[0117] Various heuristics and keypoint head variants can be used. For example, the performance of the test-time adaptive method and the training-time generalization method can depend on the quality of the pseudo-labels and features generated by the keypoint head, respectively. Therefore, in addition to Mask RCNN, three keypoint head variants can be used. Each variant and the associated heuristic are described below. In some cases, the input to all heads can have dimensions N×256×14×14, where N gives the number of object bounding boxes (e.g., bounding boxes associated with people, vehicles, etc.).

[0118] For Mask RCNN, the feature extraction subnetwork of the keypoint head Eight two-dimensional (2D) convolutional layers may be included. In some examples, each 2D convolutional layer may include 512 output channels, a 3×3 kernel size, and a stride of 1. In such examples, there are 512 additional keypoint features. In some cases, the regression subnetwork A 2D transposed convolution layer with 17 output channels (e.g., corresponding to the number of keypoints), a 4×4 kernel, and a stride of 2 may be included, followed by a bilinear upsampler to increase the keypoint heatmap resolution to N×17×56×56.

[0119] The keypoint head of Mask RCNN can estimate the keypoint locations. However, the keypoint head cannot predict whether these keypoints are visible or occluded, which can impair test-time adaptation performance. For example, the location of an occluded person's elbow can be estimated correctly. However, not knowing that the elbow is occluded can cause parts of the object that obscure the elbow to be included in the mask. To address this issue, a variant of Mask RCNN that predicts whether a keypoint is visible or occluded can be used. The first variant can be the same as Mask RCNN, except that the 2D transposed convolutional layer has 51 outputs to allow two additional outputs per position for visible / occluded predictions.

[0120] In some cases, the keypoint estimates from the convolutional head can be of very low quality, severely affecting the test-time adaptation / train-time generalization gain. A second variant of Mask RCNN can use one or more transformers to enhance the keypoint estimates through global attention. For example, the feature extraction subnetwork of the keypoint head can be modified to include a transformer decoder with six layers, eight heads, and 17 queries of width 256, operating on the backbone feature map. The input query is a trainable parameter. The output query is mapped to keypoints and decoded to a heatmap consisting of N×51×14×14 keypoints through a 3-layer multi-layer perceptron (MLP) (after reshaping). In this example, there are 51 additional keypoint features. According to the second variant, the regression subnetwork of the keypoint head A bilinear upsampler may be included to increase the heatmap resolution to 56×56.

[0121] The third variant can avoid the problems associated with the output query of the transformer, which is being decoded directly into a 14×14 heat map without using convolutional layers by using the last transformer layer value projection and its attention weights to form a 15×14×14 feature map corresponding to each keypoint. In such an example, there are (255=17×15) additional keypoint features. According to the third variant, the regression subnetwork of the keypoint head Two group 2D convolution layers and one group 2D transposed convolution layer may be included to further process the keypoint feature map (e.g., independently of each other), followed by a bilinear upsampler to obtain a 56×56 heat map.

[0122] Figure 7 is a flow chart illustrating a process 700 for performing online adaptation of one or more machine learning models in accordance with aspects of the present disclosure.At block 702, the process 700 includes obtaining features extracted by a machine learning model from a first image during inference.

[0123] At block 704, process 700 includes determining, by the machine learning model based on the features during inference, at least one of a plurality of keypoint estimates in the first image or a boundary region estimate associated with an object in the first image. In some aspects, the plurality of keypoint estimates are determined in the first image by a keypoint detection engine of the machine learning model. In some aspects, the boundary region estimate is determined in the first image by an object detection engine of the machine learning model.

[0124] In some aspects, the plurality of keypoint estimates include at least one heat map indicating a probability that a portion of the first image corresponds to a particular keypoint. In some cases, the at least one heat map includes a plurality of heat maps, each heat map in the plurality of heat maps being associated with a respective keypoint in the plurality of keypoints. In some aspects, the plurality of keypoint estimates include coordinates associated with the plurality of keypoints.

[0125] At block 706, process 700 includes generating one or more pseudo-labels based on at least one of a plurality of keypoint estimates or boundary region estimates. In some cases, to generate the one or more pseudo-labels, process 700 may include determining one or more keypoint estimates from the plurality of keypoint estimates that are greater than a single keypoint score threshold. Process 700 may also include selecting the one or more keypoint estimates as the one or more pseudo-labels based on the one or more keypoint estimates being greater than a single keypoint score threshold. In some examples, to generate the one or more pseudo-labels, process 700 may include determining that the plurality of keypoint estimates are greater than a total score threshold. Process 700 may also include selecting the plurality of keypoint estimates as the one or more pseudo-labels based on the plurality of keypoint estimates being greater than a total score threshold.

[0126] At box 708, process 700 includes: determining at least one self-supervised loss based on at least one of the multiple keypoint estimates or boundary region estimates. At box 710, process 700 includes: adapting one or more parameters of the machine learning model based on the at least one self-supervised loss. In some cases, the one or more parameters include weights of the machine learning model. In some aspects, the features are extracted from the first image by a feature extraction engine of the machine learning model. In such aspects, in order to adapt one or more parameters of the machine learning model based on the at least one self-supervised loss, process 700 may include one or more parameters of the adaptive feature extraction engine.

[0127] At block 712, process 700 includes generating a segmentation mask for the first image or the second image (e.g., a next frame from the first image (frame), another frame in a video including the first image (frame), other images, etc.) using the machine learning model with the adapted one or more parameters. In some aspects, the segmentation mask is generated by a segmentation mask engine of the machine learning model.

[0128] In some cases, process 700 may include processing the first image or the second image using a machine learning model having one or more parameters adapted to generate an updated feature representing the first image or to generate a feature representing the second image. In such cases, a segmentation mask may be generated based on processing the updated feature representing the first image or the feature representing the second image.

[0129] In some examples, the processes described herein (e.g., process 700 and / or other processes described herein) may be performed by a computing device or apparatus or a component or system of a computing device or apparatus (e.g., a chipset, one or more processors (e.g., CPU, GPU, NPU, DSP, etc.), an ML system such as a neural network model, etc.). The computing device or apparatus may be a vehicle or a component or system of a vehicle, a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device (such as a virtual reality (VR) device, an augmented reality (AR) device, and / or a mixed reality (MR) device), or other type of computing device. In some cases, the computing device or apparatus may be Figure 1D SOC 105 Figure 8 The computing system 800, Figure 1A vehicle 100 and / or other computing devices or apparatuses.

[0130] Figure 8 is an example of a system (such as Figure 7 Specifically, Figure 8An example of a computing system 800 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 805. The connection 805 may be a physical connection using a bus, or a direct connection into the processor 810, such as in a chipset architecture. The connection 805 may also be a virtual connection, a networked connection, or a logical connection.

[0131] In some aspects, computing system 800 is a distributed system in which the functionality described in this disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some aspects, a component may be a physical or virtual device.

[0132] The example system 800 includes at least one processing unit (CPU or processor) 810 and connections 805 that communicatively couple various system components including system memory 815, such as read-only memory (ROM) 820 and random access memory (RAM) 825, to the processor 810. The computing system 800 may include a cache 812 of high-speed memory directly connected to the processor 810, in close proximity to the processor, or integrated as part of the processor.

[0133] Processor 810 may include any general purpose processor and hardware or software services such as services 832, 834, and 836 stored in storage device 830 configured to control processor 810, as well as special purpose processors where software instructions are incorporated into the actual processor design. Processor 810 may essentially be a completely independent computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0134] To enable user interaction, the computing system 800 includes an input device 845 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The computing system 800 may also include an output device 835 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 800.

[0135] The computing system 800 may include a communication interface 840, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using wired and / or wireless transceivers, including utilizing an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple™ Lightning™ port / plug, an Ethernet port / plug, a fiber optic port / plug, a dedicated wired port / plug, 3G, 4G, 5G and / or other cellular data network wireless signaling, Bluetooth, or other wireless communications. TM Wireless signal transmission, Bluetooth TM Low energy (BLE) wireless signal transmission, IBEACONTM wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 840 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 800 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0136] The storage device 830 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / magnetic stripe, any other magnetic storage medium, flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Card, or a memory card. card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASH EPROM), a cache memory (e.g., a layer 1 (L1) cache, a layer 2 (L2) cache, a layer 3 (L3) cache, a layer 4 (L4) cache, a layer 5 (L5) cache, other (L#) cache), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0137] The storage device 830 may include software services, servers, services, etc., which, when the code defining such software is executed by the processor 810, causes the system to perform functions. In some aspects, hardware services that perform specific functions may include software components stored in a computer-readable medium connected to necessary hardware components (such as the processor 810, connection 805, output device 835, etc.) to perform functions. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or through wired connections. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer readable media may store thereon code and / or machine executable instructions, which may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, category, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0138] Specific details are provided in the above description to provide a thorough understanding of the various aspects and examples provided herein, but those skilled in the art will recognize that the application is not limited thereto. Thus, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including these variations, unless limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or jointly. In addition, without departing from the broader scope of the specification, the various aspects can be utilized in any number of environments and applications beyond those described herein. Therefore, the specification and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be appreciated that, in alternative aspects, each method can be performed in a different order than described.

[0139] For clarity of explanation, in some instances, the present technology may be presented as including a single functional block, which includes a device, device component, step or routine in a method embodied in software or a combination of hardware and software. Additional components other than those components shown in the drawings and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0140] In addition, it will be appreciated by those skilled in the art that the various exemplary logic blocks, modules, circuits, and algorithmic steps described in conjunction with the various aspects disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as resulting in a departure from the scope of the present disclosure.

[0141] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.

[0142] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise available computer executable instructions from a computer readable medium. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memories, USB devices with non-volatile memory, networked storage devices, etc.

[0143] In some aspects, computer readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0144] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof in some cases, depending in part on the specific application, in part on the desired design, in part on the corresponding technology, etc.

[0145] The various illustrative logic blocks, modules, and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may be implemented in any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Examples of form factors include: laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or add-in cards. By way of further example, such functionality may also be implemented on circuit boards in different chips or different processes executed on a single device.

[0146] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0147] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or individually as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium including a program code, which includes instructions for executing one or more of the above-mentioned methods, algorithms, and / or operations when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0148] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein.

[0149] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0150] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0151] The phrases “coupled to” or “communicatively coupled to” refer to any component being physically connected directly or indirectly to another component, and / or any component being in communication with another component directly or indirectly (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0152] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0153] Claim language or other language reciting "at least one processor configured to," "at least one processor configured to," etc., indicates that one processor or multiple processors (in any combination) may perform the associated operations. For example, claim language reciting "at least one processor configured to: X, Y, and Z" means that a single processor may be used to perform operations X, Y, and Z; or that multiple processors are each tasked with a particular subset of operations X, Y, and Z such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language reciting "at least one processor configured to: X, Y, and Z" may mean that any single processor may perform only at least a subset of operations X, Y, and Z.

[0154] Exemplary aspects of the present disclosure include:

[0155] Aspect 1. A processor-implemented method, the method comprising: obtaining features extracted by a machine learning model from a first image during inference; determining, by the machine learning model, at least one of a plurality of key point estimates in the first image or a boundary region estimate associated with an object in the first image based on the features during inference; generating one or more pseudo labels based on the plurality of key point estimates or at least one of the boundary region estimates; determining at least one self-supervised loss based on the plurality of key point estimates or at least one of the boundary region estimates; adapting one or more parameters of the machine learning model based on the at least one self-supervised loss; and generating a segmentation mask for the first image or a second image using the machine learning model with the adapted one or more parameters.

[0156] Aspect 2. The method according to claim 1 further comprises: using the machine learning model having the adapted one or more parameters to process the first image or the second image to generate updated features representing the first image or to generate features representing the second image.

[0157] Aspect 3. The method of claim 2, wherein the segmentation mask is generated based on processing the updated feature representing the first image or the feature representing the second image.

[0158] Aspect 4. A method according to any one of claims 1 to 3, wherein the multiple key point estimates are determined in the first image by a key point detection engine of the machine learning model.

[0159] Aspect 5. A method according to any one of claims 1 to 4, wherein the boundary region estimate is determined in the first image by an object detection engine of the machine learning model.

[0160] Aspect 6. A method according to any one of claims 1 to 5, wherein the features are extracted from the first image by a feature extraction engine of the machine learning model; and adapting the one or more parameters of the machine learning model based on the at least one self-supervised loss includes: adapting one or more parameters of the feature extraction engine.

[0161] Aspect 7. The method according to any one of claims 1 to 6, wherein the segmentation mask is generated by a segmentation mask engine of the machine learning model.

[0162] Aspect 8. The method of any one of claims 1 to 7, wherein the plurality of keypoint estimates comprises at least one heat map indicating a probability that a portion of the first image corresponds to a particular keypoint.

[0163] Aspect 9. The method of claim 8, wherein the at least one heat map comprises a plurality of heat maps, each heat map of the plurality of heat maps being associated with a corresponding key point of the plurality of key points.

[0164] Aspect 10. The method of any one of claims 1 to 7, wherein the plurality of keypoint estimates comprises coordinates associated with a plurality of keypoints.

[0165] Aspect 11. A method according to any one of claims 1 to 10, wherein generating the one or more pseudo-labels comprises: determining one or more key point estimates from the multiple key point estimates that are greater than a single key point score threshold; and selecting the one or more key point estimates as the one or more pseudo-labels based on the one or more key point estimates being greater than the single key point score threshold.

[0166] Aspect 12. The method according to any one of claims 1 to 10, wherein generating the one or more pseudo-labels comprises: determining that the multiple key point estimates are greater than a total score threshold; and selecting the multiple key point estimates as the one or more pseudo-labels based on the multiple key point estimates being greater than the total score threshold.

[0167] Aspect 13. A method according to any one of claims 1 to 12, wherein the one or more parameters include weights of the machine learning model.

[0168] Aspect 14. A device for performing online adaptation of one or more machine learning models, the device comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory and configured to: obtain features extracted by the machine learning model from a first image during inference; use the machine learning model to determine at least one of a plurality of key point estimates in the first image or boundary region estimates associated with an object in the first image based on the features during inference; generate one or more pseudo labels based on at least one of the plurality of key point estimates or the boundary region estimates; determine at least one self-supervised loss based on the plurality of key point estimates or at least one of the boundary region estimates; adapt one or more parameters of the machine learning model based on the at least one self-supervised loss; and use the machine learning model with the adapted one or more parameters to generate a segmentation mask for the first image or a second image.

[0169] Aspect 15. An apparatus according to claim 14, wherein the at least one processor is configured to: process the first image or the second image using the machine learning model having the adapted one or more parameters to generate updated features representing the first image or to generate features representing the second image.

[0170] Aspect 16. The apparatus of claim 15, wherein the at least one processor is configured to generate the segmentation mask based on processing the updated feature representing the first image or the feature representing the second image.

[0171] Aspect 17. An apparatus according to any one of claims 14 to 16, wherein the at least one processor is configured to: use a key point detection engine of the machine learning model to determine the plurality of key point estimates in the first image.

[0172] Aspect 18. An apparatus according to any one of claims 14 to 17, wherein the at least one processor is configured to: use an object detection engine of the machine learning model to determine the boundary region estimate in the first image.

[0173] Aspect 19. An apparatus according to any one of claims 14 to 18, wherein the at least one processor is configured to use a feature extraction engine of the machine learning model to extract the features from the first image; and wherein in order to self-adapt the one or more parameters of the machine learning model based on the at least one self-supervised loss, the at least one processor is configured to adapt one or more parameters of the feature extraction engine.

[0174] Aspect 20. An apparatus according to any one of claims 14 to 19, wherein the at least one processor is configured to: generate the segmentation mask using a segmentation mask engine of the machine learning model.

[0175] Aspect 21. An apparatus according to any one of claims 14 to 20, wherein the plurality of keypoint estimates comprises at least one heat map indicating a probability that a portion of the first image corresponds to a particular keypoint.

[0176] Aspect 22. The apparatus of claim 21, wherein the at least one heat map comprises a plurality of heat maps, each heat map of the plurality of heat maps being associated with a corresponding key point of the plurality of key points.

[0177] Aspect 23. An apparatus according to any one of claims 14 to 20, wherein the plurality of keypoint estimates comprises coordinates associated with a plurality of keypoints.

[0178] Aspect 24. An apparatus according to any one of claims 14 to 23, wherein in order to generate the one or more pseudo-labels, the at least one processor is configured to: determine one or more key point estimates from the multiple key point estimates that are greater than a single key point score threshold; and select the one or more key point estimates as the one or more pseudo-labels based on the one or more key point estimates being greater than the single key point score threshold.

[0179] Aspect 25. An apparatus according to any one of claims 14 to 23, wherein, in order to generate the one or more pseudo-labels, the at least one processor is configured to: determine that the multiple key point estimates are greater than a total score threshold; and select the multiple key point estimates as the one or more pseudo-labels based on the multiple key point estimates being greater than the total score threshold.

[0180] Aspect 26. An apparatus according to any one of claims 14 to 25, wherein the one or more parameters include weights of the machine learning model.

[0181] Aspect 27. A non-transitory computer-readable medium for performing online adaptation of one or more machine learning models, the computer-readable medium having instructions stored thereon, which when executed by at least one processor cause the at least one processor to perform operations according to any one of claims 1 to 13.

[0182] Aspect 28. An apparatus for performing online adaptation of one or more machine learning models, the apparatus comprising one or more components for performing the operations of any one of claims 1 to 13.

Claims

1. A processor-implemented method, the method comprising: obtaining features extracted from the first image by the machine learning model during inference; determining, by the machine learning model based on the features during inference, at least one of a plurality of keypoint estimates in the first image or a boundary region estimate associated with an object in the first image; generating one or more pseudo labels based on at least one of the plurality of keypoint estimates or the boundary region estimates; determining at least one self-supervised loss based on at least one of the plurality of keypoint estimates or the boundary region estimates; self-adapting one or more parameters of the machine learning model based on the at least one self-supervised loss; as well as A segmentation mask is generated for the first image or the second image using the machine learning model with the adapted one or more parameters.

2. The method according to claim 1, further comprising: The first image or the second image is processed using the machine learning model with the adapted one or more parameters to generate updated features representing the first image or to generate features representing the second image. 3 . The method of claim 2 , wherein the segmentation mask is generated based on processing the updated feature representing the first image or the feature representing the second image.

4. The method of claim 1 , wherein the plurality of keypoint estimates are determined in the first image by a keypoint detection engine of the machine learning model.

5. The method of claim 1, wherein the boundary region estimate is determined in the first image by an object detection engine of the machine learning model.

6. The method according to claim 1, wherein: The features are extracted from the first image by a feature extraction engine of the machine learning model; and Adapting the one or more parameters of the machine learning model based on the at least one self-supervised loss includes: adapting one or more parameters of the feature extraction engine.

7. The method of claim 1, wherein the segmentation mask is generated by a segmentation mask engine of the machine learning model.

8. The method of claim 1, wherein the plurality of keypoint estimates comprises at least one heat map indicating a probability that a portion of the first image corresponds to a particular keypoint.

9. The method of claim 8, wherein the at least one heat map comprises a plurality of heat maps, each heat map of the plurality of heat maps being associated with a corresponding key point of the plurality of key points.

10. The method of claim 1, wherein the plurality of keypoint estimates comprises coordinates associated with a plurality of keypoints.

11. The method of claim 1 , wherein generating the one or more pseudo labels comprises: determining one or more keypoint estimates from the plurality of keypoint estimates that are greater than a single keypoint score threshold; as well as The one or more keypoint estimates are selected as the one or more pseudo-labels based on the one or more keypoint estimates being greater than the single keypoint score threshold.

12. The method of claim 1, wherein generating the one or more pseudo labels comprises: determining that the plurality of keypoint estimates are greater than a total score threshold; as well as The plurality of keypoint estimates are selected as the one or more pseudo-labels based on the plurality of keypoint estimates being greater than the total score threshold.

13. The method of claim 1, wherein the one or more parameters include weights of the machine learning model.

14. An apparatus for performing online adaptation of one or more machine learning models, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtaining features extracted from the first image by the machine learning model during inference; determining, using the machine learning model, at least one of a plurality of keypoint estimates in the first image or a boundary region estimate associated with an object in the first image based on the features during inference; generating one or more pseudo labels based on at least one of the plurality of keypoint estimates or the boundary region estimates; determining at least one self-supervised loss based on at least one of the plurality of keypoint estimates or the boundary region estimates; self-adapting one or more parameters of the machine learning model based on the at least one self-supervised loss; as well as A segmentation mask is generated for the first image or the second image using the machine learning model with the adapted one or more parameters.

15. The apparatus of claim 14, wherein the at least one processor is configured to: The first image or the second image is processed using the machine learning model with the adapted one or more parameters to generate updated features representing the first image or to generate features representing the second image. 16 . The apparatus of claim 15 , wherein the at least one processor is configured to generate the segmentation mask based on processing the updated feature representing the first image or the feature representing the second image.

17. The apparatus of claim 14, wherein the at least one processor is configured to use a keypoint detection engine of the machine learning model to determine the plurality of keypoint estimates in the first image.

18. The apparatus of claim 14, wherein the at least one processor is configured to use an object detection engine of the machine learning model to determine the boundary region estimate in the first image.

19. The apparatus of claim 14, wherein: The at least one processor is configured to extract the features from the first image using a feature extraction engine of the machine learning model; and In order to adapt the one or more parameters of the machine learning model based on the at least one self-supervised loss, the at least one processor is configured to adapt one or more parameters of the feature extraction engine.

20. The apparatus of claim 14, wherein the at least one processor is configured to generate the segmentation mask using a segmentation mask engine of the machine learning model.

21. The apparatus of claim 14, wherein the plurality of keypoint estimates comprises at least one heat map indicating a probability that a portion of the first image corresponds to a particular keypoint.

22. The apparatus of claim 21, wherein the at least one heat map comprises a plurality of heat maps, each heat map of the plurality of heat maps being associated with a corresponding key point of the plurality of key points.

23. The apparatus of claim 14, wherein the plurality of keypoint estimates comprises coordinates associated with a plurality of keypoints.

24. The apparatus of claim 14, wherein to generate the one or more pseudo labels, the at least one processor is configured to: determining one or more keypoint estimates from the plurality of keypoint estimates that are greater than a single keypoint score threshold; and The one or more keypoint estimates are selected as the one or more pseudo-labels based on the one or more keypoint estimates being greater than the single keypoint score threshold.

25. The apparatus of claim 14, wherein to generate the one or more pseudo labels, the at least one processor is configured to: determining that the plurality of keypoint estimates are greater than a total score threshold; and The plurality of keypoint estimates are selected as the one or more pseudo-labels based on the plurality of keypoint estimates being greater than the total score threshold.

26. The apparatus of claim 14, wherein the one or more parameters comprise weights of the machine learning model.

27. A non-transitory computer readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to: obtaining features extracted from the first image by the machine learning model during inference; determining, using the machine learning model, at least one of a plurality of keypoint estimates in the first image or a boundary region estimate associated with an object in the first image based on the features during inference; generating one or more pseudo labels based on at least one of the plurality of keypoint estimates or the boundary region estimates; determining at least one self-supervised loss based on at least one of the plurality of keypoint estimates or the boundary region estimates; self-adapting one or more parameters of the machine learning model based on the at least one self-supervised loss; as well as A segmentation mask is generated for the first image or the second image using the machine learning model with the adapted one or more parameters.

28. An apparatus for performing online adaptation of one or more machine learning models, the apparatus comprising: means for obtaining features extracted from the first image by the machine learning model during inference; means for determining, by the machine learning model based on the features during inference, at least one of a plurality of keypoint estimates in the first image or a boundary region estimate associated with an object in the first image; means for generating one or more pseudo labels based on at least one of the plurality of keypoint estimates or the boundary region estimates; means for determining at least one self-supervised loss based on at least one of the plurality of keypoint estimates or the boundary region estimates; means for self-adapting one or more parameters of the machine learning model based on the at least one self-supervised loss; and Means for generating a segmentation mask for the first image or the second image using the machine learning model with the adapted one or more parameters.

Images