Systems and methods for collaborative enhanced sensing

The collaborative enhanced sensing system generates enhanced views through multi-view sensor data and vehicle context information, solving the collision problems caused by vehicle blind spots and improving driving safety.

CN118985011APending Publication Date: 2024-11-19QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380033015.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-18
Filing Date
2023-04-06
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The blind spots of vehicles lead to frequent collision accidents, and the prior art is difficult to effectively sense blocked or difficult to detect parts of the environment, especially when the driver's field of vision is blocked.

Method used

The collaborative augmented sensing system generates an enhanced view and outputs an enhanced representation of the environment, including difficult-to-sensing parts, for use in combination with the context information of the vehicle, such as location, direction of travel and planned movement. Provides collision alerts.

Benefits of technology

Improve the driving safety of vehicles, and by providing enhanced indications and alerts to the environment, the risk of collision caused by blind spots is reduced, and the driver's perception of the environment is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure HDA0005074463700000011
    Figure HDA0005074463700000011
  • Figure HDA0005074463700000021
    Figure HDA0005074463700000021
  • Figure HDA0005074463700000031
    Figure HDA0005074463700000031
Patent Text Reader

Abstract

Cooperative enhanced sensing systems and techniques are described. The system obtains sensor data of the environment from one or more sensors. The sensor data includes different representations of one or more portions of the environment from multiple perspectives. The system identifies subsets of different representations of one or more portions of the environment based on contexts corresponding to vehicles in the environment. The context may include, for example, a location, direction of travel, speed, and / or planned movement of the vehicle. The system causes an enhanced representation of the environment for the vehicle to be generated based at least in part on merging and / or fusing together subsets of the different representations. The system may cause an indication of the enhanced representation of the environment to be output at the vehicle.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to sensor data processing. More specifically, the present application relates to systems and methods for collaboratively enhancing sensing to indicate relevant information about an environment to a receiving device such as a vehicle in the environment. Background Art

[0002] Vehicle crashes are one of the most common causes of death or serious injury worldwide. While vehicle technology has improved over time, vehicles still have blind spots that are difficult for drivers to see, or sometimes not at all, which can lead to vehicle crashes. Sometimes, a vehicle's field of view may be blocked by objects such as buildings or large trucks, making it difficult for the vehicle's driver to perceive important aspects of the environment. This can also lead to vehicle crashes. Summary of the invention

[0003] In some examples, systems and techniques for collaborative enhanced sensing are described. A collaborative enhanced sensing system obtains sensor data of an environment from one or more sensors of at least one device, for example, using a transceiver. The sensor data includes multiple different representations of one or more parts of the environment from multiple perspectives. The collaborative enhanced sensing system identifies a subset of multiple different representations of one or more parts of the environment based on a context corresponding to a vehicle in the environment. The context may include, for example, the location, direction of travel, speed, and / or planned movement of the vehicle. The collaborative enhanced sensing system enables an enhanced view of the environment for the vehicle to be generated based at least in part on merging and / or fusing together a subset of multiple different representations. The enhanced representation of the environment may include a representation of a portion of the environment that is difficult to sense from the perspective of the vehicle, for example, due to occlusion, environmental scene conditions, sensor conditions, or some combination thereof. The collaborative enhanced sensing system may enable an indication of an enhanced representation of the environment to be output at the vehicle, such as an alarm based on the enhanced representation of the environment. Such an alarm may warn the vehicle and / or its operator about a possible collision between the vehicle and another object in the environment.

[0004] In one example, an apparatus for media processing is provided. The apparatus includes a memory and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured and can: obtain sensor data of an environment from one or more sensors, wherein the sensor data includes multiple different representations of one or more parts of the environment from multiple perspectives; identify a subset of the multiple different representations of one or more parts of the environment based on a context corresponding to a vehicle in the environment, wherein the context includes at least a location of the vehicle in the environment; and generate an enhanced representation of the environment for the vehicle based at least in part on merging the subset of the multiple different representations.

[0005] In another example, a method of image processing is provided. The method includes: obtaining sensor data of an environment from one or more sensors, wherein the sensor data includes multiple different representations of one or more parts of the environment from multiple perspectives; identifying a subset of the multiple different representations of the one or more parts of the environment based on a context corresponding to a vehicle in the environment, wherein the context includes at least a location of the vehicle in the environment; and causing an enhanced representation of the environment for the vehicle to be generated based at least in part on merging the subset of the multiple different representations.

[0006] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which instructions, when executed by one or more processors, cause the one or more processors to: obtain sensor data of an environment from one or more sensors, wherein the sensor data includes multiple different representations of one or more portions of the environment from multiple perspectives; identify a subset of the multiple different representations of the one or more portions of the environment based on a context corresponding to a vehicle in the environment, wherein the context includes at least a position of the vehicle in the environment; and cause an enhanced representation of the environment for the vehicle to be generated based at least in part on merging the subset of the multiple different representations.

[0007] In another example, an apparatus for image processing is provided. The apparatus includes: means for obtaining sensor data of an environment from one or more sensors, wherein the sensor data includes multiple different representations of one or more portions of the environment from multiple perspectives; means for identifying a subset of the multiple different representations of the one or more portions of the environment based on a context corresponding to a vehicle in the environment, wherein the context includes at least a location of the vehicle in the environment; and means for causing an enhanced representation of the environment for the vehicle to be generated based at least in part on merging the subset of the multiple different representations.

[0008] In some aspects, identifying a subset of the plurality of different representations based on the context includes identifying one or more representations of a portion of the environment that includes a location of a vehicle. In some aspects, the context includes a direction of travel of a vehicle in the environment, wherein identifying a subset of the plurality of different representations based on the context includes identifying one or more representations of a portion of the environment that is located in front of the vehicle in the direction of travel of the vehicle. In some aspects, the context includes a speed of the vehicle in the environment, wherein identifying a subset of the plurality of different representations based on the context includes identifying one or more representations of a portion of the environment that is located in front of the vehicle, wherein a size of the portion of the environment is based on the speed of the vehicle.

[0009] In some aspects, the context includes a planned movement of a vehicle in the environment, wherein identifying a subset of the plurality of different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle based on the planned movement of the vehicle. In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: detecting an indicator light of the vehicle in the sensor data; and determining the planned movement of the vehicle based on detecting the indicator light of the vehicle in the sensor data, wherein the planned movement of the vehicle is a turn. In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: receiving an indication of the planned movement of the vehicle from the vehicle, wherein the planned movement of the vehicle is part of a planned route of the vehicle.

[0010] In some aspects, the context includes objects in the environment other than the vehicle, wherein identifying a subset of the multiple different representations based on the context includes identifying one or more representations of a portion of the environment that includes the object. In some aspects, one or more of the above methods, devices, and computer-readable media also include: identifying a risk of collision between the vehicle and the object based on one or more planned movements of the vehicle; and causing an alert based on an enhanced representation of the environment to be output at the vehicle, wherein the alert indicates a risk of collision between the vehicle and the object. In some aspects, the context includes planned movements of objects in the environment other than the vehicle, wherein identifying a subset of the multiple different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle according to the planned movement of the object.

[0011] In some aspects, identifying a subset of multiple different representations based on context includes identifying that a confidence level associated with each representation in the subset of multiple different representations exceeds a confidence level threshold, wherein the confidence level of one of the multiple different representations is associated with a confidence that the one of the multiple different representations represents a portion of an environment that is located in front of a vehicle. In some aspects, the portion of the environment that is located in front of a vehicle includes the vehicle. In some aspects, the confidence level is based on a resolution of one of the multiple different representations. In some aspects, the confidence level is based on an illumination level in one of the multiple different representations. In some aspects, the confidence level is based on object detection using one of the multiple different representations.

[0012] In some aspects, receiving sensor data for the environment includes receiving at least a portion of sensor data including at least one of a plurality of different representations from a second vehicle in the environment. In some aspects, receiving sensor data for the environment includes receiving at least a portion of sensor data including at least one of a plurality of different representations from a roadside unit (RSU) in the environment. In some aspects, receiving sensor data for the environment includes receiving at least a portion of sensor data including at least one of a plurality of different representations from a camera in the environment. In some aspects, receiving sensor data for the environment includes receiving at least a portion of sensor data including at least one of a plurality of different representations from a sensor of a vehicle. In some aspects, the plurality of different representations includes depth data captured using at least one depth sensor. In some aspects, the plurality of different representations includes image data captured using at least one image sensor.

[0013] In some aspects, one or more of the above methods, apparatus, and computer-readable media further include: receiving traffic data from a traffic management controller, wherein the context includes at least a portion of the traffic data, wherein the portion of the traffic data is associated with at least a portion of an environment including a vehicle.

[0014] In some aspects, causing generation of an enhanced representation of an environment for a vehicle based at least in part on merging a subset of the plurality of different representations includes fusing together at least two different representations of the subset of the plurality of different representations. In some aspects, the at least two different representations include an image captured by an image sensor and a depth dataset captured by a depth sensor, and wherein fusing together the at least two different representations includes associating portions of the image with corresponding portions of the depth dataset representing corresponding portions of the environment.

[0015] In some aspects, identifying a subset of the plurality of different representations based on the context corresponding to the vehicle in the environment includes inputting the sensor data and the context into one or more trained machine learning models that output indications of the subset of the plurality of different representations. In some aspects, one or more of the above methods, apparatus, and computer-readable media further includes: identifying a reaction of the vehicle in response to receiving the augmented representation of the environment; and updating the one or more trained machine learning models using the context, the subset of the plurality of different representations, and the reaction as training data.

[0016] In some aspects, obtaining sensor data of the environment from one or more sensors includes obtaining sensor data of the environment from one or more sensors using at least one of a receiver, a transceiver, and / or a communication interface. In some aspects, obtaining sensor data of the environment from one or more sensors includes obtaining sensor data of the environment from at least one device including at least one of the one or more sensors. In some aspects, the at least one device is non-co-located with the device. In some aspects, obtaining sensor data of the environment from one or more sensors includes obtaining sensor data of the environment from multiple devices including at least one of the one or more sensors. In some aspects, the multiple devices are non-co-located with each other.

[0017] In some aspects, identifying a subset of the multiple different representations of one or more portions of the environment based on the context includes identifying the subset of the multiple different representations of the one or more portions of the environment without any user input. In some aspects, identifying a subset of the multiple different representations of the one or more portions of the environment based on the context includes identifying the subset of the multiple different representations of the one or more portions of the environment based on a selection made by a user using a user interface. In some aspects, identifying a subset of the multiple different representations of the one or more portions of the environment based on the context includes identifying the subset of the multiple different representations of the one or more portions of the environment based on preferences associated with the user.

[0018] In some aspects, at least a first representation of a plurality of different representations is received from a first vehicle, and at least a second representation of a plurality of different representations is received from a second vehicle. In some aspects, at least a first representation of a plurality of different representations is received from a first vehicle, and at least a second representation of a plurality of different representations is received from at least one of a roadside unit (RSU), a multi-access edge computing (MEC) system, or a traffic management controller (TMC). In some aspects, at least a first representation of a plurality of different representations is received from a first roadside unit (RSU), and at least a second representation of a plurality of different representations is received from a second RSU. In some aspects, at least a first representation of a plurality of different representations is received from a vehicle, and at least a second representation of a plurality of different representations is received from a second vehicle. In some aspects, at least a first representation of a plurality of different representations is received from a vehicle, and at least a second representation of a plurality of different representations is received from at least one of a roadside unit (RSU), a multi-access edge computing (MEC) system, or a traffic management controller (TMC).

[0019] In some aspects, the apparatus includes a computing device remote from a vehicle, further comprising: identifying a vehicle in the environment before causing an enhanced representation of the environment to be generated for the vehicle; and transmitting an indication of the enhanced representation of the environment to the vehicle. In some aspects, the computing device of the apparatus is one of a multi-access edge computing (MEC) system, a roadside unit (RSU) system, a remote server, or a vehicle computing device, wherein the vehicle computing device is part of a second vehicle in the environment.

[0020] In some aspects, the apparatus includes a vehicle computing device of a vehicle, further comprising: outputting an instruction of an augmented representation of the environment using an output device of the vehicle.

[0021] In some aspects, the apparatus is part of and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smart phone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or a component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes one or more cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0022] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0023] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Exemplary embodiments of the present application are described in detail below with reference to the following drawings, in which:

[0025] Figure 1 is a block diagram illustrating an example architecture of an image capture and processing system according to some examples;

[0026] Figure 2is a block diagram illustrating an example architecture of a collaborative enhanced sensing system that performs processes for collaborative enhanced sensing according to some examples;

[0027] Figure 3 is a top view illustrating an example of collaborative enhanced sensing in a road environment with multiple vehicles according to some examples;

[0028] Figure 4 is a top view illustrating an example of an apparatus involving wireless communications according to some examples;

[0029] Figure 5A is a top view illustrating an example of sensor sharing for a cooperative autonomous driving system based on detection of non-vehicle-to-everything (non-V2X) vehicles according to some examples;

[0030] Figure 5B is a top view illustrating an example of sensor sharing for a cooperative autonomous driving system based on obstacle detection according to some examples;

[0031] Figure 5C is a top view illustrating an example of sensor sharing for a cooperative automated driving system based on vulnerable road user (VRU) detection according to some examples;

[0032] Figure 5D is a swim lane diagram illustrating an example of sensor sharing for a collaborative automated driving system based on transmission of sensor data sharing messages (SDSMs) according to some examples;

[0033] Figure 6 is a conceptual diagram illustrating an example of sensor sharing for a cooperative autonomous driving system according to some examples;

[0034] Figure 7 is a top view illustrating an example of collaborative enhanced sensing in a road environment with multiple vehicles according to some examples;

[0035] Figure 8 is a top view illustrating an example of collaborative augmented sensing in a road environment providing different augmented views to different vehicles according to some examples;

[0036] Fig. 9 is a block diagram illustrating an example of a neural network that may be used to collaboratively enhance sensing operations according to some examples;

[0037] Fig.10 is a flow chart illustrating a collaborative enhanced sensing process according to some examples; and

[0038] Fig.11 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0039] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0040] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes can be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0041] A camera is a device that uses an image sensor to receive light and capture an image frame (such as a still image or a video frame). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings produce images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f / stop (aperture number), shutter speed, focus and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters can be applied to an image sensor used to capture one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as changes in contrast, brightness, saturation, sharpness, color scale, curves or colors. For example, settings or parameters can be applied to a processor (e.g., an image signal processor or ISP) used to process one or more image frames captured by an image sensor.

[0042] Vehicle crashes are one of the most common causes of death or serious injury worldwide. While vehicle technology has improved over time, vehicles still have blind spots that are difficult for drivers to see, or sometimes not at all, which can lead to vehicle crashes. Sometimes, a vehicle's field of view may be blocked by objects such as buildings or large trucks, making it difficult for the driver of the vehicle to perceive important aspects of the environment. This can also lead to vehicle crashes. Even vehicles with sensors may not be able to sense certain conditions in the environment that may cause the vehicle to crash, such as if a condition in the environment is outside the sensor's field of view because the sensor's field of view is blocked by another vehicle.

[0043] In some examples, systems and techniques for collaborative enhanced sensing are described. A collaborative enhanced sensing system obtains sensor data of an environment from one or more sensors. The sensor data includes multiple different representations of one or more parts of the environment from multiple perspectives. In some examples, the collaborative enhanced sensing system identifies a vehicle in the environment. The collaborative enhanced sensing system identifies a subset of multiple different representations of one or more parts of the environment based on a context corresponding to the vehicle in the environment. The context may include, for example, the location, direction of travel, speed, and / or planned movement of the vehicle. The collaborative enhanced sensing system enables an enhanced view of the environment for the vehicle to be generated based at least in part on merging and / or fusing together a subset of multiple different representations. The enhanced representation of the environment may include a representation of a portion of the environment that is difficult to sense from the perspective of the vehicle, for example, due to occlusion, environmental scene conditions, sensor conditions, or some combination thereof. The collaborative enhanced sensing system may enable an indication of an enhanced representation of the environment to be output at the vehicle, such as an alarm based on the enhanced representation of the environment. Such an alarm may warn the vehicle and / or its operator about a possible collision between the vehicle and another object in the environment.

[0044] The synergistic enhanced sensing systems and techniques described herein provide a variety of technical improvements over existing vehicle systems, sensor systems, and communication systems. For example, the synergistic enhanced sensing systems and techniques described herein can provide additional functionality to a vehicle, thereby providing improvements to vehicle travel safety by providing enhanced representations and / or alerts to the vehicle that are customized for the vehicle based on portions of the environment that may be difficult or impossible to sense from the vehicle's perspective.

[0045] Various aspects of the application will be described with reference to the accompanying drawings. Figure 1 1 is a block diagram illustrating the architecture of image capture and processing system 100. Image capture and processing system 100 includes various components for capturing and processing images of one or more scenes (e.g., images of scene 110). Image capture and processing system 100 may capture individual images (or photographs) and / or may capture a video including multiple images (or video frames) in a particular sequence. Lens 115 of system 100 faces scene 110 and receives light from scene 110. Lens 115 bends the light toward image sensor 130. Light received by lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by image sensor 130. In some examples, scene 110 is a scene in an environment. In some examples, scene 110 is a user (such as Figure 2For example, scene 110 may be a scene of one or both of the user's eyes and / or at least a portion of the user's face. In some examples, scene 110 is an environment (such as a Figure 2 In some examples, scene 110 is at least a portion of a vehicle (such as a vehicle). Figure 2 The scene of the vehicle (the vehicle that the sensor 212, sensor 218 and / or sensor 238 is facing).

[0046] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include a plurality of mechanisms and components; for example, the control mechanisms 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms other than those illustrated, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0047] A focus control mechanism 125B of the control mechanism 120 may obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B may adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may move the lens 115 closer to the image sensor 130 or further away from the image sensor 130 by actuating a motor or a servo, thereby adjusting the focus. In some cases, an additional lens may be included in the system 100, such as one or more micro lenses above each photodiode of the image sensor 130, each of which bends the light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined by contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0048] Exposure control mechanism 125A of control mechanism 120 may obtain an exposure setting. In some cases, exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0049] A zoom control mechanism 125C of the control mechanism 120 may obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C may control the focal length of a lens element assembly (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos to move one or more lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, the focusing lens may be the lens 115) that first receives light from the scene 110, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. An afocal zoom system may, in some cases, include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0050] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image generated by the image sensor 130. In some cases, different photodiodes may be covered by different color filters, and thus may measure light that matches the color of the color filter covering the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of an image is generated based on red light data from at least one photodiode covered by a red filter, blue light data from at least one photodiode covered by a blue filter, and green light data from at least one photodiode covered by a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") filters instead of or in addition to red, blue, and / or green filters. Some image sensors may have no color filters at all, and may instead use different photodiodes (in some cases vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to light of different wavelengths. Monochrome image sensors may also lack color filters and, therefore, lack color depth.

[0051] In some cases, image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier to amplify the analog signal output by the photodiode and / or an analog-to-digital converter (ADC) to convert the analog signal output by the photodiode (and / or the analog signal amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0052] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more processors of any other type of processor 1110 discussed with respect to the computing system 1100. The host processor 152 may be a digital signal processor (DSP) and / or other type of processor. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM , global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0053] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1120, read-only memory (ROM) 145 and / or 1125, cache, memory unit, another storage device, or some combination thereof.

[0054] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 1135, any other input device 1145, or some combination thereof. In some cases, subtitles may be input into the image processing device 105B through a physical keyboard or keypad of the I / O device 160, or through a virtual keyboard or keypad of a touch screen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that implement a wired connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that implement a wireless connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. Peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0055] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0056] like Figure 1 As shown, the vertical dotted line will Figure 1 1 is divided into two parts, representing image capture device 105A and image processing device 105B. Image capture device 105A includes lens 115, control mechanism 120 and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145 and I / O 160. In some cases, some components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0057] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 1102.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some specific implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, or other computing devices.

[0058] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include electronic circuits or other electronic hardware, and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0059] Figure 2 is a block diagram illustrating an example architecture of a collaborative enhanced sensing system 200 that performs processes for collaborative enhanced sensing. The collaborative enhanced sensing system 200 and the corresponding imaging processes may be used for vehicle management, autonomous vehicle control, vehicle-to-vehicle (V2V) communication systems, vehicle-to-infrastructure (V2I) communication systems, vehicle-to-network (V2N) communication systems, vehicle-to-pedestrian (V2P) communication systems, vehicle-to-everything (V2X) communication systems, cellular communication network systems, 5G network communication systems, multi-access edge computing (MEC) systems, or combinations thereof.

[0060] The collaborative augmented sensing system 200 includes one or more data sources 202, one or more traffic management controllers (TMCs) 204, one or more servers 206, and one or more recipient devices 208. Each of the data sources 202, TMCs 204, servers 206, and recipient devices 208 may include at least one computing system 1100. Each of the data sources 202, TMCs 204, servers 206, and recipient devices 208 may include, for example, one or more vehicles, vehicle computers, vehicle controllers, laptops, phones, tablet computers, mobile phones, video game consoles, vehicle computers, desktop computers, wearable devices, televisions, media centers, XR systems, other types of computing devices discussed herein, or combinations thereof. In some examples, the data source 202 includes one or more of the components illustrated in the TMCs 204, servers 206, and / or recipient devices 208.

[0061] In some examples, TMC 204 includes one or more of the components instantiated in data source 202, server 206, and / or recipient device 208. In some examples, server 206 includes one or more of the components instantiated in data source 202, TMC 204, and / or recipient device 208. In some examples, recipient device 208 includes one or more of the components instantiated in data source 202, TMC 204, and / or server 206.

[0062] Each of the data source 202, the TMC 204, the server 206, and the recipient device 208 may include one or more processors 1110. For example, the data source 202 includes the processor 210, the TMC 204 includes the processor 216, the server 206 includes the processor 224, and the recipient device 208 includes the processor 236. Each of the processors 210, 216, 224, and 236 includes one or more processors 1110.

[0063] Each of the data source 202, TMC 204, server 206, and recipient device 208 may include one or more communication interfaces 1140. For example, the data source 202 includes a transceiver 214, the TMC 204 includes a transceiver 220, the server 206 includes a transceiver 226, and the recipient device 208 includes a transceiver 240. Each of the transceivers 214, the transceivers 220, the transceivers 226, and the transceivers 240 includes one or more communication interfaces 1140. Each of the transceivers 214, the transceivers 220, the transceivers 226, and the transceivers 240 may include, for example, a communication interface 1140 using a universal serial bus (USB), One or more wired receivers, transmitters, and / or transceivers for interfaces such as Ethernet, local area network (LAN) connections, general purpose input / output (GPIO) pins, audio jacks, auxiliary jacks, other wired interfaces described herein (e.g., with respect to communication interface 1140 or other interfaces), or combinations thereof. Each of transceiver 214, transceiver 220, transceiver 226, and transceiver 240 may include, for example, a wireless network such as Wi-Fi, a wireless local area network (WLAN), a cellular network, an MEC network, a 5G network, One or more wireless receivers, transmitters, and / or transceivers of interfaces for a personal area network (PAN), near field communication (NFC), radio frequency identification (RFID), infrared, visible light communication (VLC), other wireless communication interfaces described herein (e.g., regarding communication interface 1140 or other interfaces), or combinations thereof.

[0064] Each of data source 202, TMC 204, and recipient device 208 may include one or more sensors. For example, data source 202 includes sensor 212, TMC 204 includes sensor 218, and recipient device 208 includes sensor 238. In some examples, server 206 also includes one or more sensors. Each of the sensors 212, 218, and 238 may include one or more cameras, image sensors, microphones, heart rate monitors, oximeters, biometric sensors, positioning receivers, global navigation satellite system (GNSS) receivers, inertial measurement units (IMUs), accelerometers, gyroscopes, gyrometers, barometers, thermometers, altimeters, depth sensors, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, time of flight (ToF) sensors, structured light sensors, ultrasonic sensors, range finders, depth sensors, other sensors discussed herein, input devices 1145, communication interfaces 1140, or combinations thereof. The term depth sensor may refer to a LIDAR sensor, a RADAR sensor, a SODAR sensor, a SONAR sensor, a ToF sensor, a structured light sensor, an ultrasonic sensor, a range finder, or a combination thereof. In some cases, a depth sensor may be referred to as a range sensor or a distance sensor. In some aspects, one or more of these sensors may be used to supplement or refine sensor readings from other sensors. For example, an inertial measurement unit (IMU), accelerometer, gyroscope, or other sensor may be used (e.g., by object tracking engine 228 and / or data fusion engine 230) to refine object detection, object recognition, object tracking, and / or sensor data fusion using data from other sensors.

[0065] The data source 202 uses the sensor 212 to capture sensor data. In some examples, the sensors 212 of different data sources 202 may capture sensor data including representations of different parts of the environment. In some examples, the sensors 212 of different data sources 202 may capture sensor data including representations of the environment from different perspectives. For example, the perspective and / or portion may be based on the pose (position (e.g., 3D coordinates) and / or orientation (e.g., pitch, yaw, and / or roll)) of the data source 202 and / or the sensor 212 of the data source 202. Examples of data sources 202 include vehicles with sensors 212, roadside units (RSUs) with sensors 212, network infrastructure with sensors 212, traffic infrastructure with sensors 212, buildings with sensors 212, edge nodes in a cellular network with sensors 212, user equipment (UE) in a cellular network with sensors 212, or combinations thereof. In some examples, the environment may include the data source 202, the TMC 204, the server 206, the receiving device 208, or combinations thereof.

[0066] The data source 202 may use the processor 210 to process the sensor data captured by the sensor 212. The data source 202 may use the processor 210 to actuate the motor and / or components of the sensor 212 to direct the sensor 212 toward (e.g., point the sensor 212 at and / or focus the sensor 212 toward) a specific portion of the environment to capture sensor data that represents the specific portion of the environment. For example, if the environment includes a specific recipient device 208 of interest (e.g., a vehicle of interest), the data source 202 may use the processor 210 to direct the sensor 212 toward the portion of the environment that includes the recipient device of interest 208 (e.g., a vehicle of interest) and / or toward the portion of the environment that is located in front of the recipient device of interest 208 (e.g., a vehicle of interest), for example, based on the direction, speed, and / or planned movement of the recipient device of interest 208 (e.g., a vehicle of interest).

[0067] In some cases, after processing the sensor data using processor 210, data source 202 may transmit the sensor data captured by sensor 212 to server 206 using transceiver 214. Transceiver 214 may transmit the sensor data from data source 202 to server 206 via a wired connection, a wireless connection, or a combination thereof.

[0068] Traffic management controller (TMC) 204 may also include sensor 218. Sensor 218 of TMC 204 may include any type of sensor or combination thereof described with respect to sensor 212. TMC 204 may use sensor 218 to capture any type of sensor data as described with respect to sensor 212 being used by data source 202 to capture sensor data. TMC 204 may include processor 216 that may process sensor data from sensor 218 as described with respect to processor 210 and sensor 212. TMC 204 may use processor 216 to direct sensor 218 as described with respect to processor 210 and sensor 212.

[0069] The TMC 204 also includes a traffic control engine 222. The traffic control engine 222 may receive, capture, and / or generate traffic data. The TMC 222 may include devices for various traffic management applications, such as traffic lights for controlling traffic at intersections, electronic toll collection devices for collecting tolls, electronic speed limit display signs, city monitoring devices for monitoring traffic throughout a city, road monitoring devices for monitoring traffic along a particular road, license plate recognition devices for identifying individual vehicles, public transportation stations where passengers board or leave public transportation, roadside units (RSUs), or combinations thereof. In some examples, the traffic control engine 222 of the TMC 204 may receive, capture, and / or generate traffic data that, for example, uses the traffic lights of the TMC 204 to control when passage through a particular road segment (such as an intersection) is allowed and / or not allowed (e.g., prohibited). Traffic data may, for example, dynamically indicate the timing for one or more traffic lights so that stop lights are minimized for vehicles traveling along the main roads of the city, ensuring that pedestrians have enough time to walk through the intersection, ensuring that vehicles can turn safely, or a combination thereof. In some examples, traffic data may dynamically change the speed limit on an electronic speed limit display sign, such as reducing the speed limit to increase safety around schools when schools are on or off that day, increasing the speed limit during peak traffic (e.g., rush hour) to reduce traffic congestion, etc. In some examples, traffic data may dynamically change tolls, such as increasing tolls to inhibit the use of certain roads when they are busy, or reducing tolls to encourage the use of certain roads when traffic is light. In some examples, traffic data may monitor which roads and / or areas of a region (e.g., a city) are busy or not busy at a given past or current time. In some examples, traffic data may predict which roads and / or areas of a region (e.g., a city) are busy or not busy at a given future time. In some examples, traffic data from traffic control engine 222 may detect, monitor, and / or track attributes (e.g., location, speed, direction of movement, acceleration, deceleration) of a particular vehicle within an environment (e.g., which may be one of receiving devices 208).

[0070] In some cases, after processing the sensor data and / or combining the sensor data with the traffic data using the processor 216, the TMC 204 may transmit the sensor data captured by the sensor 218 to the server 206 using the transceiver 220. In some cases, after processing the traffic data and / or combining the traffic data with the sensor data using the processor 216, the TMC 204 may transmit the traffic data captured and / or generated by the traffic control engine 222 to the server 206 using the transceiver 220. The transceiver 220 may transmit the sensor data and / or traffic data from the TMC 204 to the server 206 via a wired connection, a wireless connection, or a combination thereof. The transceiver 220 may include the components discussed with respect to the transceiver 214 and / or may function as discussed with respect to the transceiver 214.

[0071] The receiving device 208 may include, for example, a vehicle, a UE, an edge node, a mobile phone, a wearable device, a head mounted display (HMD) device (e.g., for augmented reality (AR), virtual reality (VR), mixed reality (MR), and / or extended reality (XR)), any other type of computing system 1100, or a combination thereof. The receiving device 208 may be associated with a vehicle, a pedestrian, a cyclist, an infrastructure, or a combination thereof. In some examples, one or more of the receiving devices 208 may also be one of the data sources 202. The receiving device 208 may include a sensor 238. The sensor 238 of the receiving device 208 may include any type of sensor or a combination thereof described with respect to the sensor 212 and / or the sensor 218. The receiving device 208 may use the sensor 238 to capture any type of sensor data, as described with respect to capturing sensor data using the sensor 212 and / or the sensor 218. The recipient device 208 may include a processor 236 that may process sensor data from the sensor 238, as described with respect to the processor 210 and the sensor 212 and / or the processor 216 and the sensor 218. The recipient device 208 may use the processor 236 to direct the sensor 238, as described with respect to the processor 210 and the sensor 212 and / or the processor 216 and the sensor 218.

[0072] In some cases, after processing the sensor data and / or combining the sensor data with traffic data using the processor 236, the recipient device 208 may transmit the sensor data captured by the sensor 238 to the server 206 using the transceiver 240. The transceiver 240 may transmit the sensor data from the recipient device 208 to the server 206 via a wired connection, a wireless connection, or a combination thereof. The transceiver 240 may include the components discussed with respect to the transceiver 214 and / or the transceiver 220, and / or may function as discussed with respect to the transceiver 214 and / or the transceiver 220.

[0073] Examples of server 206 include one or more MEC systems, base stations, access nodes, telecommunication nodes, Node Bs, enhanced Node Bs (eNodeBs), next generation Node Bs (gNodeBs), base transceiver stations (BTSs), radio network controllers (RNCs), base station controllers (BSCs), any type of server, any type of computing system 1100, or combinations thereof.

[0074] The server 206 includes an object tracking engine 228 that can extract features from input data, detect objects in the input data, identify objects in the input data, and / or track objects in the input data. The input data may include sensor data (e.g., sensor data captured by sensor 212, sensor data captured by sensor 218, sensor data captured by sensor 238), traffic data (e.g., captured, received, and / or generated by traffic control engine 222), processing data (e.g., processing of other input data performed by processor 210, processor 216, processor 224, and / or processor 236), or a combination thereof. The server 206 may include one or more trained machine learning (ML) models 232. In some examples, the object tracking engine 228 inputs the input data into the trained ML model 232 to perform feature extraction, feature detection, feature recognition, feature tracking, object detection, object recognition, object tracking, face detection, face recognition, face tracking, person detection, person recognition, person tracking, vehicle detection, vehicle recognition, vehicle tracking, device detection, device recognition, device tracking, or a combination thereof based on the input data. The trained ML model 232 used by the object tracking engine 228 can be trained (e.g., by the server 206) using training data that includes the same type of data as described with respect to the input data, with indications of pre-identified features, objects, persons, faces, vehicles, devices, and / or other detectable, recognizable, and / or trackable content. In some examples, the object tracking engine 228 can also output a confidence level regarding the detection, recognition, and / or tracking of objects in the input data. In some examples, if the confidence level drops below (and / or does not exceed) a predetermined confidence level threshold, the object tracking engine 228 discards the detection, identification, and / or tracking of the object in the input data. In some examples, the object tracking engine 228 may perform object detection, detection, identification, and / or tracking based on sensor data (e.g., image data captured by an image sensor and / or depth data captured by a depth sensor). In some examples, the object tracking engine 228 may perform object detection, detection, identification, and / or tracking based on traffic data (e.g., traffic data from the traffic control engine 222). In some examples, the object tracking engine 228 may perform object detection, detection, identification, and / or tracking based on data received from the transceiver 240 of the recipient device 208, for example, when the recipient device 208 is the object being tracked, the data is the positioning receiver data and / or IMU data of the recipient device 208. In some examples, the object tracking engine 228 may perform object detection, detection, identification, and / or tracking based on a combination of sensor data and traffic data.

[0075] In some examples, the object tracking engine 228 may identify the location of an object, such as a vehicle. In some examples, the object tracking engine 228 may identify various attributes of the movement of an object, such as a vehicle. For example, the object tracking engine 228 may identify the direction of movement of the object, the speed of movement of the object, the acceleration rate of the object, the deceleration rate of the object, whether the object is turning, the degree of turning of the object (e.g., the turning angle and / or the turning radius), or a combination thereof. In some examples, the object tracking engine 228 may perform one or more planned movements of the object. For example, if the object is a vehicle, the object tracking engine 228 may detect and track the illumination of the indicator lights to determine the planned movement. For example, the object tracking engine 228 may detect and track the illumination of the turn indicator lights of the vehicle, thereby indicating that the vehicle plans to turn in the indicated direction, merge / change to a different lane, pull over and / or park. The object tracking engine 228 may detect and track the illumination of the brake lights of the vehicle, thereby indicating that the vehicle is slowing down and may plan to continue to slow down. The object tracking engine 228 may detect and track the illumination of the vehicle's hazard warning lights, thereby indicating that the vehicle is planning to slow down, drive slowly, and / or stop. In some examples, the object tracking engine 228 may detect and track the vehicle's planned movement based on data received from the vehicle itself (e.g., data received from the transceiver 240 of the recipient device 208). For example, if the vehicle is autonomous or includes certain autonomous or semi-autonomous features, the vehicle may have a planned route (or a portion thereof), the vehicle may transmit the planned route to the server 206, and the object tracking engine 228 may use the planned route to determine the vehicle's planned movement. The vehicle's planned movement may be used by the object tracking engine 228 to better predict the location of the vehicle at a given future time, and therefore to better track the vehicle through the environment based on this.

[0076] The server 206 includes a data fusion engine 230. The data fusion engine 230 may merge and / or fuse together different input data sets to generate an output data set consisting of the merged and / or fused data. The input data sets may include, for example, sensor data captured by the sensor 212 of the data source 202, sensor data captured by the sensor 218 of the TMC 204, traffic data from the traffic control engine 222, sensor data captured by the sensor 238 of the recipient device 208, processed input data from the processor 210, processed input data from the processor 216, processed input data from the processor 224, processed input data from the processor 236, object detection data from the object tracking engine 228, object recognition data from the object tracking engine 228, and / or object tracking data from the object tracking engine 228. In some examples, the data fusion engine 230 may use the trained ML model 232 of the server 206 to merge and / or fuse together different input data sets into an output data set, for example by providing the input data sets as input to the trained ML model 232. The trained ML model 232 used by the object tracking engine 228 may be trained (e.g., by the server 206) using training data that includes the same type of data as described with respect to the input data sets, along with pre-merged or pre-fused combinations of such data (e.g., examples of how the output data should look for those inputs in the training data). In some examples, the data fusion engine 230 may output a confidence level regarding the merging and / or fusion of the input data to form the output data. In some examples, the data fusion engine 230 discards portions of the output data if the confidence level drops below (and / or does not exceed) a predetermined confidence level threshold.

[0077] In some examples, the data fusion engine 230 may merge and / or fuse sensor data sets from different sensors of the same sensor modality to generate output data, such as combining depth data together (e.g., merging point clouds together) and / or combining images together (e.g., stitching images into a panorama). In some examples, the data fusion engine 230 may merge and / or fuse sensor data sets from different sensors of different sensor modalities to generate output data, such as combining image data with depth data to identify three-dimensional positioning information corresponding to corresponding pixels in the image data from the depth data, and / or determine pixels of the image data corresponding to corresponding points or clusters (e.g., point clusters) or polygons (e.g., composed of points) in the depth data.

[0078] The output data generated by the data fusion engine 230 may include an enhanced representation of the environment that may be customized, personalized, and / or tailored to the recipient device 280. In some examples, the data fusion engine 230 may generate an enhanced representation of the environment by selecting a portion of the environment to focus the output data, such as by discarding data that does not fall within the selected portion of the environment. In some examples, the data fusion engine 230 may select the portion of the environment to focus the output data based on object detection data from the object tracking engine 228, object recognition data from the object tracking engine 228, and / or object tracking data from the object tracking engine 228. In some examples, the data fusion engine 230 may select the portion of the environment to focus the output data based on data received from the recipient device 208. For example, if a vehicle of the recipient device 208 is being tracked, the server 206 may identify the location of the vehicle based on the positioning receiver data received from the recipient device 208, may use the object tracking engine 228 to detect and / or identify and / or track the vehicle within the sensor data and / or traffic data, and may select a portion of the environment to focus the output data to include the vehicle, including an area in front of the vehicle in which the vehicle will likely be (e.g., based on the direction of the vehicle, the speed of the vehicle, any planned movement of the vehicle, etc.). The area in front of the vehicle may include an area in front of the vehicle in the direction in which the vehicle is determined (e.g., by the object tracking engine 228) to be moving. The size of the area in front of the vehicle may be based on the speed at which the vehicle is determined (e.g., by the object tracking engine 228) to be moving, such as having a higher speed corresponding to a larger size (e.g., because the vehicle will cover a larger distance at a higher speed during a given amount of time) and a lower speed corresponding to a smaller size (e.g., because the vehicle will cover a smaller distance at a lower speed during a given amount of time). The area in front of the vehicle may include an area that the vehicle is moving toward, determined (e.g., by the object tracking engine 228) based on one or more planned movements of the vehicle (e.g., the direction in which the vehicle plans to turn or change or merge lanes based on its turn indicators). Thus, the enhanced representation of the environment in the output data may be customized, personalized, and / or tailored to the recipient device 280 by including an enhanced representation of a selected portion of the environment that is selected as being relevant to a particular one of the recipient devices 280 (e.g., the vehicle described above).

[0079] In some examples, server 206 may include an alert engine 244 that analyzes object tracking data from object tracking engine 228, output data generated by data fusion engine 230 (e.g., including an enhanced representation of the environment for recipient device 208), and / or any input data input to server 206 (e.g., from data source 202, TMC 204, and / or recipient device 208) to identify potential issues to alert recipient device 208. For example, Figure 3 As depicted, alert engine 244 may analyze that the planned movement of vehicle 310 will result in an impending collision with vehicle 320 that is difficult (or even impossible) to sense (e.g., see or otherwise detect) from the perspective of vehicle 310, and may generate alert 345 to warn vehicle 310. In some cases, alert engine 244 may even modify the augmented representation of the environment for recipient device 208, e.g., to incorporate an alert similar to alert 345 (or other indication of a collision or other problem, such as a line indicating the path of an object in the environment) into the augmented representation of the environment for recipient device 208.

[0080] In some examples, the alert engine 244 may use the trained ML model 232 to perform the analysis and / or generate the alert data. For example, the alert engine 244 may provide object tracking data from the object tracking engine 228, output data generated by the data fusion engine 230 (e.g., including an enhanced representation of the environment for the recipient device 208), and / or any input data input to the server 206 (e.g., from the data source 202, the TMC 204, and / or the recipient device 208) as input to the trained ML model 232. The alert engine 244 may train the trained ML model 232 to generate alert data by providing these same types of training data and indicators of problems (e.g., potential collisions, dangerous maneuvers, road hazards, other types of traffic accidents) pre-identified from the remaining training data. In some examples, the alert engine 244 outputs a confidence level for the alert data and / or the analysis identifying the alert data. In some examples, if the confidence level drops below (and / or does not exceed) a predetermined confidence level threshold, the alert engine 244 discards the portion of the alert data. The alarm engine 244, the data fusion engine 230, and / or the object tracking engine 228 may each have a different predetermined confidence level threshold, or in some cases may share a predetermined confidence level threshold. In some examples, the server 206 may sort and / or assign different weights to the output data based on the corresponding confidence levels. In some examples, the confidence may be based on the quality of the input data, such as the resolution of the image data from the image sensor, the blur or sharpness of the image data from the image sensor, the color range of the image data from the image sensor, etc.

[0081] In some cases, after the object tracking data is processed and / or combined with output data generated by the data fusion engine 230 (e.g., including an enhanced representation of the environment for the recipient device 208) and / or alert data generated by the alert engine 244 (e.g., using the processor 224), the server 206 may transmit the object tracking data generated by the object tracking engine 228 to the recipient device 208 using the transceiver 226. In some cases, after the output data is processed and / or combined with the object tracking data and / or alert data generated by the alert engine 244 (e.g., using the processor 224), the server 206 may transmit the output data generated by the data fusion engine 230 (e.g., including an enhanced representation of the environment for the recipient device 208) to the recipient device 208 using the transceiver 226. In some cases, after the alert data is processed and / or combined (e.g., using processor 224) with object tracking data and / or output data generated by data fusion engine 230 (e.g., including an enhanced representation of the environment for recipient device 208), server 206 may transmit the alert data generated by alert engine 244 to recipient device 208 using transceiver 226. Transceiver 226 may transmit object tracking data from object tracking engine 228 and / or output data from data fusion engine 230 (e.g., including an enhanced representation of the environment for recipient device 208) and / or alert data generated by alert engine 244 to recipient device 208 via a wired connection, a wireless connection, or a combination thereof. Transceiver 226 may include components discussed with respect to transceiver 214, transceiver 220, and / or transceiver 240, and / or may function as discussed with respect to transceiver 214, transceiver 220, and / or transceiver 240.

[0082] The recipient device 208 can use the transceiver 240 to receive data, for example, from the server 206. The data received by the transceiver 240 of the recipient device 208 may include, for example, object tracking data from the object tracking engine 228, output data from the data fusion engine 230 (e.g., including an enhanced representation of the environment for the recipient device 208), and / or alert data generated by the alert engine 244. The recipient device 208 includes an output device 242, which can output the data received by the recipient device 208 using the transceiver 240, for example, by visually displaying the data (e.g., using a display), by audibly playing the data (e.g., using a speaker or headphones), by providing tactile feedback (e.g., using a tactile feedback actuator), or a combination thereof.

[0083] Output device 242 may include one or more visual output devices, such as a display or a connector therefor. Output device 242 may include one or more audio output devices, such as a speaker, headphones, and / or connectors therefor. Output device 242 may include one or more of output device 1135 and / or communication interface 1140 of computing system 1100. Recipient device 208 causes a display of output device 242 to display object tracking data from object tracking engine 228, output data from data fusion engine 230 (e.g., including an enhanced representation of the environment for recipient device 208), and / or alarm data generated by alarm engine 244.

[0084] In some examples, the display of the output device 242 of the recipient device 208 is used as an optical "see-through" display, which allows light from the real-world environment (scene) around the output device 242 to traverse (e.g., pass through) the display of the output device 242 to one or both eyes of the user. For example, the display of the output device 242 may be at least partially transparent, translucent, light-transmissive, light-transmissive, or a combination thereof. In an illustrative example, the display of the output device 242 includes a transparent, translucent and / or light-transmissive lens and a projector. The display of the output device 242 may include a projector that projects virtual content onto the lens. The lens may be, for example, a lens of a pair of glasses, a lens of goggles, a contact lens, a lens of a head-mounted display (HMD) device, or a combination thereof. Light from the real-world environment passes through the lens and reaches one or both eyes of the user. The projector may project virtual content onto the lens so that from the perspective of one or both eyes of the user, the virtual content appears to be superimposed on the user's view of the environment. In some examples, the projector may project virtual content onto one or both retinas of one or both eyes of the user, rather than onto a lens, which may be referred to as a virtual retinal display (VRD), a retinal scanning display (RSD), or a retinal projector (RP) display. The virtual content may include, for example, object tracking data from the object tracking engine 228, output data from the data fusion engine 230 (e.g., including an augmented representation of the environment for the recipient device 208), and / or alarm data generated by the alarm engine 244.

[0085] In some examples, the display of the output device 242 of the recipient device 208 is a digital "transparent" display that allows the user of the collaborative augmented sensing system 200 to view a view of the environment by displaying the view of the environment on the display of the output device 242. For example, based on sensor data (e.g., images, videos, depth images, point clouds, other depth data, or a combination thereof) captured by one or more environment-oriented sensors in the sensors 238, the environment view displayed on the digital transparent display can be a view of the real-world environment around the output device 242. The environment view displayed on the digital transparent display can be a virtual environment (e.g., as in VR), which in some cases can include elements based on the real-world environment (e.g., the boundaries of a room). The environment view displayed on the digital transparent display can be an augmented environment based on the real-world environment (e.g., as in AR). The environment view displayed on the digital transparent display can be a mixed environment based on the real-world environment (e.g., as in MR). The environment view displayed on the digital transparent display may include virtual content superimposed on other content otherwise incorporated into the environment view. The virtual content may include, for example, object tracking data from the object tracking engine 228 , output data from the data fusion engine 230 (eg, including an enhanced representation of the environment for the recipient device 208 ), and / or alert data generated by the alert engine 244 .

[0086] The trained ML model 232 may include one or more neural networks (NNs) (e.g., neural network 900), one or more convolutional neural networks (CNNs), one or more trained time-delay neural networks (TDNNs), one or more deep networks, one or more autoencoders, one or more deep belief networks (DBNs), one or more recurrent neural networks (RNNs), one or more generative adversarial networks (GANs), one or more conditional generative adversarial networks (cGANs), one or more other types of neural networks, one or more trained support vector machines (SVMs), one or more trained random forests (RFs), one or more computer vision systems, one or more deep learning systems, or a combination thereof.

[0087] In some examples, the collaborative augmented sensing system 200 includes a feedback engine 234. The feedback engine 234 is illustrated as part of the server 206, but may also be additionally or alternatively part of the data source 202, the TMC 204, the recipient device 208, or another device. The feedback engine 234 can detect feedback received from a user interface of any device of the collaborative augmented sensing system 200. The feedback can include feedback about object tracking data from the object tracking engine 228, output data from the data fusion engine 230 (e.g., including an enhanced representation of the environment for the recipient device 208), and / or alert data generated by the alert engine 244.

[0088] The feedback engine 234 may detect feedback received from one engine of the collaborative enhanced sensing system 200 about another engine of the collaborative enhanced sensing system 200, such as whether one engine decides to use data from other engines. For example, the feedback engine 234 may detect whether the alarm engine 244 decides to use data from the data fusion engine 230 and / or the object tracking engine 228, whether the data fusion engine 230 decides to use data from the object tracking engine 228, and so on. The feedback received by the feedback engine 234 may be positive feedback or negative feedback. For example, if one engine of the collaborative enhanced sensing system 200 uses data from another engine of the collaborative enhanced sensing system 200, or if positive feedback is received from a user through a user interface, the feedback engine 234 may interpret this as positive feedback. If one engine of the collaborative enhanced sensing system 200 rejects data from another engine of the collaborative enhanced sensing system 200, or if negative feedback is received from a user through a user interface, the feedback engine 234 may interpret this as negative feedback. Positive feedback may also be based on attributes of sensor data from sensors 238 of recipient device 208, such as a user smiling, laughing, nodding, uttering an affirmative statement (e.g., “yes,” “confirm,” “ok,” “next”), or otherwise reacting positively to the content. Negative feedback may also be based on attributes of sensor data from sensors 238 of recipient device 208, such as a user frowning, crying, shaking his head (e.g., making a “no” gesture), uttering a negative statement (e.g., “no,” “reject,” “bad,” “not this”), or otherwise reacting negatively to the content. In some examples, feedback engine 234 may identify a reaction of recipient device 208 to output data transmitted to recipient device 208 as positive feedback. For example, if recipient device 208 includes a vehicle, feedback engine 234 may detect a reaction of the vehicle to output data transmitted to recipient device 208 (e.g., a change to a current route and / or a planned route) as positive feedback.

[0089] In some examples, the feedback engine 234 provides feedback as training data to one or more ML systems of the collaborative augmented sensing system 200 to update one or more trained ML models 232 of the collaborative augmented sensing system 200. For example, the feedback engine 234 may provide feedback as training data to the ML system and / or the trained ML model 232 to update the training of the object tracking engine 228, the data fusion engine 230, the alarm engine 244, or a combination thereof. Positive feedback may be used to reinforce and / or enhance the weight associated with the output of the ML system and / or the trained ML model 232. Negative feedback may be used to weaken and / or remove the weight associated with the output of the ML system and / or the trained ML model 232.

[0090] In some examples, certain elements of the collaborative enhanced sensing system 200 (e.g., object tracking engine 228, data fusion engine 230, alarm engine 244, feedback engine 234, trained ML model 232, and / or traffic control engine 222) include software elements (such as instruction sets corresponding to programs) running on a processor (such as processor 1110 of computing system 1100, image processor 150, host processor 152, ISP 154, processor 210, processor 216, processor 224, processor 236, or a combination thereof). In some examples, these elements of the collaborative enhanced sensing system 200 include one or more hardware elements, such as a dedicated processor (e.g., processor 1110 of computing system 1100, image processor 150, host processor 152, ISP 154, processor 210, processor 216, processor 224, processor 236, or a combination thereof). In some examples, these elements of the collaborative enhanced sensing system 200 may include a combination of one or more software elements and one or more hardware elements.

[0091] In some examples, certain elements of collaborative augmented sensing system 200 (e.g., object tracking engine 228, data fusion engine 230, alert engine 244, feedback engine 234, trained ML model 232, and / or traffic control engine 222) are coupled to Figure 2206. For example, elements of server 206 (e.g., object tracking engine 228, data fusion engine 230, alarm engine 244, feedback engine 234, and / or trained ML model 232) may additionally or alternatively be located on and / or run on data source 202, TMC 204, recipient device 208, or a combination thereof. For example, in some examples, one of recipient devices 208 may generate an enhanced representation of its own environment based on data received by one of recipient devices 208 from data source 202 and / or TMC 204 and / or server 206. In some examples, a first of recipient devices 208 may generate an enhanced representation of the environment for a second of recipient devices 208 based on data received by the first of recipient devices 208 from data source 202 and / or TMC 204 and / or server 206. In some examples, one of the data sources 202 may generate an enhanced representation of the environment for one of the recipient devices 208 based on its own data and / or data it receives from other data sources 202 and / or TMC 204 and / or server 206. In some examples, TMC 204 may generate an enhanced representation of the environment for one of the recipient devices 208 based on its own data and / or data it receives from data sources 202 and / or server 206.

[0092] Figure 3 is a top view 300 illustrating an example of collaborative enhanced sensing in a road environment with multiple vehicles. Figure 3 The vehicles on the road environment of the top view 300 include vehicles 310, vehicles 320 and vehicles 330. Vehicles 310 and vehicles 330 usually travel in a first direction, while vehicle 320 usually travels in a second direction parallel to and opposite to the first direction. The road environment includes two parallel one-way lanes with a dotted line between them, usually indicating that driving across the dotted line is allowed. Vehicle 330 includes a camera 335 coupled to vehicle 330 (e.g., mounted thereon), the camera pointing to an area of ​​interest 350 of the environment including vehicle 320, so that the field of view (FOV) 340 of camera 335 includes the area of ​​interest 350 of the environment including vehicle 320. Vehicle 330 includes a camera 337 coupled to vehicle 330 (e.g., mounted thereon), the camera pointing to an area of ​​interest 352 of the environment including vehicle 310, so that the field of view (FOV) 342 of camera 337 includes the area of ​​interest 352 of the environment including vehicle 310. Vehicle 330 may be an example of one of data sources 202 . Camera 335 and camera 337 may be examples of sensor 212 of vehicle 330 .

[0093] Vehicle 310 is behind vehicle 330. Collaborative augmented sensing system 200 (e.g., server 206 and / or object tracking engine 228) may detect vehicle 310, for example, based on signals received from vehicle 310 itself (e.g., via V2X communication) and / or based on detection of vehicle 310 in area of ​​interest 352 (e.g., based on image data captured by camera 337). Collaborative augmented sensing system 200 (e.g., server 206, object tracking engine 228, and / or alert engine 244) may determine that planned route 315 of vehicle 310 includes an attempt to overtake vehicle 330 by merging left into another lane. The collaborative enhanced sensing system 200 may determine this based on: a signal received from the vehicle 310 itself indicating or suggesting the planned route 315 (e.g., via V2X communications), detection of a flashing left indicator light of the vehicle 310 (e.g., based on image data captured by the camera 337), detection of movement of the vehicle 310 to the left (e.g., based on image data captured by the camera 337), or a combination thereof.

[0094] The collaborative augmented sensing system 200 (e.g., the server 206 and / or the object tracking engine 228) may detect the vehicle 320, for example, based on signals received from the vehicle 320 itself (e.g., via V2X communications) and / or based on detection of the vehicle 320 in the area of ​​interest 350 (e.g., based on image data captured by the camera 335). The collaborative augmented sensing system 200 (e.g., the server 206, the object tracking engine 228, and / or the alert engine 244) may determine that the planned route 325 of the vehicle 320 includes continuing in its lane. The collaborative enhanced sensing system 200 may determine this based on: signals received from the vehicle 310 itself indicating or suggesting the planned route 315 (e.g., via V2X communications), detection of the current route of the vehicle 320 (e.g., based on image data captured by the camera 335) and extrapolation of continuation of that path, detection of obstacles or lack of other vehicles that the vehicle 320 wishes to overtake in the area of ​​interest 350 (e.g., based on image data captured by the camera 335), or a combination thereof.

[0095] The collaborative enhanced sensing system 200 (e.g., the server 206, the object tracking engine 228, the data fusion engine 230, and / or the alarm engine 244) can determine that the planned route 315 of the vehicle 310 and the planned route 325 of the vehicle 320 may collide by extending the planned route 315 of the vehicle 310 and the planned route 325 of the vehicle 320 forward, even when the sensor data from the camera 335 and the sensor data from the camera 337 do not cover all roadside environments. Assuming that the vehicle 310 follows the planned route 315 and the vehicle 320 follows the planned route 325, the location of the possible collision 370 between the vehicle 310 and the vehicle 320 is illustrated with a dotted line. The collaborative enhanced sensing system 200 (e.g., the server 206, the object tracking engine 228, the data fusion engine 230, and / or the alarm engine 244) can generate an alarm 345 and transmit the alarm 345 to the vehicle 310. The alarm 345 can indicate, for example, that the planned route 315 may lead to a possible collision 370. Alert 345 may represent an enhanced representation of the environment because alert 345 notifies vehicle 310 of things (vehicle 320, its planned route 325, and the resulting possible collision 370) that would otherwise be difficult (or impossible) to detect from the perspective of vehicle 310 due to the presence of vehicle 330 blocking vehicle 310's view of vehicle 320. Alert 345 may indicate a risk of collision, such as possible collision 370. Because vehicle 310 is receiving alert 345, vehicle 310 may be an example of recipient device 208.

[0096] Alert 345 Figure 3 345, is illustrated as including a visual representation of vehicle 320, vehicle 310, vehicle 330, planned route 315, planned route 325, and possible collision 370. The visual representation may be generated using data fusion engine 230 and / or alert engine 244, and may be included as part of alert 345 (e.g., as part of an enhanced representation of the environment provided to vehicle 310). Vehicle 310 may include output device 242, such as a display, which may display alert 345, play an associated sound (e.g., a siren), actuate a haptic feedback actuator to provide haptic feedback (e.g., provided to a driver of vehicle 310 in a steering wheel), or a combination thereof.

[0097] In some examples, portions of collaborative enhanced sensing system 200 (e.g., server 206, object tracking engine 228, data fusion engine 230, and / or alert engine 244) may be housed in vehicle 330, vehicle 310, vehicle 320, and / or Figure 3For example, in the illustrative example, vehicle 330 may act as both one of data sources 202 and one of servers 206, in which case vehicle 330 may generate alert 345 for the vehicle and transmit alert 345 to vehicle 310.

[0098] Figure 4 400 is a top view illustrating an example of a device involved in wireless communication. The wireless communication may include, for example, sidelink communication, V2X communication, or other device-to-device (D2D) communication. The communication may be based on a sidelink slot structure. For example, a transmitting UE 402 may send a transmission (Tx) 414, for example, including a control channel and / or a corresponding data channel, which may be received by receiving UEs 404, 406, 408 using their respective receivers (e.g., in Figure 4 The control channel may include information for decoding the data channel and may also be used by the receiving device to avoid interference by avoiding transmission on the occupied resources during data transmission. The number of transmit time intervals (TTIs) and resource blocks (RBs) that the data transmission will occupy may be indicated in a control message from the transmitting device. UEs 402, 404, 406, 408 are each capable of operating as transmitting devices in addition to operating as receiving devices. Therefore, UEs 406, 408 are illustrated as transmitting transmissions 416, 420. Transmissions 414, 416, 420 (and 418 through RSU 407) may be broadcast or multicast to nearby devices. For example, UE 414 may transmit communications intended to be received by other UEs within range 401 of UE 414. Additionally and / or alternatively, the RSU 407 may receive transmissions from and / or transmit transmissions 418 to the UE 402, 404, 406, 408. The UE 402, 404, 406, 408, or the RSU 407 may include a detection component (e.g., for error detection and / or correction, such as in a hybrid automatic repeat request (HARQ)). In some examples, the UE 402, 404, 406, 408, and / or the RSU 407 may also receive a basic safety message (BSM). In some examples, the UE 402, 404, 406, 408, and / or the RSU 407 may also include a mitigation component.

[0099] In some examples, UE 402, UE 404, UE 406, UE 408, and / or RSU 407 may be examples of data source 202. For example, transmission 414, transmission 416, transmission 418, and / or transmission 420 may include sensor data captured by sensor 212 of UE 402, UE 404, UE 406, UE 408, and / or RSU 407. In some examples, RSU 407 may be an example of TMC 204. For example, transmission 418 may include sensor data captured by sensor 218 of RSU 407 and / or traffic data from traffic control engine 222 of RSU 407. In some examples, UE 402, UE 404, UE 406, UE 408, and / or RSU 407 may be examples of server 206. For example, transmission 414, transmission 416, transmission 418, and / or transmission 420 may include object tracking data from object tracking engine 228, output data from data fusion engine 230 (e.g., including an enhanced representation of the environment for recipient device 208), and / or alert data generated by alert engine 244. In some examples, UE 402, UE 404, UE 406, UE 408, and / or RSU 407 may be examples of recipient device 208 and may therefore receive transmissions (e.g., transmission 414, transmission 416, transmission 418, and / or transmission 420) to be output using output device 242.

[0100] Figure 5A 500 is a top view illustrating an example of sensor sharing for a cooperative autonomous driving system based on detection of non-vehicle-to-everything (non-V2X) vehicles. In wireless communications (such as V2X communications), a V2X entity may perform sensor sharing with other V2X entities to achieve cooperative autonomous driving. For example, referring to Figure 5A 500, a host vehicle (HV) 502 may detect multiple objects within its environment. For example, at block 532, the HV 502 may detect the presence of a non-V2X entity (NV) 506. The HV 502 may notify other entities (such as a first remote vehicle (RV1) 504 or a roadside unit (RSU) 508) of the presence of the NV 506, provided that the RV1 504 and / or RSU 508 itself cannot detect the NV 506. The HV 502 notifies the RV1 504 and / or RSU 508 that the NV 506 is a sharing of sensor information.

[0101] Figure 5B FIG. 5 is a top view illustrating an example of sensor sharing for a cooperative autonomous driving system based on obstacle detection. Figure 5B508, the HV 502 may detect a physical obstacle 512 (such as a pothole, debris, or an object that may be an obstacle in the path of the HV 502 and / or RV1 504 that has not yet been detected by the RV1 504 and / or RSU 508). The HV 502 may notify the RV1 and / or RSU 508 of the obstacle 512 so that the obstacle 512 can be avoided.

[0102] Figure 5C FIG. 5 is a top view illustrating an example of sensor sharing for a cooperative automated driving system based on vulnerable road user (VRU) detection. Figure 5C In the overhead view 520 , the HV 502 may detect the presence of a vulnerable road user (VRU) 522 and may share the detection of the VRU 522 with the RV1 504 and the RSU 508 in instances where the RSU 508 and / or RV1 504 may not be able to detect the VRU 522 .

[0103] Figure 5D 5 is a swim lane diagram 530 illustrating an example of sensor sharing for a collaborative automated driving system based on the transmission of a sensor data sharing message (SDSM) 534. Figure 5D In the swim lane diagram 530, the HV may send a sensor data sharing message (SDSM) 534 to the RV and / or RSU to share the detection of the entity when detecting a nearby entity (e.g., NV, VRU, obstacle). The SDSM 534 may be a broadcast message so that any receiving device near the HV may receive the message. In some cases, the shared information may be relayed to other entities, such as the RV.

[0104] In some examples, the HV 502, RV1 504, NV 506, RSU 508, obstacle 512, and / or VRU 522 can be examples of data sources 202. For example, the SDSM 534 can include sensor data captured by sensors 212 of the HV 502, RV1 504, NV 506, RSU 508, obstacle 512, and / or VRU 522. In some examples, the RSU 508 can be an example of the TMC 204. For example, the SDSM 534 can include sensor data captured by sensors 218 of the RSU 508 and / or traffic data from the traffic control engine 222 of the RSU 508. In some examples, the HV 502, RV1 504, NV 506, RSU 508, obstacle 512, and / or VRU 522 can be examples of the server 206. For example, the SDSM 534 may include object tracking data from the object tracking engine 228, output data from the data fusion engine 230 (e.g., including an enhanced representation of the environment for the recipient device 208), and / or alarm data generated by the alarm engine 244. In some examples, the RV1 504 and / or the RSU 508 may be examples of the recipient device 208 and may therefore receive transmissions (e.g., the SDSM 534) that may be output using the output device 242 of the RV1 504 and / or the RSU 508.

[0105] Figure 6 FIG. 600 is a conceptual diagram illustrating an example of sensor sharing for a cooperative autonomous driving system. Figure 6 600, HV 602 may detect the presence of NV 606 and / or VRU 622. HV 602 may broadcast SDSM 610 to RSU 608 to report the detection of NV 606 and / or VRU 622. RSU 608 may relay SDSM 610 received from HV 602 to a remote vehicle so that the remote vehicle is aware of the presence of NV 606 and / or VRU 622. For example, RSU 608 may send SDSM 612 to RV1 604, wherein SDSM 612 includes information related to the detection of NV 606 and / or VRU 622.

[0106] In some examples, HV 602, RV1 604, NV 605, RSU 608, and / or VRU 622 may be examples of data source 202. For example, SDSM 610 and / or SDSM 612 may include sensor data captured by sensors 212 of HV 602, RV1 604, NV 605, RSU 608, and / or VRU 622. FIG. 5A to FIG. 5D as well as Figure 6As indicated by the illustrated and discussed detections (e.g., detection of NVs, RVs, obstacles, and / or VRUs), in some cases, the object tracking engine 228 or portions thereof may be located in the data source 202. In some examples, the RSU 608 may be an example of the TMC 204. For example, the SDSM 610 and / or the SDSM 612 may include sensor data captured by the sensors 218 of the RSU 608 and / or traffic data from the traffic control engine 222 of the RSU 608. In some examples, the HV 602, the RV1 604, the NV 605, the RSU 608, and / or the VRU 622 may be examples of the server 206. For example, the SDSM 610 and / or the SDSM 612 may include object tracking data from the object tracking engine 228, output data from the data fusion engine 230 (e.g., including an enhanced representation of the environment for the recipient device 208), and / or alarm data generated by the alarm engine 244. In some examples, RV1 604 and / or RSU 608 may be examples of a recipient device 208 and may therefore receive transmissions (eg, SDSM 610 and / or SDSM 612 ) that may be output using an output device 242 of RV1 604 and / or RSU 608 .

[0107] Figure 7 is a top view 700 illustrating an example of collaborative enhanced sensing in a road environment with multiple vehicles. Compass 790 provides Figure 7 The MEC 740 is exemplified as Figure 7 The MEC 740 may be an example of a server 206 . Figure 7 The road environment also includes an eNodeB or gNodeB 745, an RSU 750, a camera 760 on a building, and a TMC 755. The RSU 750 and the camera 760 may represent examples of a data source 202 and / or its sensor 212. The RSU 750 and the camera 760 are illustrated using arrows as transmitting data (e.g., sensor data) to the MEC 740. The dashed line extending from the camera 760 represents the field of view of the sensor 212 (e.g., an image sensor) of the camera 760.

[0108] Figure 7The vehicles in the road environment include vehicle 705 heading eastward toward the intersection, vehicle 710 heading southward toward the same intersection, vehicle 715 heading westward toward the same intersection, vehicle 720 (truck) heading eastward toward the same intersection, and vehicle 725 (truck) heading southward toward the same intersection. Each of these vehicles (vehicle 705, vehicle 710, vehicle 715, vehicle 720, and / or vehicle 725) can be an example of one of the data sources 202 and / or one of the receiving devices 208, as indicated by the bidirectional arrows connecting each of the vehicles to the MEC 740. The dotted lines extending diagonally from the respective fronts of each of the vehicles represent the fields of view of the sensors 212 (e.g., image sensors and / or depth sensors) of each of the vehicles, respectively.

[0109] The TMC 755 is illustrated at an intersection. The TMC 755 is an example of the TMC 204 and includes a traffic light (e.g., part of the traffic control engine 222 of the TMC 755) and a camera (e.g., one of the sensors 218 of the TMC 755). The dashed circle around the TMC 755 represents the field of view of the camera of the TMC 755. The RSU 750 and the TMC 755 are communicatively coupled, wherein the TMC 755 transmits data to the RSU 750 and / or to the MEC 740 through the RSU 750. The RSU 750 and / or the TMC 755 may be examples of the TMC 204 and may transmit data (e.g., sensor data and / or traffic data) to the MEC 740. The eNodeB or gNodeB 745 may be used to transmit data between a node (e.g., a vehicle, an RSU 750, a TMC 755, and / or a camera 760) and the MEC 740. The eNodeB or gNodeB 745 may represent a portion of a server 206, a data source 202 (e.g., as part of a cellular signal-based positioning system), a recipient device 208 (e.g., transmitting information to other recipient devices 208), or a combination thereof.

[0110] Figure 8 is a top view 800 illustrating an example of collaborative augmented sensing in a road environment providing different augmented views to different vehicles. Compass 890 provides Figure 8 The MEC 840 is exemplified as Figure 8 The MEC 840 may be an example of a server 206. Figure 8The road environment of FIG. 840 also includes an eNodeB or gNodeB 845, an RSU 850, a camera 860 on a building, and a TMC 855. The RSU 850 and the camera 860 may represent examples of a data source 202 and / or its sensor 212. The RSU 850 and the camera 860 are illustrated using arrows as transmitting data (e.g., sensor data) to the MEC 840. The dashed line extending from the camera 860 represents the field of view of the sensor 212 (e.g., an image sensor) of the camera 860.

[0111] Figure 8 The vehicles in the road environment include vehicle 805 heading south toward the intersection, vehicle 810 heading west toward the same intersection, vehicle 820 heading west toward the same intersection, vehicle 830 (truck) heading west toward the same intersection, and vehicle 835 (truck) heading west toward the same intersection. Each of these vehicles (vehicle 805, vehicle 810, vehicle 820, vehicle 830, and / or vehicle 835) can be an example of one of the data sources 202 and / or one of the receiving devices 208, as indicated by the bidirectional arrows connecting each of the vehicles to the MEC 840. The dotted lines extending diagonally from the respective fronts of each of the vehicles respectively represent the fields of view of the sensors 212 (e.g., image sensors and / or depth sensors) of each of the vehicles.

[0112] The dashed arrow extending from vehicle 810 represents a planned route 815 for vehicle 810. The planned route 815 for vehicle 810 indicates that vehicle 810 is planning to turn right at the intersection. The dashed arrow extending from vehicle 820 represents a planned route 825 for vehicle 820. The planned route 825 for vehicle 820 indicates that vehicle 820 is planning to turn left at the intersection. MEC 840 may determine these planned movements based on communications from vehicle 310 and / or vehicle 320 (e.g., V2X communications), based on detection of indication signals such as turn signals used by vehicle 310 and / or vehicle 320 (e.g., from the field of view of camera 860 and / or other cameras), or a combination thereof.

[0113] In response to determining the planned movement (e.g., the planned route 815 of the vehicle 810 and / or the planned route 825 of the vehicle 820), the MEC 840 generates an enhanced representation 870 of the environment for the vehicle 810 and transmits the enhanced representation 870 of the environment for the vehicle 810 to the vehicle 810. The MEC 840 generates an enhanced representation 875 of the environment for the vehicle 820 and transmits the enhanced representation 875 of the environment for the vehicle 820 to the vehicle 820. Because the vehicle 810 and the vehicle 820 are similarly positioned and both are traveling in the same direction (west) along the same road toward the same intersection, the enhanced representation 870 of the environment for the vehicle 810 and the enhanced representation 875 of the environment for the vehicle 820 are similar and each includes a representation of a respective area of ​​the environment (including the road) located ahead of the respective vehicles in that direction (west), including at least a portion of the intersection. However, because the planned movements of vehicle 810 and vehicle 820 are different, i.e., vehicle 810 is planning to turn right (to the north) and vehicle 820 is planning to turn left (to the south), the augmented representation 870 of the environment for vehicle 810 and the augmented representation 875 of the environment for vehicle 820 are different. For example, the augmented representation 870 of the environment for vehicle 810 includes an area of ​​the environment that is further north (the direction that vehicle 310 is turning) and includes a portion of the street north of the intersection, while the augmented representation 875 of the environment for vehicle 820 includes an area of ​​the environment that is further south (the direction that vehicle 320 is turning) and includes a portion of the street south of the intersection.

[0114] The TMC 855 is illustrated at an intersection. The TMC 855 is an example of a TMC 204 and includes a stop sign (e.g., part of the traffic control engine 222 of the TMC 855) and a camera (e.g., one of the sensors 218 of the TMC 855). The dashed circle around the TMC 855 represents the field of view of the camera of the TMC 855. The field of view of the camera of the TMC 855 overlaps with the augmented representation 875 of the environment for the vehicle 820, and thus sensor data from the camera of the TMC 855 can be used to generate the augmented representation 875 of the environment for the vehicle 820. The RSU 850 and the TMC 855 are communicatively coupled, wherein the TMC 855 transmits data to the RSU 850 and / or to the MEC 840 through the RSU 850. The RSU 850 and / or TMC 855 may be examples of the TMC 204 and may transmit data (e.g., sensor data and / or traffic data) to the MEC 840. The eNodeB or gNodeB 845 may be used to transmit data between a node (e.g., a vehicle, an RSU 850, a TMC 855, and / or a camera 860) and the MEC 840. The eNodeB or gNodeB 845 may represent a portion of the server 206, the data source 202 (e.g., as part of a cellular signal-based positioning system), a recipient device 208 (e.g., transmitting information to other recipient devices 208), or a combination thereof.

[0115] Fig. 9 is a block diagram illustrating an example of a neural network (NN) 900 that can be used for collaborative augmented sensing operations. The neural network 900 may include any type of deep network, such as a convolutional neural network (CNN), an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), a generative adversarial network (GAN), and / or other types of neural networks. The neural network 900 may be an example of one of the one or more trained machine learning models 232 of the collaborative augmented sensing system 200, which may be used by the object tracking engine 228, the data fusion engine 230, the alarm engine 244, or a combination thereof.

[0116] Input layer 910 of neural network 900 includes input data. The input data of input layer 910 may include data representing pixels of one or more input image frames. In some examples, the input data of input layer 910 includes data representing pixels of image data captured by one or more image sensors. Examples of image sensors include: sensor 212, sensor 218, sensor 238, image sensor of camera 335, image sensor of camera 337, Figure 4an image sensor of any one of the UEs, an image sensor of the RSU 407, an image sensor of the HV 502, an image sensor of the RSU 508, an image sensor of the HV 502, an image sensor of the RSU 508, an image sensor of the HV 602, an image sensor of the RSU 608, an image sensor of the camera 760, an image sensor of the TMC 755, an image sensor of the RSU 750, an image sensor of the eNodeB or gNodeB 745, Figure 7 an image sensor of any one of the vehicles, an image sensor of the camera 860, an image sensor of the TMC 855, an image sensor of the RSU 850, an image sensor of the eNodeB or gNodeB 845, Figure 8 1005, an image sensor of any of the vehicles, an image sensor capturing at least a portion of the sensor data of operation 1005, an input device 1145, or a combination thereof. In some examples, the input data of the input layer 910 includes depth data captured by a depth sensor and / or other sensor data captured by other sensor types described herein. Any of the examples of image sensors listed above may additionally or alternatively include a depth sensor and / or other sensor types described herein. In some examples, the input data of the input layer 910 includes metadata associated with the sensor data. In some examples, the input data of the input layer 910 additionally or alternatively includes traffic data (e.g., from the traffic control engine 222 of the TMC 204), object tracking data from the object tracking engine 228, output data from the data fusion engine 230 (e.g., including an enhanced representation of the environment for the recipient device 208), or a combination thereof. The input data may include input data (e.g., sensor data) transmitted using an SDSM (such as SDSM 534, SDSM 610, and / or SDSM 612).

[0117] The image may include image data from an image sensor, including raw pixel data (including monochrome per pixel based on, for example, a Bayer color filter) or processed pixel values ​​(e.g., RGB pixels of an RGB image). Neural network 900 includes a plurality of hidden layers 912A, 912B through 912N. Hidden layers 912A, 912B through 912N include "N" number of hidden layers, where "N" is an integer greater than or equal to one. The plurality of hidden layers may include as many layers as required for a given application. Neural network 900 also includes an output layer 914 that provides output resulting from the processing performed by hidden layers 912A, 912B through 912N.

[0118] In some examples, the output layer 914 may provide output data. Examples of output data include object tracking data from the object tracking engine 228, output data from the data fusion engine 230 (e.g., including an enhanced representation of the environment for the recipient device 208), alarm data generated by the alarm engine 244 (e.g., alarm 345), output data transmitted by the MEC 740, output data transmitted by the MEC 840, an enhanced representation 870 of the environment for the vehicle 810, an enhanced representation 875 of the environment for the vehicle 820, an enhanced representation of the environment of the operation 1015, or a combination thereof. The output data may include output data transmitted using an SDSM (such as SDSM 534, SDSM 610, and / or SDSM 612).

[0119] Neural network 900 is a multi-layer neural network of interconnected filters. Each filter can be trained to learn features that represent input data. Information associated with these filters is shared between different layers, and each layer retains information as it processes it. In some cases, neural network 900 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back to itself. In some cases, network 900 may include a recurrent neural network, which may have loops that allow information to be carried across nodes when reading in input.

[0120] In some cases, information can be exchanged between layers by node-to-node interconnection between each layer. In some cases, the network may include a convolutional neural network, which may not link each node in a layer to each other node in the next layer. In a network in which information is exchanged between layers, the nodes of the input layer 910 may activate a set of nodes in the first hidden layer 912A. For example, as shown in the figure, each input node in the input nodes of the input layer 910 may be connected to each node in the nodes of the first hidden layer 912A. The nodes of the hidden layer may transform the information by applying an activation function (e.g., a filter) to the information of each input node. Then, the information derived from the transformation may be passed to the nodes of the next hidden layer 912B and may activate these nodes, which may perform their own specified functions. Example functions include convolution functions, reduction, enlargement, data transformation and / or any other suitable functions. Then, the output of the hidden layer 912B may activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 912N may activate one or more nodes of the output layer 914, which provides a processed output image. In some cases, although a node in neural network 900 (e.g., node 916) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0121] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 900. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have a tunable digital weight that can be tuned (e.g., based on a training data set), thereby allowing neural network 900 to adapt to the input and be able to learn as more and more data is processed.

[0122] The neural network 900 is pre-trained to process features of data from the input layer 910 using different hidden layers 912A, 912B to 912N to provide an output through the output layer 914.

[0123] Fig.10 is a flow chart illustrating a collaborative enhanced sensing process 1000. The collaborative enhanced sensing process 1000 may be performed by a collaborative enhanced sensing system. In some examples, the collaborative enhanced sensing system may include, for example, an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an image processor 150, an ISP 154, a host processor 152, a collaborative enhanced sensing system 200, a data source 202, a TMC 204, a server 206, a receiving device 208, a vehicle 310, a vehicle 320, a vehicle 330, a UE 402, a UE 404, a UE 406, a UE 408, an RSU 407, a HV 502, a RV1 504, a NV 506, a RSU 508, an obstacle 512, a VRU 522, a HV 602, a RV1 604, a NV 605, a RSU 608, a VRU 622, a vehicle 705, a vehicle 710, a vehicle 715, a vehicle 720, a vehicle 725, a MEC 740, eNodeB or gNodeB 745, RSU 750, TMC 755, camera 760, vehicle 805, vehicle 810, vehicle 820, vehicle 830, vehicle 835, MEC 840, eNodeB or gNodeB 845, RSU 850, TMC 855, camera 860, NN 900, computing system 1100, processor 1110, any other system or device described herein, any component thereof, or a combination thereof.

[0124] At operation 1005, the collaborative enhanced sensing system is configured and can obtain sensor data of the environment from one or more sensors. The sensor data includes multiple different representations of one or more parts of the environment from multiple perspectives. The sensor data can be captured by one or more sensors. In some examples, two of the different representations are captured by different sensors in one or more sensors. In some examples, two of the different representations are captured by the same sensor in one or more sensors. In some examples, the collaborative enhanced sensing system is configured and can use a transceiver, a receiver and / or a communication interface 1140 to obtain sensor data of the environment from one or more sensors of at least one device. In some examples, the collaborative enhanced sensing system is configured and can obtain sensor data of the environment from one or more sensors of at least one device. In some examples, the at least one device is non-co-located with the collaborative enhanced sensing system. In some examples, the collaborative enhanced sensing system is configured and can obtain sensor data of the environment from multiple sensors from multiple devices. In some examples, the multiple devices are non-co-located with each other.

[0125] Examples of the one or more sensors include: sensor 212, sensor 218, sensor 238, an image sensor of camera 335, an image sensor of camera 337, Figure 4 a sensor of any one of the UEs, a sensor of the RSU 407, a sensor of the HV 502, a sensor of the RSU 508, a sensor of the HV 502, a sensor of the RSU 508, a sensor of the HV 602, a sensor of the RSU 608, an image sensor of the camera 760, a sensor of the TMC 755, a sensor of the RSU 750, a sensor of the eNodeB or gNodeB 745, Figure 7 a sensor of any of the vehicles, an image sensor of the camera 860, a sensor of the TMC 855, a sensor of the RSU 850, a sensor of the eNodeB or gNodeB 845, Figure 8A sensor of any of the vehicles of the invention, a sensor that captures at least a portion of the sensor data of operation 1005, an input device 1145, or a combination thereof. In some examples, the collaborative enhanced sensing system includes a sensor connector and / or a sensor interface and / or a transceiver that couples and / or connects one or more sensors to the rest of the collaborative enhanced sensing system (e.g., a processor and / or memory including the collaborative enhanced sensing system). In some examples, the collaborative enhanced sensing system receives sensor data from one or more sensors by: receiving sensor data from a sensor connector and / or a sensor interface and / or a transceiver, receiving sensor data through a sensor connector and / or a sensor interface and / or a transceiver, and / or receiving sensor data using a sensor connector and / or a sensor interface and / or a transceiver. In some examples, the collaborative enhanced sensing system includes at least a subset of one or more sensors. The sensor connector and / or the sensor interface and / or the transceiver may include one or more wired and / or wireless interfaces, connectors and / or transceivers, such as those discussed with respect to I / O 156, I / O 160, and / or communication interface 1140.

[0126] In some examples, at least some different portions of the sensor data and / or at least some different portions of the multiple different representations of one or more portions of the environment from multiple perspectives may be captured by different sensors, which may be on different devices and / or positioned at different locations within the environment. Figure 2 In the context of , different representations may be captured by different data sources 202 and / or by different TMCs 204. Figure 3 In the context of , camera 337 and camera 335 may capture different representations from different perspectives. Figure 4 In the context of , each of the UEs may capture a different representation. FIG. 5A to FIG. 5C In the context of , each of the HV 502, RV1 504, NV 506, RSU 508, obstacle 512, and / or VRU 522 may capture different portions of sensor data and / or different representations of one or more portions of the environment from multiple perspectives. Figure 6 In the context of , each of the HV 602, RV1 604, NV 606, RSU, and / or VRU 622 may capture different portions of sensor data and / or different representations of one or more portions of the environment from multiple perspectives. Figure 7 In the context of , each of vehicle 705, vehicle 710, vehicle 715, vehicle 720, vehicle 725, RSU 750, TMC 755, and / or camera 760 may capture different portions of sensor data and / or different representations of one or more portions of the environment from multiple perspectives. Figure 8 In the context of , each of vehicle 805, vehicle 810, vehicle 820, vehicle 830, vehicle 835, RSU 850, TMC 855, and / or camera 860 may capture different portions of sensor data and / or different representations of one or more portions of the environment from multiple perspectives. Fig.10 In the context of , each input device 1145 of multiple input devices of one or more computing systems 1100 can capture a different portion of sensor data and / or a different representation of one or more portions of the environment from multiple perspectives.

[0127] Examples of environments include the scene 110, the environment in which the data source 202 is located, the environment in which the TMC 204 is located, the environment in which the server 206 is located, the environment in which the recipient device 208 is located, Figure 3 The environment, FOV 340, FOV 342, region of interest 350, region of interest 352, Figure 4 The environment shown in the example, FIG. 5A to FIG. 5C The environment shown in the example, Figure 6 The environment shown in the example, Figure 7 The environment shown in the example, Figure 8 The environment illustrated in, the portion of the environment included in the enhanced representation of the environment 870, the portion of the environment included in the enhanced representation of the environment 875, the environment in which the computing system 1100 is located, another environment or area or location described herein, or a combination thereof.

[0128] Examples of different devices that one or more sensors may be a part of include image capture and processing system 100, data source 202, TMC 204, server 206, recipient device 208, vehicle 310, vehicle 320, vehicle 330, UE 402, UE 404, UE 406, UE 408, RSU 407, HV 502, RV1 504, NV 506, RSU 508, obstacle 512, VRU 522, HV 602, RV1 604, NV 605, RSU 608, VRU 622, vehicle 705, vehicle 710, vehicle 715, vehicle 720, vehicle 725, MEC 740, eNodeB or gNodeB 745, RSU 750, and vehicle 751. 750, TMC 755, camera 760, vehicle 805, vehicle 810, vehicle 820, vehicle 830, vehicle 835, MEC 840, eNodeB or gNodeB 845, RSU 850, TMC 855, camera 860, computing system 1100, or a combination thereof. In some examples, the devices may be non-co-located (e.g., at least a certain distance apart, not co-located, not in a shared location, and / or not in a shared area) relative to each other and / or relative to the collaborative enhanced sensing system. In some examples, a first device in the devices may be co-located with at least a second device in the devices and / or with the collaborative enhanced sensing system.

[0129] In some examples, the collaborative enhanced sensing system is configured to and can identify vehicles in the environment. Examples of vehicles include: a vehicle including the image capture and processing system 100, a vehicle including the data source 202, a vehicle including the recipient device 208, vehicle 310, vehicle 320, vehicle 330, UE 402, UE 404, UE 406, UE 408, HV 502, RV1 504, NV 506, obstacle 512, VRU 522, HV 602, RV1 604, NV 605, VRU 622, vehicle 705, vehicle 710, vehicle 715, vehicle 720, vehicle 725, vehicle 805, vehicle 810, vehicle 820, vehicle 830, vehicle 835, a vehicle including NN 900, a vehicle including the computing system 1100, any other vehicle described herein, or a combination thereof.

[0130] In some examples, identifying a vehicle in the environment may include, for example, identifying a vehicle in the sensor data using object tracking engine 228. In some examples, identifying a vehicle in the environment includes receiving information indicating that the vehicle is present in the environment, such as communications from a vehicle computing system of the vehicle transmitted by the vehicle computing system of the vehicle when the vehicle is in the environment. In some examples, the collaboratively augmented sensing system is a vehicle. In some examples, identifying a vehicle in the environment may include identifying the environment in which the vehicle is located.

[0131] In some examples, identifying vehicles in the environment may include: identifying vehicles with limited field of view, identifying vehicles with at least partially blocked field of view, identifying vehicles in dangerous situations (e.g., about to collide or other incident), identifying vehicles that are planning to perform dangerous maneuvers (e.g., that may result in a collision or other incident), or a combination thereof. Examples include, for example, identifying vehicle 310 because planned route 315 is dangerous, identifying HV 502 and / or NV 506 because HV 502 and / or NV 506 are close and / or NV 506 is a blind spot for HV 502, identifying HV 502 and / or RV1 504 because of obstacle 512, identifying HV 502 and / or RV1 504 because of VRU 522, identifying HV 602 and / or NV 606 because HV 602 and / or NV 606 are close and / or NV 606 is a blind spot for HV 602, identifying HV 602 and / or NV 606 and / or RV1 504 because of VRU 622. 604, identifying vehicle 705 due to its limited FOV of the environment that is blocked by vehicle 720, identifying vehicle 710 due to its limited FOV of the environment that is blocked by vehicle 725, identifying vehicle 810 due to its limited FOV of the environment that is included in the enhanced representation 870 of the environment (blocked by vehicle 830) and / or identifying vehicle 810 due to planned route 815, identifying vehicle 820 due to its limited FOV of the environment that is included in the enhanced representation 875 of the environment (blocked by vehicle 835) and / or identifying vehicle 820 due to planned route 825, identifying vehicles using NN 900, identifying vehicles using object tracking engine 228, or a combination thereof.

[0132] In some examples, receiving sensor data of the environment as in operation 1005 includes receiving at least a portion of sensor data including at least one of a plurality of different representations from a second vehicle in the environment, from a roadside unit (RSU) in the environment, from a camera in the environment, from a depth sensor in the environment, or a combination thereof. Examples of second vehicles include any of the examples of vehicles listed above. Examples of RSUs include data source 202, TMC 204, server 206, RSU 407, RSU 508, RSU 608, RSU 750, TMC 755, RSU 850, TMC 855, or a combination thereof. Examples of cameras include any of the examples of one or more sensors listed above that may include a camera and / or an image sensor.

[0133] In some examples, the multiple different representations include image data captured using at least one image sensor, such as the image sensor 130 of the image capture and processing system 100 and / or an image sensor of one of the cameras illustrated and / or described herein. In some examples, the multiple different representations include depth data captured using at least one depth sensor, such as one or more LIDAR sensors, one or more RADAR sensors, one or more SODAR sensors, one or more SONAR sensors, one or more ToF sensors, one or more structured light sensors, one or more ultrasonic sensors, one or more rangefinders, or a combination thereof. In some examples, the sensor data may include positioning data captured by a positioning receiver, such as a GNSS receiver, an IMU, an accelerometer, a gyroscope, a gyrometer, a barometer, an altimeter, or a combination thereof. The positioning receiver may be a positioning receiver of a vehicle and / or a positioning receiver coupled to a vehicle.

[0134] In some examples, at least one of the plurality of different representations is received from a vehicle. In some examples, at least one of the plurality of different representations is received from a first vehicle other than the vehicle. In some examples, at least one of the plurality of different representations is received from a second vehicle other than the vehicle. In some examples, at least one of the plurality of different representations is received from an RSU. In some examples, at least one of the plurality of different representations is received from a second RSU. In some examples, at least one of the plurality of different representations is received from an MEC. In some examples, at least one of the plurality of different representations is received from a TMC 204.

[0135] At operation 1010, the collaborative augmented sensing system is configured to and can identify a subset of multiple different representations of one or more parts of the environment based on a context corresponding to a vehicle in the environment. The context includes at least the location of the vehicle in the environment. The identified subset of multiple different representations of one or more parts of the environment can be a subset of multiple different representations of one or more parts of the environment that are most relevant to the vehicle, including parts of the environment and / or the perspective of the environment that are difficult to sense from the perspective of the vehicle (e.g., due to obstacles, environmental scene conditions, sensor conditions, and / or any other reasons discussed further below), including parts of the environment and / or the perspective of the environment that will be included in the enhanced representation of the environment, or a combination thereof. In some examples, the subset of multiple different representations of one or more parts of the environment is identified using the object tracking engine 228, the data fusion engine 230, the alarm engine 244, the trained ML model 232, or a combination thereof. In some examples, the identification of the subset of multiple different representations of one or more parts of the environment is performed automatically by the collaborative augmented sensing system. In some examples, identification of a subset of the plurality of different representations of one or more portions of the environment is automatically performed based on a selection (e.g., by a user through a user interface) and / or a preference (e.g., associated with a user.) The selection and / or preference may be associated with a user, such as an operator of a vehicle, an operator of a second vehicle in the environment, an operator of data source 202, an operator of TMC 204, an operator of server 206, an operator of recipient device 208, an operator of at least a portion of a collaborative augmented sensing system, or a combination thereof.

[0136] In some examples, identifying a subset of the plurality of different representations based on the context includes identifying one or more representations of a portion of the environment that includes the location of the vehicle and / or is adjacent to the location of the vehicle and / or is within a predetermined range of the location of the vehicle. For example, vehicle 310 is included in the representation of the environment captured by camera 337 based on FOV 342, vehicle 320 is included in the representation of the environment captured by camera 335 based on FOV 340, UE 402-406 is included in the representation of the environment captured by RSU 407 based on range 401, various vehicles 502-506 and / or RSU 508 are included in the representation of the environment captured by various vehicles 502-506 and / or RSU 508, various vehicles 602-606 and / or RSU 608 are included in the representation of the environment captured by various vehicles 602-606 and / or RSU 608, vehicle 725 is included in the representation of the environment captured by vehicle 710 and / or in the representation of the environment captured by camera 760 and / or in the representation of the environment captured by vehicle 715, and vehicle 720 is included in the representation of the environment captured by vehicle 705 and / or in the representation of the environment captured by TMC 755, vehicle 715 is included in the representation of the environment captured by vehicle 720, vehicles 830-835 are included in the representation of the environment captured by vehicles 810-820 and / or in the representation of the environment captured by camera 860 and / or in the representation of the environment captured by vehicle 805, vehicles 810-820 are included in the representation of the environment captured by camera 860, vehicle 805 is included in the representation of the environment captured by vehicle 830, and so on.

[0137] In some examples, the context includes a direction of travel of a vehicle in the environment, and identifying the subset of the plurality of different representations based on the context includes identifying one or more representations of a portion of the environment that is in front of the vehicle in the direction of travel of the vehicle. For example, region of interest 350 in FOV 340 of camera 335 is in the direction of travel of vehicle 310 based on planned route 315, region of interest 352 in FOV 342 of camera 337 is in the direction of travel of vehicle 320 based on planned route 325, Figure 8 The intersection of and the northern road in the direction of travel of the vehicle 810 based on the planned route 815, Figure 8 The intersection and the southern road are in the direction of travel of the vehicle 820 based on the planned route 825, and so on.

[0138] In some examples, the context includes the speed of the vehicle in the environment, and the subset identifying multiple different representations based on the context includes one or more representations of the part of the environment located in front of the vehicle. The size of this part of the environment can be based on the speed of the vehicle. For example, the part of the environment included in the enhanced representation 870 of the environment for the vehicle 810 extends further in front of the vehicle 810 and on the right side than in other directions, because the speed of the vehicle 810 in these directions (as indicated by the planned route 815) is higher than the speed in other directions. Similarly, the part of the environment included in the enhanced representation 875 of the environment for the vehicle 820 extends further in front of the vehicle 820 and on the left side than in other directions, because the speed of the vehicle 820 in these directions (as indicated by the planned route 825) is higher than the speed in other directions.

[0139] In some examples, the context includes a planned movement of a vehicle in the environment, and identifying a subset of the plurality of different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle according to the planned movement of the vehicle. Examples of the planned movement of the vehicle include planned route 315, planned route 325, planned route 815, and / or planned route 825.

[0140] In some examples, the collaboratively enhanced sensing system is configured and can detect an indicator light of the vehicle in the sensor data, and determine the planned movement of the vehicle based on detecting the indicator light of the vehicle in the sensor data. For example, in some examples, the indicator light can be a turn indicator light, and the planned movement of the vehicle can be a turn, such as in planned route 315, planned route 815, and / or planned route 825. In some examples, the indicator light can be a brake light, and the planned movement of the vehicle can be a deceleration or a stop.

[0141] In some examples, the collaboratively enhanced sensing system is configured and can detect the current path of the vehicle based on the sensor data and determine the planned movement of the vehicle based on the current path of the vehicle. For example, planned route 315 is an extension of the current path of vehicle 310 combined with turn instructions, planned route 325 is an extension of the current path of vehicle 320, planned route 815 is an extension of the current path of vehicle 810 combined with turn instructions, and planned route 825 is an extension of the current path of vehicle 820 combined with turn instructions.

[0142] In some examples, the collaborative enhanced sensing system is configured and may receive an indication of a planned movement of the vehicle from the vehicle (e.g., from a vehicle computing device of the vehicle), and the planned movement of the vehicle may be part of a planned route of the vehicle. For example, the collaborative enhanced sensing system may receive an indication of planned route 315 from vehicle 310, may receive an indication of planned route 325 from vehicle 320, may receive an indication of planned route 815 from vehicle 810, and / or may receive an indication of planned route 825 from vehicle 820.

[0143] In some examples, the context includes objects in the environment other than the vehicle, such as a second vehicle, a pedestrian (e.g., VRU 522, VRU 622), a bicycle, a cyclist, a motorcycle, a motorcycle rider, a scooter, a scooter operator, a structure (e.g., a building, a lamp post, an RSU), a plant (e.g., a tree), an animal, a stone, a pothole, spillage on the road, debris, an obstacle 512, or a combination thereof. In some examples, identifying a subset of multiple different representations based on the context includes identifying one or more representations of a portion of the environment that includes the object. In some examples, the collaborative augmented sensing system is configured to and can identify the risk of a collision between the vehicle and an object in the environment based on one or more planned movements of the vehicle. The collaborative augmented sensing system can cause an alert based on an augmented representation of the environment to be output at the vehicle. The alert can indicate the risk of a collision between the vehicle and the object. Alert 345 is an example of an alert indicating the risk of a possible collision 370 between the vehicle 310 and the object (e.g., the second vehicle 320). The alert engine 244 can be used to generate the alert.

[0144] In some examples, the context includes the planned movement of objects in the environment other than the vehicle, such as a second vehicle, a pedestrian (e.g., VRU 522, VRU 622), a bicycle, a bicyclist, a motorcycle, a motorcycle rider, a scooter, a scooter operator, a structure, a plant, an animal, a stone, a pothole, spillage on the road, debris, an obstacle 512, or a combination thereof. In some examples, identifying a subset of multiple different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle according to the planned movement of the object. The alert 345 is an example of an alert indicating a risk of a possible collision 370 between the vehicle 310 and the object (e.g., the second vehicle 320) based on the planned movement of the object (e.g., the planned route 325 of the second vehicle 320). The alert can be generated using the alert engine 244. In some examples, the context can identify a pedestrian's walking path (e.g., a pedestrian is crossing a street or may be crossing a street), and the alert can indicate the risk of a collision between the vehicle and the pedestrian. In some examples, the collaborative augmented sensing system may receive an indication of the object's planned route from a device associated with the object, such as a mobile phone or other user device of the pedestrian. In some examples, the collaborative augmented sensing system may detect an indication of the object's planned route from a representation of the environment, such as a crosswalk detection light indicating that a pedestrian is currently crossing, or a crosswalk light indicating "go," "stop," or "accelerate."

[0145] In some examples, identifying a subset of the plurality of different representations based on the context includes identifying that a confidence level associated with each representation in the subset of the plurality of different representations exceeds a confidence level threshold. The confidence level of one of the plurality of different representations is associated with a confidence that the one of the plurality of different representations represents a portion of an environment that includes the vehicle and / or includes an area in front of the vehicle. In some examples, the portion of the environment that is in front of the vehicle includes the vehicle.

[0146] In some examples, the confidence level is based on a resolution of the one of the multiple different representations. For example, a higher resolution representation (e.g., a higher resolution image) may correspond to a higher confidence level because the representation of the vehicle in the higher resolution representation appears clearer and sharper, while the representation of the vehicle in a lower resolution representation (e.g., a lower resolution image) may appear more blurry or pixelated. In some examples, the confidence level is based on a lighting level in the one of the multiple different representations.

[0147] In some examples, the confidence level is based on object detection using one of a plurality of different representations. In some examples, the confidence level may be based on a confidence level associated with object detection and / or recognition and / or tracking by object tracking engine 228, a confidence level of data fusion associated with data fusion engine 230, a confidence level of an alert associated with alert engine 244, a confidence level associated with an output of trained ML model 232, a confidence level associated with an output of NN 900, a confidence level in data from a particular one of data sources 202, a confidence level in data from TMC 204, a confidence level associated with traffic control engine 222, or a combination thereof.

[0148] For example, a well-lit representation (e.g., an image that is neither overexposed nor underexposed) may correspond to a higher confidence level because the representation of the vehicle in the well-lit representation has clear lines and distinct colors, whereas the representation of the vehicle in a poorly lit representation (e.g., an overexposed and / or underexposed image) may appear too bright or too dark to clearly distinguish the clear lines and distinct colors.

[0149] In some examples, the collaborative enhanced sensing system is configured to and can receive traffic data from a traffic management controller. The context may include at least a portion of the traffic data. The portion of the traffic data may be associated with at least a portion of an environment including a vehicle. For example, the traffic data may include traffic data detected and / or reported by a data source 202, a TMC 204, a server 206, a recipient device 208, a traffic control engine 222, an RSU 407, an RSU 508, an RSU 608, an RSU 750, a TMC 755, an RSU 850, a TMC 855, any example of one or more sensors listed above, any traffic data described herein, or a combination thereof.

[0150] In some examples, identifying a subset of multiple different representations based on a context corresponding to a vehicle in the environment includes inputting sensor data and context into one or more trained machine learning (ML) models (e.g., trained ML model 232 and / or NN 900) that output indications of a subset of multiple different representations. For example, the trained ML model may be associated with object tracking engine 228, data fusion engine 230, traffic control engine 222, alarm engine 244, or a combination thereof. In some examples, the collaborative augmented sensing system is configured and may identify a reaction of a vehicle in response to receiving an augmented representation of the environment, and may use the context, the subset of multiple different representations, and / or the reaction as training data to update one or more trained machine learning models. For example, the reaction may be considered feedback to feedback engine 234 and may be used to train trained ML model 232.

[0151] At operation 1015, the collaborative augmented sensing system is configured and can cause an augmented representation of the environment for the vehicle to be generated based at least in part on merging a subset of the plurality of different representations. In some examples, the collaborative augmented sensing system is configured and can cause an augmented representation of the environment for the vehicle to be generated based at least in part on merging a subset of the plurality of different representations. In some examples, the collaborative augmented sensing system is configured and can cause the merged subset of the plurality of different representations to be transmitted to another computing system that generates an augmented representation of the environment for the vehicle based on the merged subset of the plurality of different representations. Examples of enhanced representations of the environment include an enhanced representation of the environment generated using data fusion engine 230, an enhanced representation of the environment generated using alert engine 244, an enhanced representation of the environment in alert 345 for vehicle 310 (e.g., due to a majority of the viewing angle and / or FOV of vehicle 310 being blocked by vehicle 330), an enhanced representation 870 of the environment for vehicle 810 (e.g., due to a majority of the viewing angle and / or FOV of vehicle 810 being blocked by vehicle 830), an enhanced representation 875 of the environment for vehicle 820 (e.g., due to a majority of the viewing angle and / or FOV of vehicle 820 being blocked by vehicle 835), and other enhanced representations of the environment described herein.

[0152] In some examples, the enhanced representation of the environment includes a representation of a portion of the environment that is difficult to sense from the perspective (e.g., field of view) of the vehicle (e.g., from the perspective of one or more sensors of the vehicle, from the perspective of the operator of the vehicle, from the perspective of the passenger of the vehicle, or a combination thereof). In some examples, the enhanced representation of the environment includes a representation of a portion of the environment that is impossible to sense from the perspective of the vehicle. In some examples, the enhanced representation of the environment includes a representation of a portion of the environment that is at least partially blocked and / or obscured (e.g., by an obstacle or obstruction) from the perspective of the vehicle. Examples of obstacles or obstructions include a second vehicle, a pedestrian (e.g., VRU 522, VRU 622), a bicycle, a bicycle rider, a motorcycle, a motorcycle rider, a scooter, a scooter operator, a structure (e.g., a building, a lamp post, an RSU), a plant (e.g., a tree), an animal, a stone, debris, an obstacle 512, fog, rain, smoke, snow, hail, weather conditions, environmental scene conditions, or a combination thereof. In some examples, the enhanced representation of the environment includes representations of portions of the environment that are difficult to sense from the perspective of the vehicle due to lighting conditions and / or illumination conditions, such as low or dim lighting that causes the environment to appear dim and / or underexposed (e.g., below a light or illumination level threshold), excessively bright lighting that causes the environment to appear washed out and / or overexposed (e.g., exceeding a light or illumination level threshold), lighting that casts shadows in the environment or on a sensor that affect the representation of portions of the environment, lighting that produces visual artifacts (e.g., lens flare or blur) on a view of the environment that affect the representation of portions of the environment, or combinations thereof. In some examples, the enhanced representation of the environment includes a representation of a portion of the environment that is difficult to sense from the perspective of the vehicle due to a sensor condition of the vehicle, such as because the sensor is defective or damaged, the sensor's wiring is faulty, the sensor is misaligned, occluded, or damaged has affected components associated with the sensor (e.g., a camera lens is blurry, dirty, fogged, scratched, cracked, broken, or a combination thereof), the sensor has a low resolution (e.g., a resolution below a threshold), the sensor has a low accuracy (e.g., an accuracy below a threshold), the sensor has a low quality (e.g., a quality below a threshold), or a combination thereof. In some examples, the enhanced representation of the environment includes a representation of a portion of the environment that is difficult to sense from the perspective of the vehicle due to a failure in detection and / or tracking of one or more objects in the portion of the environment based on the vehicle's sensors or resulting in detection and / or tracking of the object with a low level of confidence (e.g., a confidence below a threshold).

[0153] In some examples, causing an enhanced representation of the environment for the vehicle to be generated based at least in part on merging a subset of the plurality of different representations includes fusing together at least two different representations of the subset of the plurality of different representations, for example, using data fusion engine 230. For example, the enhanced representation of the environment generated using data fusion engine 230 may include a fusion of two different representations from two different data sources 202, a data source and a TMC 204, two TMCs 204, etc. The enhanced representation of the environment in alert 345 for vehicle 310 may be based on a fusion of image data from camera 337 and image data from camera 335. The enhanced representation 870 of the environment for vehicle 810 may be based on a fusion of sensor data from camera 860, sensors of vehicle 810, sensors of vehicle 830, sensors of vehicle 805, sensors of vehicle 820, and / or sensors of vehicle 835. The enhanced representation 875 of the environment for vehicle 820 may be based on a fusion of sensor data from camera 860, sensors of vehicle 820, sensors of vehicle 835, sensors of vehicle 810, sensors of vehicle 830, sensors of vehicle 805, and / or sensors of TMC 855. In some examples, at least two different representations include an image captured by an image sensor and a depth dataset captured by a depth sensor. Fusing the at least two different representations together may include associating portions of the image with corresponding portions of the depth dataset representing corresponding portions of the environment. Thus, different pixels of the image may be associated with different depths in the depth dataset. For example, if Figure 8 The camera 860 includes both a depth sensor and an image sensor, and the fusion of the depth data from the depth sensor and the image data from the image sensor can form Figure 8 8, which depicts vehicle 830, vehicle 835, vehicle 810, and vehicle 820, indicating that vehicle 830 is closer to camera 860 than vehicle 835, and indicating that vehicle 810 is closer to camera 860 than vehicle 8205.

[0154] In some examples, the collaborative enhanced sensing system is configured and can cause an indication of an enhanced representation of the environment to be output at the vehicle. In some examples, the indication includes an alarm based on the enhanced representation of the environment. In some examples, the indication includes at least a portion of the enhanced representation of the environment. In some examples, the indication and / or the alarm may include at least a portion of the enhanced representation of the environment. In some examples, the indication and / or the alarm may include a warning to the vehicle (and / or the operator of the vehicle) based on the enhanced representation of the environment, such as indicating the risk of collision between the vehicle and another object (such as a second vehicle, a pedestrian (e.g., VRU 522, VRU 622), a bicycle, a cyclist, a motorcycle, a motorcycle rider, a scooter, a scooter operator, a structure (e.g., a building, a lamp post, an RSU), a plant (e.g., a tree), an animal, a stone, a pothole, spills on the road, debris, an obstacle 512, or a combination thereof). Examples of the indication and / or the alarm include the alarm 345. The indication and / or alert may be generated using the alert engine 244, the data fusion engine 230, the object tracking engine 228, the traffic control engine 222, the trained ML model 232, or a combination thereof. In some examples, causing the indication of an enhanced representation of the environment to be output at the vehicle includes transmitting the indication to the vehicle. In some examples, causing the indication of an enhanced representation of the environment to be output at the vehicle includes outputting the indication at the vehicle using one or more output devices (e.g., output device 242, output device 1135).

[0155] In some examples, the collaborative augmented sensing system is configured and can identify a risk of collision between the vehicle and the object based on one or more planned movements of the vehicle. The collaborative augmented sensing system can cause an indication and / or an alert based on an augmented representation of the environment to be output at the vehicle. The indication and / or alert indicates a risk of collision between the vehicle and the object. Alert 345 is an example of an alert indicating a risk of a possible collision 370 between the vehicle 310 and the object (e.g., the second vehicle 320).

[0156] In some examples, the collaborative augmented sensing system may use intelligence in how to provide alerts to vehicles and / or how to output alerts at vehicles. For example, the RSU of the collaborative augmented sensing system may broadcast and / or multicast information of different groups and may indicate to the vehicle via point-to-point communication which of the broadcast and / or multicast information is particularly relevant to the vehicle. In some examples, the MEC and / or RSU of the collaborative augmented sensing system may identify relevant sensor data and identify how to group the data from multiple vehicles into broadcast and / or multicast streams.

[0157] In some examples, the collaborative augmented sensing system performing process 1000 includes a computing device remote from the vehicle, such as a multi-access edge computing (MEC) system (e.g., server 206, MEC 740, MEC 840), a roadside unit (RSU) system (e.g., server 206, TMC 204, RSU 407, RSU 508, RSU 608, RSU 750, RSU 850), a remote server (e.g., server 206), or the vehicle computing device is part of a second vehicle in the environment. Examples of second vehicles include any of the examples of vehicles listed above. In some examples, the collaborative augmented sensing system causes an indication of an augmented representation of the environment to be output at the vehicle at least in part by transmitting an indication of the augmented representation of the environment from a computing device remote from the vehicle to the vehicle.

[0158] In some examples, the collaborative augmented sensing system performing process 1000 includes a vehicle computing device of a vehicle. In some examples, the collaborative augmented sensing system enables outputting an indication of an augmented representation of the environment at the vehicle at least in part by outputting an indication of the augmented representation of the environment using an output device of the vehicle (such as output device 242 and / or output device 1135). The output device may include, for example, a display, a speaker, a set of headphones, a headset, a tactile feedback actuator, or a combination thereof. For example, outputting an alert based on the augmented representation of the environment using an output device of the vehicle may include: displaying an alert using a display of the vehicle; playing an alert using a speaker of the vehicle; providing a tactile feedback alert using a tactile feedback actuator, output device 242, output device 1135; or a combination thereof.

[0159] In some examples, a vehicle and / or a vehicle computing system associated with the vehicle may be optimized to reduce latency and / or increase accuracy. A computing device (such as a MEC, RSU, or server) remote from the vehicle may have more computing resources and computing time available to perform operations of process 1000 and / or operations described herein as associated with process 1000, such as tracking objects, predicting potential collisions, and / or figuring out how to optimize information for those vehicles. For example, in some examples, a computing device remote from the vehicle may run a higher latency trained ML model (which may have higher accuracy) than some vehicles are able to run due to increased computing resources and / or increased computing time (during which those computing resources may be accessed and used for these operations). Thus, in some examples, the vehicle computing system may be unable to detect and / or track objects in the environment using a low-latency trained ML model, but a computing device remote from the vehicle may successfully detect and / or track objects in the environment using a higher-latency, higher-accuracy trained ML model and may transmit an indication of the object's location and / or movement to the vehicle (and / or generate an enhanced representation of the environment based on the object's location and / or movement). The vehicle computing system may even use input from the remote computing device to retrain and / or update its trained ML model to help optimize its weights to reduce the likelihood of failing to detect and / or track an object.

[0160] In some examples, the mechanisms and / or techniques for providing data to various types of computing devices as part of the collaborative enhanced sensing system performing process 1000 and / or receiving data at the computing device may be different. For example, the MEC of the collaborative enhanced sensing system may use a 5G Uu link to receive data. The RSU of the collaborative enhanced sensing system may use PC5 to receive data. Some mechanisms and / or techniques for providing and / or receiving data may have limited bandwidth capabilities (e.g., constrained for licensed bandwidth), such as PC5. In the case where such mechanisms and / or techniques for providing and / or receiving data have limited bandwidth capabilities, the transmitted data (e.g., sensor data and / or an indication of an enhanced representation of the environment) may be compressed before being transmitted to and / or received at the computing device as part of the collaborative enhanced sensing system. The transmission mechanism and / or technique may be optimized for limited bandwidth usage, such as by transmitting a smaller portion of the data at a time, not transmitting redundant data, compressing the data, using or increasing error checking (e.g., via automatic repeat request (ARQ) and / or hybrid ARQ (HARQ) schemes), bandwidth throttling, or a combination thereof.

[0161] In some examples, an imaging system may include: components for receiving sensor data of an environment, wherein the sensor data includes multiple different representations of one or more portions of the environment from multiple perspectives; components for identifying a subset of the multiple different representations of the one or more portions of the environment based on a context corresponding to a vehicle in the environment, wherein the context includes at least a position of the vehicle in the environment; and components for causing an enhanced representation of the environment for the vehicle to be generated based at least in part on merging the subsets of the multiple different representations.

[0162] In some examples, components for receiving sensor data include: image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, image sensor 130, sensor 212, sensor 218, sensor 238, image sensor of camera 335, image sensor of camera 337, Figure 4 a sensor of any one of the UEs, a sensor of the RSU 407, a sensor of the HV 502, a sensor of the RSU 508, a sensor of the HV 502, a sensor of the RSU 508, a sensor of the HV 602, a sensor of the RSU 608, an image sensor of the camera 760, a sensor of the TMC 755, a sensor of the RSU 750, a sensor of the eNodeB or gNodeB 745, Figure 7 sensors of any of the vehicles, image sensors of the camera 860, sensors of the TMC 855, sensors of the RSU 850, sensors of the eNodeB or gNodeB 845, Figure 8 A sensor of any of the vehicles, a sensor that captures at least a portion of the sensor data of operation 1005, an input device 1145, or a combination thereof, another sensor described herein, or a combination thereof.

[0163] In some examples, components for identifying subsets of multiple different representations include: image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, collaborative augmented sensing system 200, data source 202, TMC 204, server 206, recipient device 208, traffic control engine 222, object tracking engine 228, data fusion engine 230, trained ML model 232, vehicle 310, vehicle 320, vehicle 330, UE 402, UE 404, UE 406, UE 408, RSU 407, HV 502, RV1 504, NV 506, RSU 508, obstacle 512, VRU 522, HV 602, RV1 604, NV 605, RSU 608, VRU 622, vehicle 705, vehicle 710, vehicle 715, vehicle 720, vehicle 725, MEC 740, eNodeB or gNodeB 745, RSU 750, TMC 755, camera 760, vehicle 805, vehicle 810, vehicle 820, vehicle 830, vehicle 835, MEC 840, eNodeB or gNodeB 845, RSU 850, TMC 855, camera 860, NN 900, computing system 1100, processor 1110, any other system or device described herein, any component thereof, or any combination thereof.

[0164] In some examples, components for causing output of an indication of an augmented representation at a vehicle include: an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an image processor 150, an ISP 154, a host processor 152, a collaborative augmented sensing system 200, a data source 202, a TMC 204, a server 206, a recipient device 208, a traffic control engine 222, an object tracking engine 228, a data fusion engine 230, an alert engine 244, a trained ML model 232, a vehicle 310, a vehicle 320, a vehicle 330, a UE 402, a UE 404, a UE 406, a UE 408, an RSU 407, a HV 502, a RV1 504, a NV 506, a RSU 508, an obstacle 512, a VRU 522, a HV 602, a RV1 604, a NV 605, a RSU 608, VRU 622, vehicle 705, vehicle 710, vehicle 715, vehicle 720, vehicle 725, MEC 740, eNodeB or gNodeB 745, RSU 750, TMC 755, camera 760, vehicle 805, vehicle 810, vehicle 820, vehicle 830, vehicle 835, MEC 840, eNodeB or gNodeB 845, RSU 850, TMC 855, camera 860, NN 900, computing system 1100, processor 1110, any other system or device described herein, any component thereof, or any combination thereof.

[0165] In some examples, the processes described herein (e.g., Figure 1 The imaging process Figure 2 The collaborative enhanced sensing process Figure 3 Detection and alarm process, Figure 4 The communication process FIG. 5A to FIG. 5D The detection and communication process Figure 6 The detection and communication process Figure 7 The collaborative enhanced sensing process Figure 8 The collaborative enhanced sensing process Fig. 9 The neural network process Fig.10 The collaborative enhanced sensing process 1000, Fig.11The computing processes and / or other processes described herein) may be performed by a computing device or apparatus. In some examples, the processes described herein may be performed by the following: image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, collaborative enhanced sensing system 200, data source 202, TMC 204, server 206, recipient device 208, vehicle 310, vehicle 320, vehicle 330, UE 402, UE 404, UE 406, UE 408, RSU 407, HV 502, RV1 504, NV 506, RSU 508, obstacle 512, VRU 522, HV 602, RV1 604, NV 605, RSU 608, VRU 622, vehicle 705, vehicle 710, vehicle 715, vehicle 720, vehicle 725, MEC 740, eNodeB or gNodeB 745, RSU 750, TMC 755, camera 760, vehicle 805, vehicle 810, vehicle 820, vehicle 830, vehicle 835, MEC 840, eNodeB or gNodeB 845, RSU 850, TMC 855, camera 860, NN 900, computing system 1100, processor 1110, any other system or device described herein, any component thereof, or a combination thereof.

[0166] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR head-mounted device, an AR head-mounted device, an AR glasses, a network-connected watch or smart watch or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device with resource capabilities to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0167] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0168] The processes described herein are illustrated as logic flow diagrams, block diagrams, or conceptual diagrams, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer executable instruction stored on one or more computer-readable storage media that performs the described operations when executed by one or more processors. Generally speaking, computer executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0169] In addition, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or a combination of the above. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0170] Fig.11 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.11 An example of a computing system 1100 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1105. The connection 1105 may be a physical connection using a bus, or a direct connection into the processor 1110, such as in a chipset architecture. The connection 1105 may also be a virtual connection, a networked connection, or a logical connection.

[0171] In some embodiments, computing system 1100 is a distributed system, where the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some embodiments, a component may be a physical device or a virtual device.

[0172] The example system 1100 includes at least one processing unit (CPU or processor) 1110 and connections 1105 that couple various system components including system memory 1115, such as read only memory (ROM) 1120 and random access memory (RAM) 1125, to the processor 1110. The computing system 1100 may include a cache 1112 of high-speed memory directly connected to, close to, or integrated as part of the processor 1110.

[0173] Processor 1110 may include any general purpose processor and hardware or software services, such as services 1132, 1134, and 1136 stored in storage device 1130, that are configured to control processor 1110 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1110 may be essentially a completely self-contained computing system that includes multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0174] To enable user interaction, the computing system 1100 includes an input device 1145 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1100 may also include an output device 1135, which may be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 may include a communication interface 1140, which may generally govern and manage user input and system output. The communication interface may perform or facilitate the use of a wired and / or wireless transceiver to receive and / or send wired or wireless communications, including those utilizing the following: audio jacks / plugs, microphone jacks / plugs, universal serial bus (USB) ports / plugs, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transmission, Low power (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 1102.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1140 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 1100 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0175] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0176] Storage device 1130 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1110, causes the system to perform functions. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1110, connection 1105, output device 1135, etc.

[0177] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0178] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0179] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments can be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid blurring each embodiment.

[0180] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flow charts may describe operations as sequential processes, many operations in the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. The process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0181] The processes and methods according to the above examples can be implemented using computer executable instructions stored or otherwise obtained from a computer readable medium. These instructions may include, for example, instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0182] The equipment implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed in a single device.

[0183] Instructions, media for transmitting such instructions, computing resources for executing the instructions, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0184] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiment can be used for any number of environments and applications beyond the environment and application described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in an order different from the described order.

[0185] One of ordinary skill in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.

[0186] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0187] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component and / or any component is in direct or indirect communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0188] Claim language or other language that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that states "at least one of A and B" means A, B, or A and B. In another example, claim language that states "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that states "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0189] Various exemplary logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on specific applications and the design constraints proposed to the entire system. Technicians can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as making it depart from the scope of the application.

[0190] The technology described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technology can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these technologies can be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions, which execute one or more of the above methods when executed. Computer-readable data storage media can form part of a computer program product, which can include packaging materials. Computer-readable media can include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0191] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0192] Illustrative aspects of the present disclosure include:

[0193] Aspect 1: A device for media processing, the device comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: obtain sensor data of an environment from one or more sensors of at least one device, wherein the sensor data comprises multiple different representations of one or more parts of the environment from multiple perspectives; identify a subset of the multiple different representations of the one or more parts of the environment based on a context corresponding to a vehicle in the environment, wherein the context comprises at least a position of the vehicle in the environment; and generate an enhanced representation of the environment for the vehicle based at least in part on merging the subset of the multiple different representations.

[0194] Aspect 2. An apparatus according to Aspect 1, wherein in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify one or more representations of a portion of the environment including the location of the vehicle.

[0195] Aspect 3. An apparatus according to any one of Aspects 1 to 2, wherein the context includes a direction of travel of the vehicle in the environment, and wherein in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify one or more representations of a portion of the environment located in front of the vehicle in the direction of travel of the vehicle.

[0196] Aspect 4. An apparatus according to any one of Aspects 1 to 3, wherein the context includes a speed of the vehicle in the environment, and wherein in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify one or more representations of a portion of the environment located in front of the vehicle, wherein the size of the portion of the environment is based on the speed of the vehicle.

[0197] Aspect 5. An apparatus according to any one of Aspects 1 to 4, wherein the context includes the planned movement of the vehicle in the environment, and wherein in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify one or more representations of a portion of the environment located in front of the vehicle based on the planned movement of the vehicle.

[0198] Aspect 6. An apparatus according to Aspect 5, wherein the one or more processors are configured to: detect an indicator light of the vehicle in the sensor data; and determine the planned movement of the vehicle based on detecting the indicator light of the vehicle in the sensor data, wherein the planned movement of the vehicle is a turn.

[0199] Aspect 7. An apparatus according to any one of Aspects 5 to 6, wherein the one or more processors are configured to: receive an indication of the planned movement of the vehicle from the vehicle, wherein the planned movement of the vehicle is part of a planned route of the vehicle.

[0200] Aspect 8. An apparatus according to any one of Aspects 1 to 7, wherein the context includes objects in the environment other than the vehicle, and wherein in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify one or more representations of the portion of the environment that includes the objects.

[0201] Aspect 9. An apparatus according to any one of Aspects 1 to 8, wherein the one or more processors are configured to: identify a risk of collision between the vehicle and an object in the environment based on one or more planned movements of the vehicle; and cause an alarm based on the enhanced representation of the environment to be output at the vehicle, wherein the alarm indicates the risk of collision between the vehicle and the object.

[0202] Aspect 10. An apparatus according to any one of Aspects 1 to 9, wherein the context includes planned movement of objects in the environment other than the vehicle, and wherein in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify one or more representations of a portion of the environment located in front of the vehicle based on the planned movement of the objects.

[0203] Aspect 11. An apparatus according to any one of Aspects 1 to 10, wherein, in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify that a confidence level associated with each representation in the subset of the multiple different representations exceeds a confidence level threshold, wherein the confidence level of one of the multiple different representations is associated with a confidence that the one of the multiple different representations represents a portion of the environment located in front of the vehicle.

[0204] Aspect 12. The apparatus according to aspect 11, wherein the portion of the environment located in front of the vehicle includes the vehicle.

[0205] Aspect 13. An apparatus according to any one of aspects 11 to 12, wherein the confidence level is based on a resolution of the one of the multiple different representations.

[0206] Aspect 14. An apparatus according to any one of aspects 11 to 13, wherein the confidence level is based on an illumination level in the one of the plurality of different representations.

[0207] Aspect 15. An apparatus according to any one of aspects 11 to 14, wherein the confidence level is based on object detection using the one of the multiple different representations.

[0208] Aspect 16. An apparatus according to any one of Aspects 1 to 15, wherein, in order to receive the sensor data of the environment, the one or more processors are configured to receive at least a portion of the sensor data including at least one of the multiple different representations from a second vehicle in the environment.

[0209] Aspect 17. An apparatus according to any one of Aspects 1 to 16, wherein, in order to receive the sensor data of the environment, the one or more processors are configured to receive at least a portion of the sensor data including at least one of the multiple different representations from a roadside unit (RSU) in the environment.

[0210] Aspect 18. An apparatus according to any one of Aspects 1 to 17, wherein, in order to receive the sensor data of the environment, the one or more processors are configured to receive at least a portion of the sensor data including at least one of the multiple different representations from a camera in the environment.

[0211] Aspect 19. An apparatus according to any one of aspects 1 to 18, wherein the plurality of different representations comprises depth data captured using at least one depth sensor.

[0212] Clause 20. An apparatus according to any one of Clauses 1 to 19, wherein the plurality of different representations comprises image data captured using at least one image sensor.

[0213] Aspect 21. An apparatus according to any one of Aspects 1 to 20, wherein the one or more processors are configured to: receive traffic data from a traffic management controller, wherein the context includes at least a portion of the traffic data, wherein the portion of the traffic data is associated with at least a portion of the environment including the vehicle.

[0214] Aspect 22. An apparatus according to any one of Aspects 1 to 21, wherein, in order to generate the enhanced representation of the environment for the vehicle based at least in part on merging the subset of the multiple different representations, the one or more processors are configured to fuse together at least two different representations of the subset of the multiple different representations.

[0215] Aspect 23. An apparatus according to Aspect 22, wherein the at least two different representations include an image captured by an image sensor and a depth dataset captured by a depth sensor, wherein, in order to fuse the at least two different representations together, the one or more processors associate portions of the image with corresponding portions of the depth dataset representing corresponding portions of the environment.

[0216] Aspect 24. An apparatus according to any one of Aspects 1 to 23, wherein, in order to identify the subset of the multiple different representations based on the context corresponding to the vehicle in the environment, the one or more processors are configured to input the sensor data and the context into one or more trained machine learning models that output indications of the subset of the multiple different representations.

[0217] Aspect 25. An apparatus according to Aspect 24, wherein the one or more processors are configured to: identify a reaction of the vehicle in response to receiving the enhanced representation of the environment; and update the one or more trained machine learning models using the context, the subset of the multiple different representations, and the reaction as training data.

[0218] Aspect 26. An apparatus according to any one of Aspects 1 to 25, wherein, in order to obtain the sensor data of the environment from the one or more sensors, the one or more processors are configured to obtain the sensor data of the environment from the one or more sensors using at least one of a receiver, a transceiver and / or a communication interface.

[0219] Aspect 27. An apparatus according to any one of Aspects 1 to 26, wherein, in order to obtain the sensor data of the environment from the one or more sensors, the one or more processors are configured to obtain the sensor data of the environment from at least one device including at least one of the one or more sensors.

[0220] Aspect 28. The apparatus according to aspect 27, wherein the at least one device is not co-located with the apparatus.

[0221] Aspect 29. An apparatus according to any one of Aspects 1 to 28, wherein, in order to obtain the sensor data of the environment from the one or more sensors, the one or more processors are configured to obtain the sensor data of the environment from multiple devices including at least one of the one or more sensors.

[0222] Aspect 30. The apparatus of aspect 29, wherein the plurality of devices are not co-located with each other.

[0223] Aspect 31. An apparatus according to any one of Aspects 1 to 30, wherein, in order to identify the subset of the multiple different representations of the one or more parts of the environment based on the context, the one or more processors are configured to identify the subset of the multiple different representations of the one or more parts of the environment without any user input.

[0224] Aspect 32. An apparatus according to any one of Aspects 1 to 31, wherein, in order to identify the subset of the multiple different representations of the one or more parts of the environment based on the context, the one or more processors are configured to identify the subset of the multiple different representations of the one or more parts of the environment based on a selection made by a user using a user interface.

[0225] Aspect 33. An apparatus according to any one of Aspects 1 to 32, wherein, in order to identify the subset of the multiple different representations of the one or more parts of the environment based on the context, the one or more processors are configured to identify the subset of the multiple different representations of the one or more parts of the environment based on preferences associated with a user.

[0226] Aspect 34. An apparatus according to any one of Aspects 1 to 33, wherein at least a first representation of the plurality of different representations is received from a first vehicle, and wherein at least a second representation of the plurality of different representations is received from a second vehicle.

[0227] Aspect 35. An apparatus according to any one of Aspects 1 to 34, wherein at least a first representation of the multiple different representations is received from a first vehicle, and wherein at least a second representation of the multiple different representations is received from at least one of a roadside unit (RSU), a multi-access edge computing (MEC) system, or a traffic management controller (TMC).

[0228] Aspect 36. An apparatus according to any one of Aspects 1 to 35, wherein at least a first representation of the plurality of different representations is received from a first roadside unit (RSU), and wherein at least a second representation of the plurality of different representations is received from a second RSU.

[0229] Aspect 37. An apparatus according to any one of Aspects 1 to 36, wherein at least a first representation of the plurality of different representations is received from the vehicle, and wherein at least a second representation of the plurality of different representations is received from a second vehicle.

[0230] Aspect 38. An apparatus according to any one of Aspects 1 to 37, wherein at least a first representation of the multiple different representations is received from the vehicle, and wherein at least a second representation of the multiple different representations is received from at least one of a roadside unit (RSU), a multi-access edge computing (MEC) system, or a traffic management controller (TMC).

[0231] Aspect 39. An apparatus according to any one of Aspects 1 to 38, wherein the apparatus includes a computing device remote from the vehicle, and wherein the one or more processors are configured to: identify the vehicle in the environment before generating the enhanced representation of the environment for the vehicle; and transmit an indication of the enhanced representation of the environment to the vehicle.

[0232] Aspect 40. An apparatus according to Aspect 39, wherein the computing device of the apparatus is one of a multi-access edge computing (MEC) system, a roadside unit (RSU) system, a remote server, or a vehicle computing device, wherein the vehicle computing device is part of a second vehicle in the environment.

[0233] Aspect 41. An apparatus according to any one of Aspects 1 to 39, wherein the apparatus includes a vehicle computing device of the vehicle, and wherein the one or more processors are configured to: output an indication of the enhanced representation of the environment using an output device of the vehicle.

[0234] Aspect 42. A method for imaging, the method comprising: obtaining sensor data of an environment from one or more sensors of at least one device, wherein the sensor data comprises multiple different representations of one or more parts of the environment from multiple perspectives; identifying a subset of the multiple different representations of the one or more parts of the environment based on a context corresponding to a vehicle in the environment, wherein the context includes at least a position of the vehicle in the environment; and generating an enhanced representation of the environment for the vehicle based at least in part on merging the subset of the multiple different representations.

[0235] Aspect 43. The method of aspect 42, wherein identifying the subset of the plurality of different representations based on the context comprises identifying one or more representations of a portion of the environment including the location of the vehicle.

[0236] Aspect 44. A method according to any one of Aspects 42 to 43, wherein the context includes a direction of travel of the vehicle in the environment, and wherein identifying the subset of the multiple different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle in the direction of travel of the vehicle.

[0237] Aspect 45. A method according to any one of Aspects 42 to 44, wherein the context includes a speed of the vehicle in the environment, and wherein identifying the subset of the multiple different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle, wherein the size of the portion of the environment is based on the speed of the vehicle.

[0238] Aspect 46. A method according to any one of Aspects 42 to 45, wherein the context includes planned movement of the vehicle in the environment, and wherein identifying the subset of the multiple different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle based on the planned movement of the vehicle.

[0239] Aspect 47. The method according to Aspect 46 further includes: detecting an indicator light of the vehicle in the sensor data; and determining the planned movement of the vehicle based on detecting the indicator light of the vehicle in the sensor data, wherein the planned movement of the vehicle is a turn.

[0240] Aspect 48. The method according to any one of Aspects 46 to 47, further comprising: receiving an indication of the planned movement of the vehicle from the vehicle, wherein the planned movement of the vehicle is part of a planned route of the vehicle.

[0241] Aspect 49. A method according to any one of Aspects 42 to 48, wherein the context includes objects in the environment other than the vehicle, and wherein identifying the subset of the multiple different representations based on the context includes identifying one or more representations of the portion of the environment that includes the objects.

[0242] Aspect 50. A method according to any one of Aspects 42 to 49, the method further comprising: identifying a risk of collision between the vehicle and an object in the environment based on one or more planned movements of the vehicle; and causing an alarm based on the enhanced representation of the environment to be output at the vehicle, wherein the alarm indicates the risk of collision between the vehicle and the object.

[0243] Aspect 51. A method according to any one of Aspects 42 to 50, wherein the context includes planned movement of objects in the environment other than the vehicle, and wherein identifying the subset of the multiple different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle based on the planned movement of the objects.

[0244] Aspect 52. A method according to any one of Aspects 42 to 51, wherein identifying the subset of the multiple different representations based on the context includes identifying that a confidence level associated with each representation in the subset of the multiple different representations exceeds a confidence level threshold, wherein the confidence level of one of the multiple different representations is associated with a confidence that the one of the multiple different representations represents a portion of the environment located in front of the vehicle.

[0245] Aspect 53. A method according to Aspect 52, wherein the portion of the environment located in front of the vehicle includes the vehicle.

[0246] Aspect 54. A method according to any one of Aspects 52 to 53, wherein the confidence level is based on a resolution of the one of the multiple different representations.

[0247] Aspect 55. A method according to any one of aspects 52 to 54, wherein the confidence level is based on an illumination level in the one of the plurality of different representations.

[0248] Aspect 56. A method according to any one of aspects 52 to 55, wherein the confidence level is based on object detection using the one of the multiple different representations.

[0249] Aspect 57. A method according to any one of Aspects 42 to 56, wherein receiving the sensor data of the environment includes receiving at least a portion of the sensor data including at least one of the multiple different representations from a second vehicle in the environment.

[0250] Aspect 58. A method according to any one of Aspects 42 to 57, wherein receiving the sensor data of the environment includes receiving at least a portion of the sensor data including at least one of the multiple different representations from a roadside unit (RSU) in the environment.

[0251] Aspect 59. A method according to any one of Aspects 42 to 58, wherein receiving the sensor data of the environment includes receiving at least a portion of the sensor data including at least one of the multiple different representations from a camera in the environment.

[0252] Aspect 60. A method according to any one of aspects 42 to 59, wherein the plurality of different representations comprises depth data captured using at least one depth sensor.

[0253] Aspect 61. A method according to any one of Aspects 42 to 60, wherein the plurality of different representations comprises image data captured using at least one image sensor.

[0254] Aspect 62. A method according to any one of Aspects 42 to 61, the method further comprising: receiving traffic data from a traffic management controller, wherein the context includes at least a portion of the traffic data, wherein the portion of the traffic data is associated with at least a portion of the environment including the vehicle.

[0255] Aspect 63. A method according to any one of Aspects 42 to 62, wherein generating the enhanced representation of the environment for the vehicle based at least in part on merging the subset of the multiple different representations includes fusing together at least two different representations of the subset of the multiple different representations.

[0256] Aspect 64. A method according to Aspect 63, wherein the at least two different representations include an image captured by an image sensor and a depth dataset captured by a depth sensor, and wherein fusing the at least two different representations together includes associating portions of the image with corresponding portions of the depth dataset representing corresponding portions of the environment.

[0257] Aspect 65. A method according to any one of Aspects 42 to 64, wherein identifying the subset of the multiple different representations based on the context corresponding to the vehicle in the environment includes inputting the sensor data and the context into one or more trained machine learning models that output indications of the subset of the multiple different representations.

[0258] Aspect 66. The method according to Aspect 65 further includes: identifying a reaction of the vehicle in response to receiving the enhanced representation of the environment; and updating the one or more trained machine learning models using the context, the subset of the multiple different representations, and the reaction as training data.

[0259] Aspect 67. A method according to any one of Aspects 42 to 66, wherein obtaining the sensor data of the environment from the one or more sensors includes obtaining the sensor data of the environment from the one or more sensors using at least one of a receiver, a transceiver and / or a communication interface.

[0260] Aspect 68. A method according to any one of Aspects 42 to 67, wherein obtaining the sensor data of the environment from the one or more sensors includes obtaining the sensor data of the environment from at least one device including at least one of the one or more sensors.

[0261] Aspect 69. The method according to aspect 68, wherein the at least one device is not co-located with the apparatus performing the method.

[0262] Aspect 70. A method according to any one of Aspects 42 to 69, wherein obtaining the sensor data of the environment from the one or more sensors includes obtaining the sensor data of the environment from a plurality of devices including at least one of the one or more sensors.

[0263] Aspect 71. The method according to aspect 70, wherein the plurality of devices are not co-located with each other.

[0264] Aspect 72. A method according to any one of Aspects 42 to 71, wherein identifying the subset of the multiple different representations of the one or more parts of the environment based on the context includes identifying the subset of the multiple different representations of the one or more parts of the environment without any user input.

[0265] Aspect 73. A method according to any one of Aspects 42 to 72, wherein identifying the subset of the multiple different representations of the one or more parts of the environment based on the context includes identifying the subset of the multiple different representations of the one or more parts of the environment based on a selection made by a user using a user interface.

[0266] Aspect 74. A method according to any one of Aspects 42 to 73, wherein identifying the subset of the multiple different representations of the one or more parts of the environment based on the context includes identifying the subset of the multiple different representations of the one or more parts of the environment based on preferences associated with a user.

[0267] Aspect 75. A method according to any one of Aspects 42 to 74, wherein at least a first representation of the plurality of different representations is received from a first vehicle, and wherein at least a second representation of the plurality of different representations is received from a second vehicle.

[0268] Aspect 76. A method according to any one of Aspects 42 to 75, wherein at least a first representation of the multiple different representations is received from a first vehicle, and wherein at least a second representation of the multiple different representations is received from at least one of a roadside unit (RSU), a multi-access edge computing (MEC) system, or a traffic management controller (TMC).

[0269] Aspect 77. A method according to any one of Aspects 42 to 76, wherein at least a first representation of the multiple different representations is received from a first roadside unit (RSU), and wherein at least a second representation of the multiple different representations is received from a second RSU.

[0270] Aspect 78. A method according to any one of Aspects 42 to 77, wherein at least a first representation of the plurality of different representations is received from the vehicle, and wherein at least a second representation of the plurality of different representations is received from a second vehicle.

[0271] Aspect 79. A method according to any one of Aspects 42 to 78, wherein at least a first representation of the multiple different representations is received from the vehicle, and wherein at least a second representation of the multiple different representations is received from at least one of a roadside unit (RSU), a multi-access edge computing (MEC) system, or a traffic management controller (TMC).

[0272] Aspect 80. A method according to any one of Aspects 42 to 79, wherein the method is performed by an apparatus comprising a computing device remote from the vehicle, the method further comprising: identifying the vehicle in the environment before generating the enhanced representation of the environment for the vehicle; and transmitting an indication of the enhanced representation of the environment to the vehicle.

[0273] Aspect 81. A method according to Aspect 80, wherein the computing device of the apparatus is one of a multi-access edge computing (MEC) system, a roadside unit (RSU) system, a remote server, or a vehicle computing device, wherein the vehicle computing device is part of a second vehicle in the environment.

[0274] Aspect 82. A method according to any one of Aspects 42 to 81, wherein the method is performed by an apparatus including a vehicle computing device of the vehicle, and the method further comprises: using an output device of the vehicle to output an indication of the enhanced representation of the environment.

[0275] Aspect 83: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: obtain sensor data of an environment from one or more sensors of at least one device, wherein the sensor data includes multiple different representations of one or more parts of the environment from multiple perspectives; identify a subset of the multiple different representations of the one or more parts of the environment based on a context corresponding to a vehicle in the environment, wherein the context includes at least a position of the vehicle in the environment; and cause an enhanced representation of the environment for the vehicle to be generated based at least in part on merging the subset of the multiple different representations.

[0276] Aspect 84: The non-transitory computer-readable medium according to aspect 83, further comprising operations according to any one of aspects 2 to 41 and / or any one of aspects 43 to 82.

[0277] Aspect 85: An apparatus for image processing, the apparatus comprising: a component for obtaining sensor data of an environment from one or more sensors of at least one device, wherein the sensor data comprises multiple different representations of one or more parts of the environment from multiple perspectives; a component for identifying a subset of the multiple different representations of the one or more parts of the environment based on a context corresponding to a vehicle in the environment, wherein the context comprises at least a position of the vehicle in the environment; and a component for generating an enhanced representation of the environment for the vehicle based at least in part on merging the subset of the multiple different representations.

[0278] Aspect 86: The apparatus according to aspect 85, further comprising means for performing the operations according to any one of aspects 2 to 41 and / or any one of aspects 43 to 82.

Claims

1. A device for processing sensor data, the device comprising: at least one memory; and one or more processors coupled to the at least one memory, the one or more processors configured to: obtaining sensor data of an environment from one or more sensors, wherein the sensor data includes a plurality of different representations of one or more portions of the environment from a plurality of viewpoints; identifying a subset of the plurality of different representations of the one or more portions of the environment based on a context corresponding to a vehicle in the environment, wherein the context includes at least a location of the vehicle in the environment; as well as An enhanced representation of the environment of the vehicle is generated based at least in part on merging the subset of the plurality of different representations.

2. The apparatus of claim 1, wherein to identify the subset of the plurality of different representations based on the context, the one or more processors are configured to identify one or more representations of a portion of the environment that includes the location of the vehicle.

3. An apparatus according to claim 1, wherein the context includes a direction of travel of the vehicle in the environment, and wherein in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify one or more representations of a portion of the environment located in front of the vehicle in the direction of travel of the vehicle.

4. An apparatus according to claim 1, wherein the context includes a speed of the vehicle in the environment, and wherein in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify one or more representations of a portion of the environment located in front of the vehicle, wherein the size of the portion of the environment is based on the speed of the vehicle.

5. An apparatus according to claim 1, wherein the context includes a planned movement of the vehicle in the environment, and wherein in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify one or more representations of a portion of the environment located in front of the vehicle based on the planned movement of the vehicle.

6. The apparatus of claim 5, wherein the one or more processors are configured to: detecting an indicator light of the vehicle in the sensor data; and The planned movement of the vehicle is determined based on detecting the indicator light of the vehicle in the sensor data, wherein the planned movement of the vehicle is a turn.

7. The apparatus of claim 5, wherein the one or more processors are configured to: An indication of the planned movement of the vehicle is received from the vehicle, wherein the planned movement of the vehicle is part of a planned route of the vehicle.

8. An apparatus according to claim 1, wherein the context includes objects in the environment other than the vehicle, and wherein in order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify one or more representations of the portion of the environment that includes the objects.

9. The apparatus of claim 1, wherein the one or more processors are configured to: identifying a risk of collision between the vehicle and an object in the environment based on one or more planned movements of the vehicle; and An alert based on the augmented representation of the environment is caused to be output at the vehicle, wherein the alert indicates the risk of collision between the vehicle and the object.

10. The device according to claim 1, wherein: In order to identify the subset of the multiple different representations based on the context, the one or more processors are configured to identify that a confidence level associated with each representation in the subset of the multiple different representations exceeds a confidence level threshold, wherein the confidence level of one of the multiple different representations is associated with a confidence that the one of the multiple different representations represents a portion of the environment located in front of the vehicle.

11. The device according to claim 1, wherein: To receive the sensor data of the environment, the one or more processors are configured to receive at least a portion of the sensor data including at least one of the plurality of different representations from a second vehicle in the environment.

12. The device according to claim 1, wherein: To receive the sensor data of the environment, the one or more processors are configured to receive at least a portion of the sensor data including at least one of the plurality of different representations from a roadside unit (RSU) in the environment.

13. The device according to claim 1, wherein: To receive the sensor data of the environment, the one or more processors are configured to receive at least a portion of the sensor data including at least one of the plurality of different representations from a camera in the environment.

14. The apparatus of claim 1, wherein the plurality of different representations comprises depth data captured using at least one depth sensor.

15. The apparatus of claim 1, wherein the plurality of different representations comprises image data captured using at least one image sensor.

16. The device according to claim 1, wherein To cause the enhanced representation of the environment for the vehicle to be generated based at least in part on merging the subset of the plurality of different representations, the one or more processors are configured to fuse together at least two different representations of the subset of the plurality of different representations.

17. The apparatus of claim 1, wherein the apparatus comprises a computing device remote from the vehicle, and wherein the one or more processors are configured to: identifying the vehicle in the environment prior to causing generation of the augmented representation of the environment for the vehicle; and An indication of the enhanced representation of the environment is transmitted to the vehicle.

18. An apparatus according to claim 17, wherein the computing device of the apparatus is one of a multi-access edge computing (MEC) system, a roadside unit (RSU) system, a remote server, or a vehicle computing device, wherein the vehicle computing device is part of a second vehicle in the environment.

19. The apparatus of claim 1, wherein the apparatus comprises a vehicle computing device of the vehicle, and wherein the one or more processors are configured to: An indication of the augmented representation of the environment is output using an output device of the vehicle.

20. A method for sensor data processing, the method comprising: obtaining sensor data of an environment from one or more sensors, wherein the sensor data includes a plurality of different representations of one or more portions of the environment from a plurality of viewpoints; identifying a subset of the plurality of different representations of the one or more portions of the environment based on a context corresponding to a vehicle in the environment, wherein the context includes at least a location of the vehicle in the environment; as well as An enhanced representation of the environment of the vehicle is generated based at least in part on merging the subset of the plurality of different representations.

21. The method of claim 20, wherein identifying the subset of the plurality of different representations based on the context comprises identifying one or more representations of a portion of the environment that includes the location of the vehicle.

22. A method according to claim 20, wherein the context includes a direction of travel of the vehicle in the environment, and wherein identifying the subset of the multiple different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle in the direction of travel of the vehicle.

23. A method according to claim 20, wherein the context includes a speed of the vehicle in the environment, and wherein identifying the subset of the multiple different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle, wherein the size of the portion of the environment is based on the speed of the vehicle.

24. A method according to claim 20, wherein the context includes planned movement of the vehicle in the environment, and wherein identifying the subset of the multiple different representations based on the context includes identifying one or more representations of a portion of the environment located in front of the vehicle based on the planned movement of the vehicle.

25. The method of claim 20, wherein the context includes objects in the environment other than the vehicle, wherein identifying the subset of the plurality of different representations based on the context includes identifying one or more representations of portions of the environment that include the objects.

26. A method according to claim 20, wherein identifying the subset of the multiple different representations based on the context includes identifying that a confidence level associated with each representation in the subset of the multiple different representations exceeds a confidence level threshold, wherein the confidence level of one of the multiple different representations is associated with a confidence that the one of the multiple different representations represents a portion of the environment located in front of the vehicle.

27. The method of claim 20, wherein receiving the sensor data of the environment comprises receiving at least a portion of the sensor data comprising at least one of the plurality of different representations from a second vehicle in the environment.

28. The method of claim 20, wherein causing generation of the enhanced representation of the environment for the vehicle based at least in part on merging the subset of the plurality of different representations comprises fusing together at least two different representations of the subset of the plurality of different representations.

29. The method of claim 20, wherein the method is performed by an apparatus comprising a computing device remote from the vehicle, the method further comprising: identifying the vehicle in the environment prior to causing generation of the augmented representation of the environment for the vehicle; as well as An indication of the enhanced representation of the environment is transmitted to the vehicle.

30. The method of claim 20, wherein the method is performed by an apparatus including a vehicle computing device of the vehicle, the method further comprising: An indication of the augmented representation of the environment is output using an output device of the vehicle.