Sensor output modification

Abstract

The present disclosure provides "sensor output modification". A computer includes a processor and a memory including instructions executable by the processor to actuate components of a device based on parameter outputs from a machine learning application trained with first output data from a first sensor, the first output data has been modified (1) according to a first specified characteristic of the first sensor and (2) according to a second specified characteristic of a second sensor.

Description

Technical Field

[0001] This disclosure relates to modifications of sensor outputs in a device. Background Technology

[0002] Computer-operable mobile systems may include vehicles, robots, drones, and / or object tracking systems. Data, including images, lidar measurement points, and radar range and velocity measurements, can be acquired by sensors and processed by a computer to determine the system's position relative to its environment and relative to static or moving objects within that environment. The computer can use sensor data to determine one or more trajectories and / or actions for operating a system or device or its components within the environment. Summary of the Invention

[0003] Systems that move and / or include moving parts (including vehicles, robots, land-based or aerial drones, mobile phones, etc.) operate by acquiring sensor data (including data about the environment surrounding the system) and processing the acquired sensor data to determine the position of an object in the environment surrounding the system. The determined position data can be processed to guide the operation of the system or parts thereof. For example, a robot can determine the position of a static or moving object near its arm. The robot can use the determined position of the static or moving object to determine a path to move a gripper over it to grasp the object. In another example, a first vehicle can determine the position of a second vehicle traveling on a road. The first vehicle can use the determined position of the second vehicle to determine a path to operate over it while maintaining a predetermined distance from the second vehicle. Alternatively or additionally, the first vehicle can actuate its display to provide text and / or graphics indicating the position and / or speed of the second vehicle.

[0004] For example, a vehicle computer can use input data acquired by one or more sensors to determine the location of static or moving objects within the vehicle's environment. For instance, the vehicle can utilize a lidar sensor that provides output data representing a measurement point indicating the distance between the lidar sensor and the static or moving object. The output data from the lidar sensor can be supplemented by output data from, for example, a radar sensor, which can provide an indication of whether the detected object is stationary or moving at a specific velocity vector. The output data from the lidar sensor can be further supplemented by output data from a camera, which the vehicle computer can use to classify static or moving objects (such as road markings, road signs, stationary vehicles, moving vehicles, cyclists, natural objects, animals, etc.). In one example, the vehicle computer can execute procedural steps that allow the fusion of output data from different types or modalities of sensors (such as cameras, lidar, radar, etc.) to provide classification and / or localization of objects within the vehicle's operating environment (e.g., within the sensor's field of view).

[0005] In one example, the vehicle computer may utilize a machine learning system (such as a convolutional neural network) trained to classify static or moving objects in the driving environment based on images captured by a camera or another imaging device. In another example, the vehicle computer may be trained via a machine learning system to classify certain types of static or moving objects based on radar signals returned from the objects. In yet another example, the vehicle computer may be trained via a machine learning system to classify certain types of static or moving objects based on measurement point clouds obtained by a lidar sensor. Furthermore, the vehicle computer may be trained to appropriately fuse output signals from multiple sensor device types (such as camera sensors, radar sensors, lidar sensors, etc.) to provide an accurate indication of static or moving objects in the driving environment based on the fused output signals from multiple sensor device types.

[0006] Training a machine learning system may involve utilizing a training dataset that includes numerous (e.g., thousands or even millions) still or camera images, numerous LiDAR measurement point clouds, numerous signals representing radar signal echoes, and / or numerous other types of sensor measurements. In a training environment for a machine learning system used in a vehicle, video and / or still images, LiDAR measurement point clouds, and data representing radar signal echoes from still or moving objects can be collected from sensors mounted at locations representing the actual sensor positions of the vehicle when it will be used in a driving environment. In one example, the process of training a machine learning system to classify still or moving objects based on images from a camera mounted at a specific location on the vehicle can be accelerated and / or enhanced by utilizing images collected by a camera that approximates (or even simulates) the camera of a vehicle intended for use in a real traffic environment. In another example, the process of training a machine learning system to classify still or moving objects can be accelerated and / or enhanced by utilizing training images captured by a camera with inherent characteristics (e.g., camera distortion, field of view, spectral filtering characteristics of the camera lens, etc.) that approximate (or even simulate) the inherent parameters of a camera intended for use in a real traffic environment. In another example, the process of training a machine learning system to classify lidar measurement point clouds can be accelerated using lidar sensors that are approximately (or simulate) mounted on a vehicle at a location similar to or the same as the lidar sensor mounting location of a vehicle intended for use in a real traffic environment.

[0007] The techniques described herein can be used to accelerate and / or enhance the training of machine learning systems by modifying measurements from a first sensor (e.g., a first system) to conform to specified characteristics of a second sensor (e.g., a second system) for use in a real-world operating environment. Thus, as further described herein, output data from a first sensor can be modified by transforming it to represent or approximate data acquired from a second sensor, which may include a sensor mounted, for example, on a vehicle intended for use in a traffic environment. Therefore, a dataset can be modified to provide images suitable for training a machine learning system, the dataset comprising, for example, thousands or millions of camera images obtained from a first camera with specific characteristics and / or mounted at a specific location on a first vehicle, the machine learning system utilizing a second camera with specific characteristics and / or mounted at different locations on a second vehicle. Alternatively or additionally, a dataset from a first vehicle can be modified to provide a dataset suitable for training a machine learning system, the dataset from the first vehicle comprising thousands or millions of point clouds representing lidar measurements, data representing radar signal echoes, or data representing output signals from another sensor, the machine learning system utilizing a second sensor of a similar type mounted on a second vehicle.

[0008] In one example, a method may include actuating components of a device based on parameter output from a machine learning application, the machine learning application being trained with first output data from a first sensor, the first output data having been (1) modified according to a first specified characteristic of the first sensor, and (2) modified according to a second specified characteristic of a second sensor.

[0009] The first specified characteristic of the first sensor may be the noise content of the output data from the first sensor, the field of view of the first sensor, the detection range of the first sensor, the resolution of the first sensor, or the sampling interval of the first sensor.

[0010] The first output data from the first sensor, which has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor, can reference the first output data from a first specified reference point on the first vehicle to a second specified reference point on the second vehicle.

[0011] The first sensor may be a first lidar sensor, and the second sensor may be a second lidar sensor.

[0012] The first specified characteristic may be the field of view of the first lidar sensor, the second specified characteristic may be the field of view of the second lidar sensor, and the first output data from the first sensor that has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor may be processed to omit measurement points that are outside the field of view of the second lidar sensor but within the field of view of the first lidar sensor.

[0013] The first specified characteristic may be the detection range of the first lidar sensor, the second specified characteristic may be the detection range of the second lidar sensor, and the first output data from the first sensor that has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor may omit the measurement point representing the distance outside the detection range of the second lidar sensor.

[0014] The first specified characteristic may be the first scanning interval of the first lidar sensor, the second specified characteristic may be the second scanning interval of the second lidar sensor, and the first output data from the first sensor that has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor may omit measurement points collected outside the second scanning interval during the first scanning interval.

[0015] The first specified characteristic may be the first scanning resolution of the first lidar sensor, the second specified characteristic may be the second scanning resolution of the second lidar sensor, and the first output data from the first sensor that has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor may omit the measurement points collected at the first scanning resolution that is different from the second scanning resolution.

[0016] The first sensor may be a first radar sensor, and the second sensor may be a second radar sensor.

[0017] The first specified characteristic may be the first detection range of the first radar sensor, the second specified characteristic may be the second detection range of the second radar sensor, and the first output data from the first sensor that has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor omits measurement points outside the second detection range of the first detection range.

[0018] The first specified characteristic may be the first scanning interval of the first radar sensor, the second specified characteristic may be the second scanning interval of the second radar sensor, and the first output data from the first sensor that has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor may align the first radar scanning interval with the second radar scanning interval.

[0019] The first sensor may be a first camera sensor, and the second sensor may be a second camera sensor.

[0020] The first specified characteristic may be a first distortion parameter of the first camera sensor, and the second specified characteristic may be a second distortion parameter of the second camera sensor. The first output data from the first sensor, which has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor, may be modified using the first distortion parameter to generate the pixel position of the first back-projection. The second sensor may further apply the second distortion parameter to the generated first pixel position of the back-projection.

[0021] The first specified characteristic may be a first pixel gain parameter of the first camera sensor, and the second specified characteristic may be a second pixel gain parameter of the second camera sensor. The first output data from the first sensor, which has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor, may be modified using the first pixel gain parameter to generate a first back-projected pixel gain value. The second sensor may further apply the second pixel gain parameter to the first back-projected pixel gain value.

[0022] In one example, a system may include a computer including a processor and a memory storing instructions that can be executed by the processor to output components of an actuator based on parameters trained with first output data from a first sensor, the first output data being (1) modified according to a first specified characteristic of the first sensor and (2) modified according to a second specified characteristic of a second sensor.

[0023] The first specified characteristic of the first sensor may be the noise content of the output data from the first sensor, the field of view of the first sensor, the detection range of the first sensor, the resolution of the first sensor, or the sampling interval of the first sensor.

[0024] The first output data from the first sensor, which has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor, can reference the first output data from a first specified reference point on the first vehicle to a second specified reference point on the second vehicle.

[0025] The first sensor may be a first radar sensor, and the second sensor may be a second radar sensor.

[0026] The first sensor may be a first camera sensor, and the second sensor may be a second camera sensor.

[0027] The first sensor may be a first lidar sensor, and the second sensor may be a second lidar sensor. Attached Figure Description

[0028] Figure 1 This is a block diagram of the example first vehicle system.

[0029] Figure 2 This is a block diagram of the example second vehicle system.

[0030] Figure 3 This is a diagram of an example system used to modify sensor measurements acquired by the first sensor.

[0031] Figure 4 This is a diagram illustrating the differences in the characteristics of the sensors in the first system and the second system.

[0032] Figure 5 This is a diagram illustrating an example process for modifying an image captured from a first sensor.

[0033] Figure 6 This is a flowchart of an example process for modifying sensor measurements from a first sensor.

[0034] Figure 7 This is a flowchart of an example process for actuating vehicle components of a second system based on parameters uploaded from a training network. Detailed Implementation

[0035] Figure 1 This is a block diagram of an example first vehicle system 100. The first vehicle system 100 includes a vehicle body 102, a vehicle computer 104 included within the vehicle body 102, and numerous sensors 108 mounted on the vehicle body 102. The first vehicle system 100 can represent any type of vehicle, such as a sedan, SUV, truck, bus, or any other vehicle operable on a road 150. The sensors 108 may include lidar sensors 108A, camera sensors 108B, radar sensors 108C and 108D, and other sensors such as wheel speed sensors, navigation sensors (e.g., sensors for satellite positioning systems, sensors for inertial measurement units, outdoor air temperature sensors, engine and drivetrain monitoring sensors, etc.). The vehicle computer 104 can receive data about the operation of the first vehicle system 100 from the sensors 108 using a vehicle communication bus 106. The vehicle computer 104 can operate the first vehicle system 100 and actuate vehicle components 110 based on data received from sensors 108. These vehicle components may include vehicle steering components, vehicle propulsion components (i.e., controlling the speed and / or speed changes of the first vehicle system 100 by controlling one or more of an internal combustion engine, electric motor, hybrid engine, etc.), electric and / or electro-hydraulic engines and transmission components, climate controls, interior and exterior lighting equipment, etc. The vehicle computer 104 can also determine whether and when it, rather than a human operator, controls such operations.

[0036] Vehicle computer 104 may include one or more processors and memories, as known herein. Furthermore, the memory may include one or more forms of non-volatile computer-readable medium storing instructions that can be executed by the processor to perform various operations, including those disclosed herein. Vehicle computer 104 may generally be arranged for communication on any suitable type of vehicle communication bus 106 (i.e., including Controller Area Network (CAN), Local Area Network (LIN), or another suitable communication bus architecture). Vehicle communication bus 106 may include known wired or wireless communication mechanisms (i.e., Ethernet, Bluetooth, or another communication protocol).

[0037] Via the vehicle communication bus 106, the vehicle computer 104 can transmit messages to and receive messages from various subsystems, components, and devices of the first vehicle system 100. Alternatively or additionally, in examples where the vehicle computer 104 actually comprises multiple devices, the vehicle communication bus 106 can be used for communication between devices represented as the vehicle computer 104 in this disclosure. Furthermore, as mentioned below, various controllers or sensing elements (such as sensor 108) can provide data to the vehicle computer 104 via the vehicle communication bus 106.

[0038] Additionally, the vehicle computer 104 can be configured to communicate via a vehicle-to-infrastructure (V2I) interface using the communication component 114. The communication interface may include wireless fidelity. Interface, cellular network interface, The vehicle computer 104 may be configured to communicate with other vehicles via a vehicle-to-the-world (V2X) interface, including interfaces such as Bluetooth Low Energy (BLE), Ultra Wideband (UWB), peer-to-peer communication, and / or wired and wireless packet networks or technologies. The vehicle computer 104 can be configured to communicate with other vehicles via a vehicle-to-vehicle network (i.e., based on or including cellular communication (C-V2X), wireless communication, Dedicated Short Range Communication (DSRC), etc.) formed between neighboring vehicles on the basis of a mobile ad hoc network or through an infrastructure-based network. The vehicle computer 104 may record data by storing the data in non-volatile memory for subsequent retrieval and transmission via the vehicle communication network and the vehicle-to-infrastructure (V2I) interface.

[0039] The vehicle computer 104 may additionally communicate with the human-machine interface (HMI) 112 via the vehicle communication bus 106. In one example, in response to communication from the vehicle computer 104, the HMI 112 may provide audio signals and / or activation of haptic actuators (such as vibration actuators on the steering wheel or in the cushions of the vehicle system 100) to provide notification to the operator of the vehicle system 100.

[0040] Sensor 108 may include a variety of known devices to provide data to vehicle computer 104 via vehicle communication bus 106. Figure 1In the example, sensor 108A may represent a lidar sensor that provides measurement points representing the distance between lidar sensor 108A and a static or moving object located in the forward, rearward, or side direction of vehicle body 102. In one example, lidar sensor 108A may include a plurality of laser radiating elements (e.g., 16 lasers, 32 lasers, 64 lasers, etc.) that provide a point cloud representing a plurality of measurement points, each representing the distance between lidar sensor 108A and a static or moving object in the traffic environment of vehicle system 100. Although lidar sensor 108A is indicated to be mounted on an outward-facing surface of the upper structure of vehicle body 102, in other examples, lidar sensor 108A may be mounted on an outward-facing surface of vehicle body 102, such as on the hood of vehicle body 102, front grille of vehicle body 102, etc.

[0041] Sensor 108 may include a camera sensor, such as camera sensor 108B positioned near the upper boundary of the windshield of vehicle body 102. Camera sensor 108B may include a camera for capturing static or video scenes, including objects located outside vehicle body 102, such as stationary or moving vehicles, animals, natural objects, lane markings, traffic signs, etc. In one example, camera sensor 108B may detect electromagnetic radiation within a certain wavelength range. For example, camera sensor 108B may detect visible light, infrared radiation, ultraviolet light, or wavelengths including the range of visible light, infrared light, and / or ultraviolet light. For example, camera sensor 108B may include an image sensor such as a charge-coupled device (CCD), an active pixel sensor such as a complementary metal-oxide-semiconductor (CMOS) sensor, etc.

[0042] Sensor 108 may include radar sensors 108C and 108D mounted on opposite sides of the front bumper of vehicle body 102. In one example, radar sensors 108C and 108D may provide output data representing the distance (e.g., range) between the radar sensors and a static or moving object within the radar field of view. Radar sensors 108C and 108D may also provide output data representing the velocity of an object within the radar field of view.

[0043] Sensor 108 may include altimeters, ultrasonic sensors, infrared sensors, pressure sensors, accelerometers, gyroscopes, temperature sensors, Hall effect sensors, mechanical sensors such as switches, etc. Such additional sensors can be used to provide output data representing the environment in which the first vehicle system 100 operates. For example, some sensors in sensor 108 can be used to detect phenomena such as weather conditions (precipitation, ambient temperature, etc.), road gradient, road position (i.e., using road edges, lane markings, etc.), or the position of static or moving objects (such as adjacent vehicles). Some sensors in sensor 108 may additionally collect data representing the operation of the first vehicle system 100, such as speed, yaw rate, steering angle, engine speed, oil pressure, power applied to components 110 of the vehicle system 100, connectivity between components, and the performance of components of the vehicle system 100.

[0044] The vehicle computer 104 may additionally include a communication interface with the sensor measurement database 120. In one example, when the vehicle system 100 is in operation, sensor measurements (such as measurements performed by sensors 108 (e.g., lidar sensor 108A, camera sensor 108B, radar sensors 108C and 108D, etc.)) may be stored for use in training a machine learning system. Therefore, the sensor measurement database 120 can store numerous sensor measurements, such as thousands of sensor measurements, millions of sensor measurements, etc. (See also: Regarding...) Figures 2 to 6 As described, measurements acquired by the sensor 108 of the first vehicle system 100 can be modified, such as regarding... Figure 3 As described above, in order to approximate or simulate the second system (such as...) Figure 2 Sensor measurements are performed by sensors in the second vehicle system 200. Therefore, as further described herein, the sensor measurements stored in the sensor measurement database 120 represent measurements from a first system (such as a vehicle system), which can be modified and used to train a machine learning system to work with the second system (such as...). Figure 2 (and the system) are used together.

[0045] Figure 2 This is a block diagram of an example second vehicle system 200. The second vehicle system 200 may include components, actuators, sensors, etc., that are similar in type to those in the first vehicle system 100. Therefore, the second vehicle system 200 includes a vehicle body 202, a vehicle computer 204 included within the vehicle body 202, and numerous sensors 208. Sensors 208 may include a lidar sensor 208A, a camera sensor 208B, radar sensors 208C and 208D, and other sensors such as wheel speed sensors, navigation sensors (e.g., sensors for satellite positioning systems, sensors for inertial measurement units, outdoor air temperature sensors, engine and powertrain monitoring sensors, etc.).

[0046] The second vehicle system 200 may include a car, truck, SUV, bus, or any other vehicle capable of operating on roads. Figure 2 In the example, the second vehicle system 200 includes a body 202, which is larger than the body 102. The second vehicle system 200 includes sensors (i.e., lidar sensor 208A, camera sensor 208B, and radar sensors 208C and 208D), some of which (e.g., lidar sensor 208A) may be mounted on the body 202 at locations corresponding to the mounting locations of similar sensors on or within the vehicle system 100. Figure 2 In the example, the lidar sensor 208A is shown mounted on the front grille portion of the vehicle body 202, which is consistent with... Figure 1 The upper structure of the lidar sensor 108A shown in the example is different. Figure 2 The example further illustrates a camera sensor 208B mounted on the upper part of the windshield of the vehicle body 202. However, because the size of the vehicle body 202 is larger than that of the vehicle body 102, the camera sensor 208B is positioned at a greater distance from the road 250 than the camera sensor 108B is positioned relative to the road 150. Therefore, the viewing angle of the camera sensor 208B is different from that of the camera sensor 108B. Consequently, an image of a static or moving object captured by the camera sensor 208B may look different from an image of an object captured by the camera sensor 108B. Similarly, because the size of the vehicle body 202 is larger than that of the vehicle body 102, radar sensors 208C and 208D are positioned at a greater distance from the road 250 than the radar sensors 108C and 108D are positioned relative to the road 150. Therefore, the elevation angle of arrival of the radar signal echoes from static or moving objects detected by radar sensors 208C and 208D may differ from the elevation angle of arrival of the radar signal echoes from objects captured by radar sensors 108C and 108D.

[0047] Vehicle computer 204 can receive data about the operation of the second vehicle system 200 from sensor 108 via vehicle communication bus 206. Vehicle computer 204 can operate vehicle system 200 and actuate vehicle components 210 based on the data received from sensor 208. These vehicle components may include vehicle steering components, vehicle propulsion components (i.e., controlling the speed and / or speed changes of the second vehicle system 200 by controlling one or more of an internal combustion engine, electric motor, hybrid engine, etc.), electric and / or electro-hydraulic engines and transmission components, climate controls, interior and exterior lighting equipment, etc. Vehicle computer 204 can also determine whether and when vehicle computer 204, rather than a human operator, controls such operations.

[0048] Similar to vehicle computer 104, vehicle computer 204 may include one or more processors and memory. Furthermore, the memory may include one or more forms of non-volatile computer-readable medium storing instructions executable by the processor to perform various operations as disclosed herein. Vehicle computer 204 may typically be arranged for communication on any suitable type of vehicle communication bus 206 (i.e., including Controller Area Network (CAN), Local Area Network (LIN), or another suitable communication bus architecture). Vehicle communication bus 206 may include known wired or wireless communication mechanisms, i.e., Ethernet, Bluetooth, or other communication protocols.

[0049] Via the vehicle communication bus 206, the vehicle computer 204 can transmit messages to and receive messages from various subsystems, components, and devices of the second vehicle system 200. Alternatively or additionally, in examples where the vehicle computer 204 actually comprises multiple devices, the vehicle communication bus 206 can be used for communication between devices represented as the vehicle computer 204 in this disclosure. Furthermore, as mentioned below, various controllers or sensing elements (such as sensor 208) can provide data to the vehicle computer 204 using the vehicle communication bus 206.

[0050] Vehicle computer 204 can be configured to use the vehicle-to-infrastructure (V2I) interface utilizing communication component 214 via Interface, cellular network interface, The vehicle computer 204 can communicate via a vehicle-to-the-world (V2X) interface, including interfaces such as Bluetooth Low Energy (BLE), Ultra Wideband (UWB), peer-to-peer communication, and / or another interface utilizing wired and wireless packet networks or technologies. The vehicle computer 204 can be configured to communicate with other vehicles using a vehicle-to-vehicle network (i.e., based on or including cellular communication (C-V2X), wireless communication, Dedicated Short Range Communication (DSRC), etc.) formed between neighboring vehicles on the basis of a mobile ad hoc network or through an infrastructure-based network, via a vehicle-to-the-world (V2X) interface. The vehicle computer 204 can record data by storing it in non-volatile memory for subsequent retrieval and transmission via the vehicle communication network and the vehicle-to-infrastructure (V2I) interface.

[0051] The vehicle computer 204 may additionally communicate with the human-machine interface (HMI) 212 via the vehicle communication bus 206. In one example, in response to communication from the vehicle computer 204, the HMI 212 may provide audio signals and / or activation of haptic actuators, such as vibration actuators on the steering wheel or in the cushions of the second vehicle system 200.

[0052] Sensor 208 may include a variety of known devices to provide data to vehicle computer 204 via vehicle communication bus 206. Figure 2 In the example, sensor 208A may represent a lidar sensor that provides measurement points representing the distance between lidar sensor 208A and a static or moving object located in the forward, rearward, or side direction of vehicle body 202. In one example, lidar sensor 208A may include a plurality of laser radiating elements (e.g., 16 lasers, 32 lasers, 64 lasers, etc.) that provide a point cloud representing a plurality of measurement points, each representing the distance between lidar sensor 208A and a static or moving object in the traffic environment of vehicle system 100. Although lidar sensor 208A is indicated to be mounted on an outward-facing surface of the front grille portion of vehicle body 202, in other examples, lidar sensor 208A may be mounted on another outward-facing surface of vehicle body 202, such as on the hood of vehicle body 202, on the upper structure of vehicle body 202, etc.

[0053] exist Figure 2 In the example, the lidar sensor 208A is mounted to the front grille of the vehicle body 202. Therefore, referring to... Figure 1 The lidar measurement points collected by lidar sensor 108A represent measurement points collected from a different perspective than those collected by lidar sensor 208A. Therefore, sensor measurement modifier 220 can, for example, adjust the position of the measurement points collected via lidar sensor 108A in the negative (-) vertical direction along axis 235 to represent the measurement points collected by lidar sensor 208A. In another example, also as... Figure 2As shown, since the size of vehicle body 202 is larger than that of vehicle body 102, camera sensor 208B can be mounted at a vertical distance from road 250 greater than the distance of camera sensor 108B relative to road 150. Therefore, the image captured by camera sensor 208B is collected from a different perspective than the camera measurements collected by camera sensor 108B. Thus, sensor measurement modifier 220 can, for example, adjust the image captured by camera sensor 108B in the positive (+) vertical direction along axis 235 to represent the image collected by camera sensor 208B. In another example, also because the size of vehicle body 202 is larger than that of vehicle body 102, radar sensors 208C and 208D are mounted at a vertical distance from road 250 greater than the distance of radar sensors 108C and 108D relative to road 150. Therefore, the sensor measurement modifier 220 can, for example, adjust the radar signal echo captured by radar sensors 108C and 108D in the positive (+) vertical direction along axis 235 to represent the radar signal measurement collected using radar sensors 208C and 208D. (See reference...) Figure 3 The sensor measurement modifier 220 can perform additional modifications based on specified characteristics of sensors 108 and 208, such as modifications to sensor field of view, sensor sampling rate, sensor noise content, inherent sensor characteristics, etc. In this disclosure, the term "inherent" camera sensor parameter refers to characteristics internal to camera sensors 108B and 208B. Therefore, inherent camera sensor parameters include focal length, skew, field of view, pixel resolution, pixel noise, pixel gain, aperture diameter, distortion, lens spectral filtering or shading, and camera sensor depth of field. In this disclosure, the term "external" camera parameter refers to characteristics external to camera sensors 108B and 208B. Therefore, external camera parameters may include camera mounting position on a vehicle system (e.g., vehicle systems 100, 200), camera orientation, obstacles in the camera's field of view, etc.

[0054] Therefore, in Figure 2In the example, sensor measurements from sensor measurement database 120 (representing sensor measurements collected using the first vehicle system 100) can be modified to approximate sensor measurements collected using the second system 200. Such modified sensor measurements, which may include thousands or millions of sensor measurements, can be input into machine learning system 230. Therefore, by performing this training using modified sensor measurements collected by the first vehicle system 100, machine learning system 230 can be trained to identify or classify static or moving objects measured using sensor 208. After training machine learning system 230, such as during the manufacturing or testing phase of system 200, machine learning system 230 can develop parameters that can be uploaded to a memory accessible to computer 204. Such parameters can help vehicle computer 204 classify static or moving objects represented by point clouds generated by lidar measurements, images (or features within images) of a scene captured via camera sensor 208B, static or moving objects represented by signal echoes from radar sensors 208C and 208D, etc.

[0055] In one example, the machine learning system 230 may include a convolutional neural network. In this context, a convolutional neural network is a feedforward artificial neural network having at least three layers (i.e., an input layer, an output layer, and at least one hidden layer). In one example, the input layer receives a set of measurement points from a lidar sensor 108A, data representing an image captured by a camera sensor 108B, and data representing echo signals from a radar sensor 108C, 108D, or another sensor among the sensors 108 of the first vehicle system 100. The output data may represent relatively large static or moving objects (e.g., buses, trucks, etc.) and relatively small static or moving objects (e.g., bicycles, compact vehicles, etc.). The output signals may additionally represent objects moving relative to the first vehicle system 100 at relatively low speeds (e.g., 5 km / h, 10 km / h, 15 km / h) and at higher speeds (e.g., 25 km / h, 30 km / h, 40 km / h, etc.).

[0056] The machine learning system 230 can compute a loss function representing the system's ability to accurately predict expected outputs. The loss function can be backpropagated through the hidden layers of the machine learning system 230, incrementally changing the weights or settings stored in the hidden layers to minimize the loss function. In this context, "weights" or "settings" refer to parameters of the hidden layers of the machine learning system 230, which at least partially control or manage the transformation or output of data from the machine learning system 230 by performing operations (e.g., addition, multiplication, convolution, or another function) to provide, for example, data at the output layer that can be observed by humans and / or the computer 204 of the second vehicle system 200. In response to the loss function being sufficiently minimized, the machine learning system 230 can be considered trained, and current parameters (e.g., formulated or derived from the weights and / or settings within the hidden layers of the machine learning system 230) can be uploaded for use by the vehicle computer 204.

[0057] Figure 3 This is a diagram of an example system 300 used to modify sensor measurements acquired by a first sensor. (Example:) Figure 3 As shown, the sensor measurement modifier 220 includes a sensor characteristic component 310, which includes a database, table, or other type of list of characteristics of sensors (e.g., sensor 108) of the first vehicle system 100. In this context, the term "sensor characteristic" refers to data describing a sensor, which may be related to interpreting output data from the sensor. For example, a sensor characteristic may be an attribute that distinguishes a first sensor (e.g., 108A, 108B, 108C, 108D) from a second sensor (e.g., 208A, 208B, 208C, 208D). The sensor measurement modifier 220 includes a sensor characteristic component 320, which similarly includes a database, table, or other type of list of characteristics of sensors (e.g., sensor 208) of the second vehicle system 200. In one example, sensor characteristics included in sensor characteristic components 310, 320 may include sensor angular field of view (e.g., 90°, 120°, 135°, etc.), sensor operating range (e.g., 50 m to 100 m, 100 m to 500 m, 250 m to 750 m, etc.), sensor sampling interval and / or update rate (e.g., 10 Hz, 25 Hz, 50 Hz, etc.), camera sensor inherent parameters (e.g., focal length, skew, pixel gain, spectral correlation lens shading, lens distortion, etc.), sensor noise content, and numerous other characteristics of sensors 108, 208. Figure 3In the example, the outputs from sensor characteristic components 310 and 320 are input to sensor measurement transformation component 330, which executes program instructions to modify the outputs from sensor measurement database 120 to approximate or simulate the sensor outputs from sensor 208 of the second vehicle system 200. The modified sensor measurements collected by the first vehicle system 100 can be used as training input to a machine learning system 230 to train the system 230 to classify static or moving objects measured using sensor 208. Based on the training of the machine learning system 230 (which may occur, for example, during the manufacturing or testing phase of system 200), the machine learning system 230 can develop parameters that can be uploaded to a memory accessible to computer 204. Using such parameters, vehicle computer 204 can execute instructions to classify static or moving objects in the traffic environment of the second vehicle system 200.

[0058] In one example, the sensor measurement transformation unit 330 can align the sampling intervals between lidar sensors 108A and 208A. For example, lidar sensor 108A can be specified to perform lidar scans at 0.5-second intervals, and lidar sensor 208A can be specified to perform lidar scans at 1.0-second intervals. In such an example, sensor characteristic unit 310 can transmit data to indicate that lidar sensor 108A includes a 0.5-second scan interval, and sensor characteristic unit 320 can transmit data to indicate that lidar sensor 208A includes a 1.0-second scan interval. Based on such input, instructions executed by the sensor measurement transformation unit 330 can filter the output from the sensor measurement database 120 to omit lidar sensor measurement points collected outside of one-second intervals (e.g., 0.5 seconds, 1.5 seconds, 2.5 seconds, 3.5 seconds, etc.) so that the output is lidar measurement points collected at one-second intervals (e.g., 1.0 seconds, 2.0 seconds, 3.0 seconds, etc.), thereby omitting measurement points collected outside of one-second intervals. The filtered output signal from the sensor measurement transformation unit 330 can then be input into the machine learning system 230, which allows the system 230 to be trained using lidar measurement points collected at one-second intervals to approximate the output data from the lidar sensor 208A.

[0059] In another example, the sensor measurement transformation component 330 can modify the scanning resolution characteristics of the lidar sensor 108A. For example, the lidar sensor 108A can be specified to include a scanning resolution of 1.0 cm, and the lidar sensor 208A can be specified to include a scanning resolution of 2.0 cm. In such an example, instructions executed by the sensor measurement transformation component 330 can filter the output from the sensor measurement database 120 to adjust the output data from the sensor measurement database 120 so that the output represents a lidar measurement with a scanning resolution of 2.0 cm. The filtered output signal from the sensor measurement transformation component 330 can then be input into the machine learning system 230, which can allow the system 230 to be trained using lidar measurements including a scanning resolution of 2.0 cm (such as measurement points from the lidar sensor 208A).

[0060] In another example, the sensor measurement transformation unit 330 can modify specified characteristics of a camera image captured by camera sensor 108B. For example, camera sensor 108B can be specified to include a pixel gain value of 1.8e- / count, and camera sensor 208B can be specified to include a pixel gain value of 2.0e- / count. In such an example, instructions executed by the sensor measurement transformation unit 330 can increase the pixel gain value of a camera image file from sensor measurement database 120 to increase the pixel gain value of the stored image (e.g., by approximately 11%), so as to output an image with a pixel gain value of 2.0e- / count.

[0061] In another example, the sensor measurement transformation component 330 can modify a specified noise and / or characteristic of one or more of the sensors 108B. In one example, the lidar sensor 108A can be specified to include positional uncertainty caused by noise at a measurement point of 0.5 cm, and the lidar sensor 208A can be specified to include positional uncertainty caused by noise at a measurement point of 1.0 cm. In such an example, the sensor measurement transformation component 330 can modify, for example, the measurement points of the lidar measurement point cloud to provide a significant increase in uncertainty in the output data representing the measurement points from the lidar sensor 108A. In another example, the camera sensor 108B can be specified to include a pixel noise content of 1.0 e- / count, and the camera sensor 208B can be specified to include a pixel noise content of 1.5 e- / count. In such an example, instructions executed by the sensor measurement transformation component 330 can add pixel noise to the output data representing the camera image to provide a significant increase in the pixel noise content of the image from the camera sensor 108B. In another example, radar sensors 108C and 108D can be specified to include a range uncertainty of 5.0 cm, and radar sensors 208C and 208D can be specified to include a range uncertainty of 5.5 cm. In such an example, instructions executed by the sensor measurement transformation unit 330 can add the range uncertainty to the output data representing the range (or the velocity of a moving object based on continuous range measurements) to provide a significant increase in the range uncertainty of the objects detected by radar sensors 108C and 108D.

[0062] Figure 4 This is diagram 400 illustrating the differences in the characteristics of the sensors in the first and second systems. Figure 4 In the example, sensor 108 may represent lidar sensor 108A, radar sensor 108C, radar sensor 108D, or another sensor of the first system 100. Sensor 108 may include a designated field of view 405 with an azimuth angle of, for example, 110°, 120°, 135°, etc., and may include a designated maximum detection range of, for example, 200 meters. In one example, sensor 108 has determined that point 420 is located at a distance of approximately 140 meters from sensor 108, point 415 is located at a distance of approximately 145 meters from sensor 108, and point 425 is located at a distance of approximately 160 meters from sensor 108A. The measurements represented by points 415, 420, and 425 may be stored in sensor measurement database 120. Points 415, 420, and 425 may represent, for example, points in a lidar point cloud, or may represent radar signal echoes from static or moving objects in the traffic environment of the first vehicle system 100.

[0063] Figure 4Sensor 208 may represent lidar sensor 208A, radar sensor 208C, radar sensor 208D, or another sensor in the second system 200. Sensor 208 may include a specified field of view 410 smaller than the field of view 405, such as, for example, 90°, 100°, 105°, etc. Additionally, sensor 208 may include a specified maximum detection range of, for example, 150 meters. Therefore, in one example, during the training of machine learning system 230 to identify and / or classify objects within the field of view 410, instructions executed by sensor measurement transformation unit 330 may remove measurement points 415 and 425 outside the field of view of sensor 208. Furthermore, instructions executed by sensor measurement transformation unit 330 may remove sensor measurement points located outside a specified azimuth range of sensor 208. That is, removed sensor measurement points may be omitted from the training data.

[0064] Figure 5 This is a diagram illustrating an example process (500) for modifying a captured image. In one example, a camera sensor 108B of a first vehicle system 100 can capture images of objects in a traffic environment of system 100. The captured images can be stored in a sensor measurement database 120. A sensor measurement transformation unit 330 can access output data representing the image from the sensor measurement database 120 and execute instructions to modify the output data based on data from a sensor characteristics unit 310. In one example, the modification may involve backprojecting the captured image onto an image plane using distortion parameters (e.g., lens distortion parameters that may include spectral shading, skew, etc.) to obtain a raw pixel image at the image plane. The backprojected raw pixel image can then be modified based on specified characteristics obtained from a sensor characteristics unit 320, which applies the distortion parameters of the camera sensor 208B. The image modified according to the sensor characteristics can then be used in the process of training a machine learning system 230 using images that appear to have been captured using the camera sensor 208B.

[0065] Process 500 begins at block 505, which includes capturing images using the camera sensor 108B of the first vehicle system 100. Block 505 may additionally include storing output data representing the captured images in a sensor measurement database 120.

[0066] Process 500 continues at block 510, which includes accessing the characteristics of the camera of the first vehicle system 100. Camera characteristics may include distortion characteristics of the camera lens (e.g., angular displacement of pixels in the image plane based on pixel positions relative to the camera's line of sight in a first direction), spectral filtering of the lens, camera focal length, camera skew, pixel gain value, pixel noise characteristics, pixel color mapping, etc. Characteristics of the camera sensor 108B may additionally include camera azimuth field of view (in degrees), camera sensor range (in meters), etc.

[0067] Process 500 continues at block 515, which includes backprojecting the captured image onto an image plane. During the backprojection process, data representing the captured image may be modified to reverse distortion (e.g., applying angular displacement of pixels in the image plane in a second direction opposite to the first direction based on pixel positions relative to the camera's line of sight, amplifying pixel values ​​to reverse spectrally correlated lens shading, modifying the camera focal length, correcting camera skew and / or inserting different values ​​for camera skew, processing pixel values ​​to insert or reduce noise, inverting pixel color maps, etc.). In one example, block 515 may include performing geometry-based transformation calculations to reference output data representing the captured image from a first designated reference point on the first vehicle system 100 to a second designated reference point on the second vehicle system 200.

[0068] Process 500 continues at block 520, which includes sensor measurement transformation unit 330 accessing sensor characteristics of camera sensor 208B. Camera characteristics may include lens distortion characteristics of camera sensor 208B (e.g., angular displacement of pixels in the image plane based on pixel position relative to the camera's line of sight), spectral filtering of the camera lens, camera focal length, camera skew, pixel gain value, pixel noise characteristics, pixel color mapping, etc. The characteristics of camera sensor 208B may also include camera azimuth field of view (in degrees), camera sensor range (in meters), etc.

[0069] Process 500 continues at box 525, which includes modifying the back-projected image based on the camera characteristics accessed at box 520. Modifying the back-projected image may include applying lens distortion characteristics of the camera sensor 208B, applying spectral correlation amplification of pixel values, modifying the camera focal length, correcting camera skew and / or inserting different values ​​for camera skew, processing pixel values ​​to insert or reduce noise, inverting the pixel color map, etc.

[0070] Process 500 continues at box 530, which includes inputting a modified image to train machine learning system 230. Based on the training input, machine learning system 230 can develop parameters that can be uploaded to a memory accessible to computer 204. Such parameters can help vehicle computer 204 classify static or moving objects represented by point clouds generated by LiDAR measurements, or images (or features within images) of a scene captured via camera sensor 208B.

[0071] Process 500 continues at box 535, which includes uploading parameters to the second vehicle system 200. In one example, the parameters uploaded to the second vehicle system 200 can help the vehicle computer 204 identify and / or classify objects detected by camera sensor 204B.

[0072] After execution box 535, process 500 ends.

[0073] Figure 6 This is a flowchart of an example process for training a network that uploads parameters to a second system. In one example, a lidar sensor 108A of the first vehicle system 100 may output data representing one or more measurement points of a lidar point cloud, which represents objects detected in the traffic environment of the first vehicle system 100. In another example, one or more of the radar sensors 108C, 108D of the first vehicle system 100 may output data representing echo signals reflected from objects in the traffic environment. In such examples, the output data from the sensors (108A, 108C, 108D) may be stored in a sensor measurement database 120. In one example, a sensor measurement transformation unit 330 may access the output data from the sensors and execute instructions to modify the output data based on data from sensor characteristic units 310, 320. In another example, the modification may include modifying the sensor field of view to omit first sensor measurements from the first sensor that are outside the field of view of the second sensor. In yet another example, the modification may include modifying sensor measurements from lidar measurement points or radar signal echoes from objects located at distances outside the range of the second sensor. In other examples, modifications may include adjusting the scanning resolution of the first sensor, increasing or decreasing the noise content of the data from the first sensor, removing data collected at sampling intervals of the first sensor that differ from the sampling rate of the second sensor, etc. After modifications (including omitting removed data, making adjustments, etc.), the sensor measurements can be used to train the machine learning system 230. Based on the training of the machine learning system 230, the machine learning system can upload the parameters to the second vehicle system 200.

[0074] Process 600 begins at block 605, which includes acquiring and / or storing sensor measurements, such as measurement points of a lidar point cloud, returned radar signals, or other signals indicating measurements of objects in the traffic environment. The sensor measurements may be stored in a sensor measurement database 120 of the first vehicle system 100.

[0075] Process 600 continues at block 610, which includes accessing the characteristics of the first sensors (108A, 108C, 108D) of the first vehicle system 100. Characteristics may include the field of view of the first sensor, the detection range of the first sensor, the noise content of measurements from the first sensor, the sampling rate or sampling interval of the first sensor, the scan resolution of measurements performed by the first sensor, etc.

[0076] Process 600 continues at block 615, which includes accessing characteristics of a second sensor (e.g., 208A, 208C, 208D). These characteristics may include the field of view of the first sensor, the detection range of the first sensor, the noise content of measurements from the first sensor, the sampling rate or sampling interval of the first sensor, the scan resolution of measurements performed by the first sensor, etc.

[0077] Process 600 continues at block 620, which includes sensor measurement transformation component 330 modifying sensor measurements to represent sensor measurements collected from second sensors (e.g., 208A, 208C, 208D) of the second vehicle system 200. In one example, modification of sensor measurements may include removing sensor measurements from the first sensor of the first vehicle system 100 that are outside the field of view of the second sensor of the second vehicle system 200. In another example, modification may include modifying sensor measurements from a lidar measurement point or radar signal echoes from an object located at a distance outside the range of the second sensor. In other examples, modification may include adjusting the scan resolution of the first sensor, increasing or decreasing the noise content of data from the first sensor, removing data collected at a sampling interval of the first sensor that differs from the sampling rate of the second sensor, etc.

[0078] Process 600 continues at box 625, which includes training the machine learning system 230 using the sensor data modified at box 620. In one example, training the machine learning system 230 may include thousands or even millions of output datasets using numerous modified LiDAR point clouds, modified radar signal echoes, or other modified output datasets from sensors 108 of the first vehicle system 100. In response to the loss function of the machine learning system 230 being sufficiently minimized, the current parameters (e.g., weights and / or settings formulated or derived from the hidden layers of the machine learning system 230) may be uploaded at box 630 for use by the vehicle computer 204 of the second vehicle system 200.

[0079] After the parameters are uploaded to the vehicle computer 204 of the second system 200, the process 600 ends.

[0080] Figure 7 This is a flowchart of an example process 700 for actuating vehicle components based on parameters uploaded from a training network. In process 700, a second vehicle computer 204 may utilize parameters uploaded from a machine learning system 230 to perform assisted driving for the second vehicle system 200. For example, computer 204 may actuate the display of the second vehicle system 200 to notify the operator of the system 200 of static or moving objects in the traffic environment of the second vehicle system 200. In another example, computer 204 may actuate a steering component, a propulsion component, or another control component without input from the operator of the second vehicle system 200.

[0081] Process 700 begins at block 705, which includes the computer 204 of the second vehicle system 200 acquiring parameters uploaded to a memory accessible to the computer 204.

[0082] Process 700 continues at block 710, where computer 204 actuates the display of the second vehicle system 200 to notify the operator of system 200 of static or moving objects in the traffic environment. This notification may include, for example, displaying text or symbols to categorize static or moving objects, displaying the speed of moving objects, displaying the direction of moving objects, etc. Alternatively or additionally, computer 204 may actuate steering components, propulsion components, etc., based on the second vehicle system 200 being in an auxiliary operating mode, in which computer 204 executes programming to actuate such components.

[0083] After execution box 710, process 700 ends.

[0084] Generally speaking, the described computing system and / or device may employ any of a variety of computer operating systems, including but not limited to the following versions and / or types: Ford Applications; AppLink / Smart Device Connectivity Middleware; Microsoft Operating system; Microsoft Operating system; Unix operating system (e.g., released by Oracle Corporation of Redwood Coast, California). Operating systems: AIX UNIX (published by International Business Machines Corporation, Armonk, New York); Linux; Mac OSX and iOS (published by Apple Inc., Cupertino, California); BlackBerry (published by BlackBerry Ltd., Waterloo, Canada); Android (developed by Google and the Open Handset Alliance); or provided by QNX Software Systems. In-vehicle infotainment platform. Examples of computing devices include, but are not limited to, in-vehicle computers, computer workstations, servers, desktop computers, laptops, mobile computers or handheld computers, or other computing systems and / or devices.

[0085] Computing devices typically include computer-executable instructions, which can be executed by one or more computing devices such as those listed above. Computer-executable instructions can be compiled or interpreted from computer programs created using a variety of programming languages ​​and / or technologies, which, individually or in combination, include, but are not limited to, Java. TM Languages ​​such as C, C++, Matlab, Simulink, Stateflow, Visual Basic, JavaScript, Python, Perl, and HTML are used. Some of these applications can be compiled and executed on virtual machines such as the Java Virtual Machine and the Dalvik Virtual Machine. Generally, a processor (e.g., a microprocessor) receives instructions from, for example, memory, computer-readable media, and executes those instructions to perform one or more processes, including one or more processes described herein. Such instructions and other data can be stored and transferred using various computer-readable media. Files in a computing device are typically collections of data stored on computer-readable media such as storage media, random access memory, etc.

[0086] Computer-readable media (also known as processor-readable media) include any non-transitory (e.g., tangible) medium that contributes to providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, and internals that constitute a system bus coupled to the computer's processor. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, flash EEPROM, any other memory chip or magnetic tape, or any other medium from which a computer can read.

[0087] The databases, data repositories, or other data stores described herein can include various mechanisms for storing, accessing, and retrieving various types of data, including hierarchical databases, file sets in file systems, application databases in proprietary formats, relational database management systems (RDBMS), NoSQL databases, graph databases (GDB), and so on. Each such data store is typically contained within a computing device employing a computer operating system such as those mentioned above, and can be accessed via a network in any of a variety of ways. File systems can be accessed from the computer operating system and can include files stored in various formats. In addition to languages ​​used for creating, storing, editing, and executing stored programs (such as the PL / SQL language mentioned above), RDBMS typically employs Structured Query Language (SQL).

[0088] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) and stored on computer-readable media (e.g., disks, storage, etc.) associated therewith. Computer program products may include such instructions stored on computer-readable media for performing the functions described herein.

[0089] In the accompanying drawings, the same reference numerals indicate the same elements. Furthermore, some or all of these elements may be changed. Regarding the media, processes, systems, methods, inspirations, etc., described herein, it should be understood that although the steps of such processes, etc., are described as occurring in a certain ordered order, such processes can be practiced by performing the steps in an order different from that described herein. It should also be understood that some steps may be performed simultaneously, other steps may be added, or some steps described herein may be omitted. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable owner / user manuals and / or safety guidelines.

[0090] This disclosure has been described in an illustrative manner, and it should be understood that the terminology used is intended to describe the nature of the words, not to be restrictive. The adjectives “first” and “second” are used throughout this document as identifiers and are not intended to indicate importance, order, or quantity. The use of “in response to” and “after determining…” indicates a causal relationship, not merely a temporal one. In view of the foregoing teachings, many modifications and variations of this disclosure are possible, and this disclosure may be practiced in ways other than those specifically described.

[0091] According to the present invention, a method includes: actuating components of a device based on parameter output from a machine learning application, the machine learning application being trained with first output data from a first sensor, the first output data being (1) modified according to a first specified characteristic of the first sensor, and (2) modified according to a second specified characteristic of a second sensor.

[0092] In one aspect of the invention, the first specified characteristic of the first sensor is the noise content of the output data of the first sensor, the field of view of the first sensor, the detection range of the first sensor, the resolution of the first sensor, or the sampling interval of the first sensor.

[0093] In one aspect of the invention, the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, references the first output data from a first specified reference point on the first vehicle to a second specified reference point on the second vehicle.

[0094] In one aspect of the invention, the first sensor is a first lidar sensor, and the second sensor is a second lidar sensor.

[0095] In one aspect of the invention, the first specified characteristic is the field of view of the first lidar sensor, wherein the second specified characteristic is the field of view of the second lidar sensor, and wherein the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, has been processed to omit measurement points within the field of view of the first lidar sensor but outside the field of view of the second lidar sensor.

[0096] In one aspect of the invention, the first specified characteristic is the detection range of the first lidar sensor, wherein the second specified characteristic is the detection range of the second lidar sensor, and wherein the first output data from the first sensor, which has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor, omits measurement points representing distances outside the detection range of the second lidar sensor.

[0097] In one aspect of the invention, the first specified characteristic is a first scanning interval of the first lidar sensor, wherein the second specified characteristic is a second scanning interval of the second lidar sensor, and wherein the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, omits measurement points collected outside the second scanning interval during the first scanning interval.

[0098] In one aspect of the invention, the first specified characteristic is a first scanning resolution of the first lidar sensor, and wherein the second specified characteristic is a second scanning resolution of the second lidar sensor, and wherein the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, is omitted for measurement points collected at the first scanning resolution, which is different from the second scanning resolution.

[0099] In one aspect of the invention, the first sensor is a first radar sensor, and the second sensor is a second radar sensor.

[0100] In one aspect of the invention, the first specified characteristic is a first detection range of the first radar sensor, wherein the second specified characteristic is a second detection range of the second radar sensor, and wherein the first output data from the first sensor, which has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor, omits measurement points outside the second detection range of the first detection range.

[0101] In one aspect of the invention, the first specified characteristic is a first scanning interval of the first radar sensor, wherein the second specified characteristic is a second scanning interval of the second radar sensor, and wherein the first output data from the first sensor, which has been modified (1) according to the first specified characteristic of the first sensor and (2) according to the second specified characteristic of the second sensor, aligns the first radar scanning interval with the second radar scanning interval.

[0102] In one aspect of the invention, the first sensor is a first camera sensor, and the second sensor is a second camera sensor.

[0103] In one aspect of the invention, the first specified characteristic is a first distortion parameter of the first camera sensor, wherein the second specified characteristic is a second distortion parameter of the second camera sensor, and wherein the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, is: applied to generate the pixel position of the first back projection using the first distortion parameter; and the second distortion parameter is applied to the pixel position of the generated first back projection.

[0104] In one aspect of the invention, the first specified characteristic is a first pixel gain parameter of the first camera sensor, wherein the second specified characteristic is a second pixel gain parameter of the second camera sensor, and wherein the first output data from the first sensor that has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor: applies the first pixel gain parameter to generate a pixel gain value of a first back projection; and applies the second pixel gain parameter to the pixel gain value of the first back projection.

[0105] According to the present invention, a system is provided having a computer including a processor and a memory storing instructions executable by the processor to: output components of an actuating device based on parameters trained with first output data from a first sensor, the first output data being (1) modified according to a first specified characteristic of the first sensor and (2) modified according to a second specified characteristic of a second sensor.

[0106] According to one embodiment, the first specified characteristic of the first sensor is the noise content of the output data from the first sensor, the field of view of the first sensor, the detection range of the first sensor, the resolution of the first sensor, or the sampling interval of the first sensor.

[0107] According to one embodiment, the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, references the first output data from a first specified reference point on the first vehicle to a second specified reference point on the second vehicle.

[0108] According to one embodiment, the first sensor is a first radar sensor, and the second sensor is a second radar sensor.

[0109] According to one embodiment, the first sensor is a first camera sensor, and the second sensor is a second camera sensor.

[0110] According to one embodiment, the first sensor is a first lidar sensor, and the second sensor is a second lidar sensor.

Claims

1. A method comprising: The components of the device are actuated based on parameter outputs from a machine learning application, which is trained with first output data from a first sensor, the first output data having been (1) modified according to a first specified characteristic of the first sensor and (2) modified according to a second specified characteristic of a second sensor.

2. The method of claim 1, wherein the first specified characteristic of the first sensor is the noise content of the output data from the first sensor, the field of view of the first sensor, the detection range of the first sensor, the resolution of the first sensor, or the sampling interval of the first sensor.

3. The method of claim 1, wherein the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, references the first output data from a first specified reference point on the first vehicle to a second specified reference point on the second vehicle.

4. The method of claim 1, wherein the first sensor is a first lidar sensor, and wherein the second sensor is a second lidar sensor, wherein the first specified characteristic is the field of view of the first lidar sensor, wherein the second specified characteristic is the field of view of the second lidar sensor, and wherein the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, has been processed to omit measurement points within the field of view of the first lidar sensor but outside the field of view of the second lidar sensor.

5. The method of claim 1, wherein the first sensor is a first lidar sensor, and wherein the second sensor is a second lidar sensor, wherein the first specified characteristic is the detection range of the first lidar sensor, wherein the second specified characteristic is the detection range of the second lidar sensor, and wherein the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, omits measurement points representing distances outside the detection range of the second lidar sensor.

6. The method of claim 1, wherein the first sensor is a first lidar sensor, and wherein the second sensor is a second lidar sensor, wherein the first specified characteristic is a first scan interval of the first lidar sensor, wherein the second specified characteristic is a second scan interval of the second lidar sensor, and wherein the first output data from the first sensor that has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor omits measurement points collected outside the second scan interval during the first scan interval.

7. The method of claim 1, wherein the first sensor is a first lidar sensor, and wherein the second sensor is a second lidar sensor, wherein the first specified characteristic is a first scan resolution of the first lidar sensor, and wherein the second specified characteristic is a second scan resolution of the second lidar sensor, and wherein the first output data from the first sensor that has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor is omitted for measurement points collected at the first scan resolution which is different from the second scan resolution.

8. The method of claim 1, wherein the first sensor is a first radar sensor, and wherein the second sensor is a second radar sensor.

9. The method of claim 8, wherein the first specified characteristic is a first detection range of the first radar sensor, wherein the second specified characteristic is a second detection range of the second radar sensor, and wherein the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, omits measurement points outside the second detection range of the first detection range.

10. The method of claim 8, wherein the first specified characteristic is a first scan interval of the first radar sensor, wherein the second specified characteristic is a second scan interval of the second radar sensor, and wherein the first output data from the first sensor, which has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor, aligns the first radar scan interval with the second radar scan interval.

11. The method of claim 1, wherein the first sensor is a first camera sensor, and wherein the second sensor is a second camera sensor.

12. The method of claim 11, wherein the first specified characteristic is a first distortion parameter of the first camera sensor, wherein the second specified characteristic is a second distortion parameter of the second camera sensor, and wherein the first output data from the first sensor has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor: The first distortion parameter is applied to generate the pixel position of the first back projection; and The second distortion parameter is applied to the pixel position of the generated first back projection.

13. The method of claim 11, wherein the first specified characteristic is a first pixel gain parameter of the first camera sensor, wherein the second specified characteristic is a second pixel gain parameter of the second camera sensor, and wherein the first output data from the first sensor has been (1) modified according to the first specified characteristic of the first sensor and (2) modified according to the second specified characteristic of the second sensor: The first pixel gain parameter is applied to generate the pixel gain value of the first back projection; and The second pixel gain parameter is applied to the pixel gain value of the first back projection.

14. A computer programmed to perform the method as described in any one of claims 1 to 13.

15. A vehicle comprising the computer as described in claim 14.

Images