PICTURE-BASED OPERATION WITH MACHINE LEARNING

Abstract

Fisheye images containing objects at first, second, and third angles are transformed into rectilinear images using a first image transformation. These rectilinear images are then transformed into bird's-eye views using a second image transformation. The bird's-eye views can be further transformed into multiple images containing objects at various angles between the first, second, and third angles using a third image transformation. This generates a training dataset that provides ground truth regarding objects at multiple angles. A machine learning model can then be trained on this dataset. Finally, a machine, such as a vehicle, can be operated using output from this machine learning model.

Description

AREA OF TECHNOLOGY

[0001] This revelation concerns image-based operation with machine learning. GENERAL STATE OF THE ART

[0002] Computers can operate systems and devices that include vehicles, robots, drones, and / or object tracking systems. Data, including images, can be captured by sensors and processed by a computer to determine a trajectory for a system in relation to its environment and to objects within that environment. A computer can then use this trajectory to operate the system or to control its components within the environment. SUMMARY

[0003] Systems that move and / or have mobile components, including vehicles, robots, drones, mobile phones, etc., can be operated by acquiring sensor data, including data about the environment surrounding the system, and processing this sensor data to determine the locations of objects in the environment around the system. The determined position data could then be processed to determine the operation of the system or parts of the system. For example, a robot could determine the position of the arm of another nearby robot. The robot could then use this determined robot arm position to determine a path along which a gripper should be moved to grasp a workpiece without colliding with the other robot's arm.In another example, a vehicle could determine its location relative to its surroundings and the locations of objects, such as the roadway and other vehicles, within those surroundings. The vehicle could use its determined location and the locations of the objects to determine a path along which to operate, while maintaining a predetermined relationship to the objects. Vehicle operation is used here as a non-restrictive example of object identity and location determination in the description below.

[0004] A machine learning model can be trained and installed in a computer within a vehicle to receive sensor data from sensors integrated into the vehicle. The machine learning model can then make predictions based on the received sensor data to assist in vehicle operation. For example, a machine learning model can be trained to receive images from a video camera and determine a predicted state for objects in the vehicle's surroundings. A predicted state output from the machine learning model might include predictions of an object's location and orientation relative to the vehicle, encompassing the distance and angle between the vehicle and the object.The object prediction data can be used by a computing device included in the vehicle to determine a trajectory the vehicle could travel to reach a predicted future location. The computing device can then guide the vehicle along the trajectory by issuing commands to controllers that operate vehicle components such as the drive, steering, and brakes, as described below. Fig. 1 described.

[0005] In an example of operating a vehicle based on a trained machine learning model, a rear-facing video camera mounted in the vehicle can capture images of a trailer parked behind the vehicle. By determining the location and orientation of the trailer's coupling section relative to a tow ball attached to the vehicle, a machine learning model can determine a vehicle trajectory. This trajectory can then be translated by the computing device into commands to be sent to controllers located within the vehicle to issue commands to vehicle components. These components can be commanded to operate the vehicle, moving the tow ball to a position beneath the coupling, allowing the coupling to be lowered onto the tow ball, and connecting the trailer to the vehicle for towing.

[0006] Obtaining useful results from a trained machine learning system can depend on the system's ability to generalize a training dataset to produce useful results based on real-world input data. In the context of this application, useful results are those that operate the vehicle to achieve a goal, such as placing a coupling ball under a coupling coupler, while maintaining limits on vehicle speed, rates of change of speed and direction, and braking force. Generating a training dataset that includes a range of trailer types, trailer locations and orientations, and environmental conditions, including lighting and weather, can require thousands or millions of images.Each image must be processed to determine ground truth regarding the location and orientation of the coupling coupler relative to the coupling ball in order to train the machine learning model. Ground truth refers to data acquired independently of the machine learning model training process. For example, the location and orientation of the coupling coupler can be physically measured at the time the image data is acquired. In other examples, image processing software, such as Adobe Photoshop, can be used to determine the location and orientation of a coupling coupler in real-world coordinates. Adobe Photoshop is available from Adobe, Inc., at Adobe.com as of the filing date of this application. Acquiring and generating ground truth for a comprehensive dataset of real-world images for training a machine learning model can require more time and computational resources than are available.

[0007] Another technique for generating a training dataset is the generation of simulated images. An example of a software program for generating a training dataset of photorealistically rendered images to train a machine learning model is Unreal Engine, available at unrealengine.com as of the filing date of this application by Epic Games, Inc. Photorealistically rendered images have the advantage that the input data used to generate the image data includes the ground truth regarding the location and orientation of objects in the environment surrounding the vehicle. A potential disadvantage of training a machine learning model using simulated images is domain shift. Domain shift occurs when there is a mismatch between data in a training domain and data in a target domain where the machine learning model is used, such as simulated images versus real-world images.Domain shift can cause a machine learning model to, for example, misidentify or mislocate objects.

[0008] The training dataset generation techniques described here can improve machine learning model training by generating images for training datasets based on a limited number of captured real-world images. The generated images include ground-truth data, reducing the need to label large numbers of images for training datasets and thus reducing the computational resources typically required to generate large training datasets. Furthermore, generating training images based on real-world images, rather than simulated images, can mitigate domain shift. Domain shift occurs when the images used to train a machine learning model differ in appearance from images captured at inference time.For example, using simulated images rendered by a software program for training, and using real images at inference time.

[0009] Generating simulated images through rendering can require large amounts of computing resources. Generating simulated images that attempt to mitigate domain shift by increasing resolution and detail can require even greater amounts of computing resources and may be unsuccessful. Simulated images can be made more realistic by processing them with generative adversarial networks, which can increase the amount of computing resources used to generate training images. Machine learning models can be trained to compensate for domain shift by using dual networks and cross-correlated latent intermediate variables when forming loss functions, which again increases the required computing resources.The techniques described herein for generating training datasets mitigate domain shift without increasing the required computing resources. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram of an example image-based system. Fig. Figure 2 is a representation of an exemplary fisheye image. Fig. Figure 3 is a representation of an exemplary rectilinear image. Fig. Figure 4 is a representation of an exemplary bird diagram. Fig. Figure 5 is an example of a rotated image. Fig. Figure 6 is a representation of an exemplary synthetic image. Fig. Figure 7 is a representation of an exemplary system for transforming images. Fig. Figure 8 is a representation of an exemplary system for training a machine learning model. Fig. Figure 9 is a flowchart for operating a vehicle based on the training of a machine learning model. DETAILED DESCRIPTION

[0010] The techniques described herein for generating training datasets begin with capturing a limited number of representative images for each type of object to be identified and located by a machine learning model. In this example, one type of object might be a brand and model of a pendant. Representative images might be the pendant in three cardinal positions—zero degrees, 90 degrees, and 180 degrees—or within ±10 degrees of those cardinal positions. The images can be determined by examining captured video data of pendants and manually selecting and labeling the representative images. Once the representative images are captured and labeled, a software program running on a server computer can transform the representative images, generate intermediate frames, and assemble them into a training dataset, as described below. Fig. 2-6 described. A second software program, running on the server computer, can train the machine learning model using the training dataset, as described below in relation to the following. Fig. 7 described.

[0011] This document discloses a method that involves transforming fisheye images containing objects at first, second, and third angles into rectilinear images using a first image transformation. The rectilinear images can be transformed into bird's-eye views using a second image transformation. The bird's-eye views can then be transformed into multiple images containing objects at multiple angles between the first, second, and third angles using a third image transformation, in order to generate a training dataset that provides ground truth regarding the objects at multiple angles. A machine learning model can be trained on this training dataset. The first image transformation can be based on intrinsic fisheye camera parameters that include fisheye distortion parameters.The second image transformation can be based on intrinsic camera parameters, including focal length in x and y, optical center in x and y, magnification, optical center in x and y, and offset.

[0012] The second image transformation can be based on extrinsic camera parameters, including a camera pose with six degrees of freedom. This second image transformation can involve an affine transformation that places a coupling ball at a predetermined location within the images. The first angle can be 0 degrees, the second 90 degrees, and the third 180 degrees. The third image transformation can be based on generating intermediate angle images in 10-degree increments between 0 degrees and 180 degrees. The machine learning model can be a convolutional neural network. The objects can include a trailer. The first, second, and third angles can be based on the angle of a trailer hitch relative to the location of a coupling ball. The machine learning model can be trained to determine the location and angle of the trailer hitch relative to the location of the coupling ball.The trained machine learning model can be housed in a second computer for a vehicle, with the second computer programmed to operate the vehicle by determining a vehicle trajectory based on predictions output by the trained machine learning model. The second computer can be programmed to operate the vehicle along the vehicle trajectory by issuing commands to control system components. The convolutional neural network can include multiple convolutional layers and multiple fully connected layers.

[0013] Furthermore, a computer-readable medium is disclosed that stores program instructions for executing some or all of the preceding process steps. A computer programmed to execute some or all of the preceding process steps is also disclosed, comprising a computer device programmed to transform fisheye images containing objects at first, second, and third angles into rectilinear images using a first image transformation. The rectilinear images can be transformed into bird's-eye views using a second image transformation. The bird's-eye views can be transformed into multiple images containing objects at multiple angles between the first, second, and third angles using a third image transformation to generate a training dataset containing ground truth regarding the objects at multiple angles.A machine learning model can be trained using the training dataset. The first image transformation can be based on intrinsic fisheye camera parameters, including fisheye distortion parameters. The second image transformation can be based on intrinsic camera parameters, including focal length in x and y, optical center in x and y, magnification, and offset.

[0014] The instructions can also include instructions where the second image transformation can be based on extrinsic camera parameters, including a camera pose with six degrees of freedom. The second image transformation can include an affine transformation that places a coupling ball at a predetermined location in the images. The first angle can be 0 degrees, the second angle 90 degrees, and the third angle 180 degrees. The third image transformation can be based on generating intermediate angle images in 10-degree increments between 0 degrees and 180 degrees. The machine learning model can be a convolutional neural network. The objects can include a trailer. The first, second, and third angles can be based on the angle of a trailer hitch relative to the location of a coupling ball.The machine learning model can be trained to determine the position and angle of the trailer drawbar relative to the location of the coupling ball. The trained machine learning model can be implemented in a second computer for a vehicle, which is programmed to operate the vehicle by determining a vehicle trajectory based on predictions output by the trained machine learning model. The second computer can be programmed to operate the vehicle along this trajectory by issuing commands to control system components. The convolutional neural network can include multiple convolutional layers and multiple fully connected layers.

[0015] Fig. Figure 1 is a representation of an image-based system 100. In this example, the system 100 includes a vehicle 110, but in other examples, the system 100 could include other devices that move and / or have movable components, such as a robot, a drone, or an object tracking device. In examples where the system 100 includes a robot, a drone, or an object tracking device, the controllers 112, 113, 114 would be changes to controllers that control components of the robot, drone, or object tracking device. In the examples described in this document, the system 100 includes a vehicle 110, a computing device 115 contained in the vehicle 110, and a server computer 120 located remotely from the vehicle 110. One or more computing devices 115 of the vehicle 110 can receive data regarding the operation of the vehicle 110 from sensors 116.The computing device 115 can operate the vehicle 110 based on data received from the sensors 116 and data received from the remote server computer 120. The server computer 120 can communicate with the vehicle 110 via a network 130.

[0016] The computing device 115 includes a processor and a memory, as known. Furthermore, the memory includes one or more forms of computer-readable media and stores instructions that can be executed by the processor to perform various operations, including those disclosed herein. For example, the computing device 115 may include programming to operate one or more of the vehicle's brakes, drive (i.e., control of speed in the vehicle 110 by controlling one or more internal combustion engines, electric motors, hybrid motors, etc.), steering, air conditioning, interior and exterior lights, etc., and to determine whether and when the computing device 115 should control such operations, as opposed to a human driver.The computing device 115 can also control the temporal alignment of the lighting to the sensor detection in order to take into account the color effects of vehicle lights or exterior lights.

[0017] The computing device 115 can include more than one computing device, i.e., controllers or the like, which are incorporated in the vehicle 110 for monitoring and / or controlling various vehicle components, i.e., a drive control 112, a brake control 113, a steering control 114, etc., or be communicatively coupled with this / these, i.e., via a vehicle communication bus, as described in more detail below. The computing device 115 is generally arranged for communication on a vehicle communication network, i.e., including a bus in the vehicle 110, such as a Controller Area Network (CAN) or the like; the network of the vehicle 110 can additionally or alternatively include wired or wireless communication mechanisms of a kind known to exist, i.e., Ethernet or other communication protocols.

[0018] The computing device 115 can transmit messages to various devices in the vehicle 110 via the vehicle network and / or receive messages from the various devices, e.g., controllers, actuators, sensors, etc., including the sensors 116. Alternatively or additionally, in cases where the computing device 115 actually comprises several devices, the vehicle communication network can be used for communication between devices that are referred to in this disclosure as the computing device 115. Furthermore, as mentioned below, various controllers or sensing elements, such as the sensors 116, can provide data to the computing device 115 via the vehicle communication network.Additionally, the computing device 115 can be configured to communicate with a remote server computer 120, i.e., a cloud server, via a network 130, which, as described below, includes hardware, firmware, and software that allow the computing device 115 to communicate with a remote server computer 120 via a network 130, such as wireless internet (WI-FI®) or cellular networks, through a vehicle-to-infrastructure (V2I) interface 111. The V2X interface 111 can accordingly include processors, memory, transceivers, etc., configured to utilize various wired and / or wireless network technologies, i.e., cellular, BLUETOOTH®, Bluetooth Low Energy (BLE), Ultra-Wide Band (UWB), peer-to-peer communication, UWB-based radar, IEEE 802.11, and other wired and / or wireless packet networks or technologies.The computing device 115 can be configured to communicate with other vehicles 110 via the V2X (vehicle-to-everything) interface 111 using vehicle-to-vehicle (V-to-V) networks, i.e., cellular communications (C-V2X), wireless cellular communications, dedicated short-range communications (DSRC), and the like, i.e., networks that are formed ad hoc between vehicles 110 in close proximity or via infrastructure-based networks. The computing device 115 also includes non-volatile memory, as known. The computing device 115 can log data in non-volatile memory for later retrieval and transmission via the vehicle communication network and a vehicle infrastructure (V2I) interface 111 to a server computer 120 or a mobile user device 160.

[0019] As already mentioned, instructions stored in memory and executable by the processor of the computing device 115 generally include programming to operate one or more components of the vehicle 110, i.e., braking, steering, propulsion, etc., without human intervention. Using data received by the computing device 115, i.e., sensor data from the sensors 116, the server computer 120, etc., the computing device 115 can make various decisions and / or control various components and / or operations of the vehicle 110. For example, the computing device 115 can program the operation of the vehicle 110 (i.e., physical manifestations of the vehicle's operation), such as speed, steering, etc., and its tactical behavior (i.e.,Control of operational behavior in a manner typically intended to achieve efficient route driving), such as a distance between vehicles and / or a time interval between vehicles, lane changes, a minimum distance between vehicles, a minimum time to cross the road when turning left, a time to arrive at a specific location, and a minimum time to cross the intersection at an intersection (without traffic lights).

[0020] For the purposes of this text, controllers include computing devices typically programmed to monitor and control a specific vehicle subsystem. Examples include a powertrain controller 112, a brake controller 113, and a steering controller 114. A controller may be an electronic control unit (ECU) of any kind, which may include additional programming as described herein. The controllers may be communicatively connected to the computing device 115 and receive instructions from it to operate the subsystem in accordance with those instructions. For example, the brake controller 113 may receive instructions from the computing device 115 to operate the vehicle's brakes 110.

[0021] The one or more controllers 112, 113, 114 for the vehicle 110 may include known electronic control units (ECUs) or the like, which, as non-limiting examples, include one or more drive controllers 112, one or more brake controllers 113, and one or more steering controllers 114. Each of the controllers 112, 113, 114 may include respective processors and memory and one or more actuators. The controllers 112, 113, 114 may be programmed and connected to a communication bus of the vehicle 110, such as a Controller Area Network (CAN) bus or a Local Interconnect Network (LIN) bus, to receive instructions from the computing device 115 and to control actuators based on these instructions.

[0022] The sensors 116 can include a variety of devices known for providing data via the vehicle communication bus. For example, a radar mounted on a front bumper (not shown) of the vehicle 110 can provide a distance from the vehicle 110 to the next vehicle in front of it, or a sensor for a global positioning system (GPS) located in the vehicle 110 can provide geographic coordinates of the vehicle 110. The distance(s) provided by the radar and / or the other sensors 116 and the geographic coordinates provided by the GPS sensor can be used by the computing device 115 to operate the vehicle 110, for example, autonomously or semi-autonomously.

[0023] The vehicle 110 is generally a land vehicle 110 capable of autonomous and semi-autonomous operation and has three or more wheels, i.e., a passenger car, a light truck, etc. The vehicle 110 includes one or more sensors 116, the V2I interface 111, the computing device 115, and one or more controllers 112, 113, 114. The sensors 116 can collect data relating to the vehicle 110 and the environment in which the vehicle 110 operates. By way of example, and not limited to, the sensors 116 can include, for example, altimeters, cameras, LiDAR, radar, ultrasonic sensors, infrared sensors, pressure sensors, accelerometers, gyroscopes, temperature sensors, Hall sensors, optical sensors, voltage sensors, current sensors, mechanical sensors such as switches, etc. The sensors 116 can be used to detect the environment in which the vehicle 110 is operated, i.e.The sensors 116 can detect phenomena such as weather conditions (precipitation, ambient temperature, etc.), the gradient of a road, the location of a road (i.e., using road edges, lane markings, etc.), or the locations of target objects, such as neighboring vehicles 110. The sensors 116 can also be used to collect data that includes dynamic data about the vehicle 110 relating to its operations, such as speed, yaw rate, steering angle, engine speed, brake pressure, oil pressure, power applied to the controls 112, 113, 114 in the vehicle 110, connectivity between components, and the precise and current performance of the vehicle 110's components.

[0024] The server computer 120 typically has features that it shares with the V2I interface 111 of the vehicle 110 and the computing device 115 (e.g., a computer processor and memory, and a configuration for communication over a network 130), and these features are therefore not described in detail to reduce redundancy. A server computer 120 can be used to develop and train machine learning models that can be transferred to a computing device 115 in a vehicle 110.

[0025] Fig. Figure 2 shows an example fisheye image 200, captured by a fisheye camera included in or on the vehicle 110. A fisheye camera includes an ultra-wide-angle lens (fisheye lens) that captures images with an extremely wide field of view. Fisheye cameras are included in the vehicle 110 because they can capture image data from a field of view that would require two or more cameras with rectilinear lenses to cover. The fisheye image 200 includes a trailer 202, which is attached to the vehicle 110 via a trailer drawbar 210 and a trailer coupler 206. The trailer coupler 206 rests on the coupling ball 208, which is connected to the vehicle 110 and a bumper 204 by a trailer hitch 212.

[0026] Despite its advantage of covering a large field of view, a fisheye image 200 has the disadvantage of distorting objects within that field of view. A convex distortion inherent in the fisheye image 200 can cause lines that are straight in the real world to appear curved in the fisheye image 200, for example, the edges of the bumper 204. Furthermore, the object distortion varies depending on the object's position within the field of view, making it difficult to process the fisheye image 200 with a machine learning model. To overcome this difficulty, the fisheye image 200 can be transformed into a rectilinear image using a fisheye-to-rectilinear transformation.

[0027] Capturing a fisheye image 200 with a fisheye camera can be mathematically described as first projecting world coordinates, i.e. global coordinates included in a real traffic scene, into camera coordinates, i.e. coordinates measured relative to the camera sensor plane: [XCYCZC]=cRW(XWYWZW)+ctW

[0028] In equation 1, X W , Y W , Z W The three axis coordinates of a point in real coordinates are X C , Y C , Z C the three axis coordinates of a point in camera coordinates, is C R W a 3x3 rotation matrix that rotates a point in three-dimensional space, and is C t WA 1x3 matrix that translates a point in three-dimensional space. Imaging a point in three-dimensional space with a fisheye lens can be modeled by projecting the point onto a unit sphere using the following equation: [XSYSZS]=[XCXC2+YC2+ZC2YCXC2+YC2+ZC2ZCXC2+YC2+ZC2]

[0029] In equation 2, X S , Y S , Z S The three axis coordinates of a point projected onto the unit sphere. The point on the unit sphere is then projected onto a normalized plane to obtain normalized coordinates x, as described by the following equation. ud , Y ud to result: [xudyud]=[XSZS+ξYSZS+ξ]

[0030] Distortion parameters related to the distortion of the fisheye lens k1, k2, p1, p2 can be estimated by determining the intrinsic calibration of the fisheye lens. The intrinsic calibration includes the parameters that determine the distortion of the fisheye lens, which occurs in addition to the distortion due to the spherical lens. The distortion parameters of the fisheye lens are applied to the normalized coordinates to obtain the undistorted coordinates x. ud , y ud in distorted coordinates x d , y d to transform: [xdyd]=[xud(1+k1(x2+y2)+k2(x2+y2)2)+2p1xudyud+p2((x2+y2)+2x2)yid(1+k1(x2+y2)+k2(x2+y2)2)+2p2xudyud+p1((x2+y2)+2x2)]

[0031] A generalized camera projection matrix that converts the recorded normalized fisheye coordinates into camera coordinates. p=[uv] converts, using camera parameters for focal length f x, f y in x and y, optical center c x , c y in x and y and offset s: p=[uv]=[fsscx0fycy][xdyd1]

[0032] Applying equations (1)-(5) to the real coordinates X W , Y W , Z W This can yield camera coordinates p, i.e., applying equations (1)-(5) to a real scene can yield a fisheye image 200. Equations (1)-(5) can be combined by the following equation: F(p)=∏(∅) where F(p) is a fisheye image, Π is the transformation that includes equations (1)-(5), and Ø is a set of data points in three-dimensional real coordinates. The fisheye-to-rectilinear transformation that transforms the fisheye image 200 into a rectilinear image 300, as in Fig. Figure 3 illustrates that it is based on reversing the above equations (1)-(5) by inverting the matrix operations in equations (1)-(5).

[0033] Inverting the matrix operations in equations (1)-(5) can be based on intrinsic camera parameters of the fisheye camera, which include the lens used to capture the fisheye image 200. Intrinsic parameters include the camera focal length f. x , f y in x and y, magnification, optical center c x , c yin x and y and offset s, where, if present, is an angular difference of 90 degrees that defines the x and y dimensions. Inverting the matrix operations can be based on intrinsic fisheye camera parameters. Intrinsic parameters include fisheye distortion parameters k1, k2, p1, p2, which can be determined by capturing an image of a specified pattern, such as a chessboard, at a specified distance from the camera and analyzing the resulting pattern.

[0034] Inverting the matrix can also be based on extrinsic camera parameters. These extrinsic fisheye camera parameters include the fisheye camera's location in real x, y, and z coordinates and the camera's orientation in roll, pitch, and yaw coordinates relative to the x, y, and z axes. Extrinsic camera parameters include location coordinates in x, y, and z and camera rotation coordinates in roll, pitch, and yaw, defining a camera pose with six degrees of freedom. The extrinsic parameters can be measured relative to a ground plane, for example, a roadway on which the vehicle 110, containing the fisheye camera, is supported.

[0035] Fig. Figure 3 is a representation of a rectilinear image 300, which is obtained by transforming the fisheye image 200 using the above in relation to Fig. The fisheye-to-rectilinear transformation described in Section 2 is determined. The rectilinear image 300 contains the same elements as the fisheye image 200, namely a trailer 302, a vehicle bumper 304, a trailer drawbar 310, and a trailer coupler 306, which connect the trailer 202 to the vehicle 110 via a coupling ball 308 and a trailer hitch 312. Although the rectilinear image 300 is free of fisheye distortion, it still contains perspective distortion, which changes the apparent size, shape, and location of objects depending on their distance from the camera. For example, the bumper 304 is altered from its actual rectangular shape and appears larger than the trailer 302. The perspective distortion changes with the location of objects relative to the optical center of the image.Changing the shape, size, and location of objects can lead to variance in the results obtained by a trained machine learning model.

[0036] Techniques for generating training datasets for machine learning models, discussed herein, can mitigate the effects of perspective distortion in rectilinear images by performing a rectilinear-to-bird's-eye view transformation. A rectilinear-to-bird's-eye view transformation uses intrinsic and extrinsic camera parameters to transform a rectilinear image, captured by a camera located within a vehicle, into a bird's-eye view, as described in Fig. Figure 4 illustrates this. Homography is a type of image transformation that describes the relationship between two images of the same planar object taken from different positions. Determining a bird's-eye view 400 from a rectilinear image can be done by applying a homography matrix H to the pixels of the rectilinear image 300 R to form a bird's-eye view 400 B using matrix multiplication: B=H∗R

[0037] Where the homography matrix H is a 3x3 matrix: H=|h11h12h13h21h22h23h31h32h33|

[0038] where the elements h ijThe homography matrix H is determined based on the focal length of the video camera in x and y, the vanishing point of the image, the horizon line (defined with respect to a ground plane), and the rotation and tilt of the video camera relative to the ground plane. The determination of the homography matrix H is described in "A Geometric Approach to Obtain a Bird's Eye View from an Image" by Ammar Abbas and Andrew Zisserman. This article is available at https: / / arxiv.org / abs / 1905.02231 as of the filing date of this application.

[0039] Fig. Figure 4 is a representation of a bird's-eye view 400. The bird's-eye view 400 is derived from a rectilinear image 300 based on the information relating to Fig. The transformation described in section 3 is generated. The bird's-eye view 400 includes a bumper 404 attached to a vehicle 110, a coupling coupler 406, and a trailer drawbar 410. The coupling coupler 406 and the trailer drawbar 410 can be connected to the vehicle 110 via a coupling ball 408 located below the coupling coupler 406 and the trailer coupling 412. The bird's-eye view 400 allows for more accurate processing by a machine learning model by reducing the perspective distortion present in the rectilinear image 300. In particular, the bird's-eye view 400 enables a more accurate determination of the trailer angle 414 between the coupling coupler 406 and the bumper 404 by a machine learning model than in the rectilinear image 300, which contains perspective distortion.

[0040] A bird's-eye view 400 can be further improved to allow for an accurate determination of the trailer angle 414 by shifting the pixels of the bird's-eye view 400 to position the center of the coupling ball 408 at a predetermined location within the bird's-eye view 400. Since extrinsic and intrinsic camera parameters are determined at the time of manufacture, the location and orientation of the coupling ball 408 within the bird's-eye view 400 can be determined. To improve the accuracy of determining the trailer angle 414 using a machine learning model, the pixels of the bird's-eye view 400 can be shifted and rotated by image processing software that performs an affine transformation to position the coupling ball 408 at a predetermined location and in a predetermined orientation within the bird's-eye view 400.The bird's-eye view 400 can also be adjusted for the field of view by changing the zoom factor so that the trailer 402, the trailer drawbar 410 and the coupling coupler 406 are made the same size in the bird's-eye views 400.

[0041] The techniques described herein can improve the training of a machine learning model to determine the trailer angle 414 by ensuring that the coupling ball 408 is in the same location and orientation, and that the trailer 402, the trailer drawbar 410, and the coupling coupler 406 are of the same size and location during training and inference. By ensuring that the coupling ball 408 is in the same location and orientation, and that the trailer 402, the trailer drawbar 410, and the coupling coupler 406 are of the same size and location during training and inference, the training time can be reduced, which in turn reduces the computational resources required to train the machine learning model, and the accuracy of determining the trailer angle 414 at the time of inference can be increased.

[0042] Fig. Figure 5 is a representation of a rotated image 500. The rotated image 500 is formed by rotating an entire bird's-eye view 400 using image processing software that performs an affine transformation on the pixels of the bird's-eye view 400. Following the rotation, sections of the bird's-eye view 400 that have been rotated out of the rectangular frame of the rotated image 500 can be cropped. For example, the first trailer angle 414 can be 90 degrees. The bird's-eye view 400 can be rotated 80 degrees clockwise around the location of the coupling ball 508 to form, for example, a rotated image 500 that involves rotating the vehicle 110, the bumper 504, the coupling coupler 506, the trailer hitch 512, the trailer 502, the trailer drawbar 510, and the coupling connector 508.

[0043] Bird's-eye views 400 can be rotated in 10-degree increments to produce, for example, multiple intermediate angle images between 0 and 180 degrees. The input bird's-eye views 400 can include images containing varying pendant angles 414. The input bird's-eye views 400 can include pendant angles 414 equal to 0, 90, and 180 degrees, which are designated as cardinal pendant angles according to the cardinal compass directions (e.g., north, south, east, and west). The techniques described herein can work with any number of bird's-eye views 400; however, three bird's-eye views 400 in each of the cardinal angles are optimal. The bird's-eye views 400 can be rotated either clockwise or counterclockwise, depending on which of the cardinal angle images is closest to the angle of the desired intermediate pendant angles 414.

[0044] In some examples, the input data might consist of only one or two images captured at random pendant angles between 0 and 180 degrees. Techniques for generating training datasets described herein can work with fewer than three Bird's-Eye 400 images and with three Bird's-Eye 400 images captured at angles other than the cardinal angles; however, three Bird's-Eye 400 images at each of the cardinal angles are optimal.

[0045] Fig. Figure 6 is a representation of a synthetic image 600. The synthetic image 600 is formed by cropping sections of the rotated image 500, which include the vehicle 110, the bumper 504, and the trailer hitch 512, based on determining a mask based on the bird's-eye view 400. The location of the mask can be determined based on data regarding the location and size of the vehicle 110, the trailer hitch 412, and the bumper 404, which are determined based on image data available at the time of production. Since the intrinsic and extrinsic camera parameters do not change, the mask location is the same for subsequently acquired images. The mask can be used to crop sections of the rotated image 500, which include the vehicle 110, the bumper 504, and the trailer hitch 512, from the rotated image 500.The mask can be rotated around the location of the coupling ball 508 to place the cropped section of the rotated image 500 back into its original positions, which are similar to its positions in the bird's-eye view 400, leaving an empty section 616. The cropped section can then be inserted into the synthetic image at the positions of the vehicle 110, the bumper 404, and the trailer coupling 412 in the bird's-eye view 400 to form a synthetic image 600 that includes the vehicle 110, the bumper 604, and the trailer coupling 612, leaving the trailer 602, the trailer drawbar 610, and the coupling connector 608 in their rotated positions.

[0046] The empty sections 616 of the synthetic image 600 can then be filled with road textures from the bird's-eye view 400 using suitable image processing techniques to form a synthetic image 600 that includes a trailer 602 at a new trailer angle 614 relative to the vehicle 110. The road textures can be obtained, for example, from the bird's-eye view 400. Determining training dataset images in this way allows the generation of large numbers of training images with precisely known ground truth data (e.g., the trailer angle 614) based on the input rotation angle applied to a small number (1-3) of input fisheye images 200.This technique for generating training dataset images improves training dataset generation by reducing the number of images that need to be processed to determine ground truth and eliminating or reducing the need for photorealistically rendered images, both of which reduce the amount of computational resources required to generate a training dataset. Generating training dataset images in this way also reduces the need to use generative adversarial neural networks or unsupervised multi-path learning to make rendered images more realistic for training, thus reducing the computational resources required to train a machine learning model.

[0047] Fig. Figure 7 is a representation of a data set generation system 700. The data set generation system is a software program that can be executed on a server computer 120. The data set generation system receives a fisheye image 702 at a fisheye-to-rectilinear transformation 704, which transforms a fisheye image 702 into a rectilinear image 300, as described above with respect to Fig. 2 described. The fisheye-to-straight transformation 704 outputs a straight-line image 300 to a straight-line-to-bird's-eye view transformation 706, which transforms the straight-line image 300 into a bird's-eye view image 400, while correcting the location and scale as described above with respect to Fig. 3 described.

[0048] The straight-line-to-bird's-eye view transformation 706 outputs a bird's-eye view 400 to an angular transformation 708. The angular transformation 708 receives a bird's-eye view 400 at a first trailer angle 410 and rotates the received bird's-eye view 400 to form a rotated image 500 at a second trailer angle 614. The angular transformation 708 then crops and blends the rotated image 500 to form a synthetic image 600. The synthetic image 600 uses elements from the rotated image 500 to create an image that appears as if it were a real image captured at the second trailer angle 614.

[0049] The data set generation system 700 is programmed to receive a set of one to three real-world images captured at one or more cardinal pendant angles, for example, zero degrees, 90 degrees, and 180 degrees. The data set generation system 700 is programmed to generate a series of synthetic images 600 from the input images, incorporating pendant angles 614 from zero to 180 degrees in selected increments, for example, 10 degrees. The data set generation system 700 selects the input image that most closely approximates a selected pendant angle 614 and uses this input image to generate the selected pendant angle 614. The angle transformation 708 generates the series of synthetic images 600 at the selected intermediate pendant angles 614 and outputs them to the training data set 710.The training dataset 710 includes the synthetic images 600 and ground truth data regarding the trailer angles 614, which are included in the synthetic images 600.

[0050] Fig. Figure 8 is a representation of a training system 800 for machine learning models. The training system 800 for machine learning models is a software program that can be executed on the server computer 120 to train a machine learning model 804. The machine learning model 804 can be, for example, a convolutional neural network. A convolutional neural network can contain multiple convolutional layers, followed by several fully connected layers. The convolutional neural network receives an input image 802 from the training data set 710 and outputs a prediction 806 regarding the pendant angle 614 contained in the input image 802.

[0051] The machine learning model 804 can be trained by receiving an input image 802, which generates a prediction 806 regarding the trailer angle 614 contained in the input image 802. The prediction 806 of the trailer angle 614 can be compared to a ground-truth trailer angle contained in the training dataset 710 to determine a loss function. A loss function indicates how closely the prediction 806 of the trailer angle 614 compares to or matches the ground-truth trailer angle. The machine learning model training system 800 can repeat the process hundreds or thousands of times for each image, while the loss function is backpropagated through the layers of the machine learning model 804 to determine the weights that program the layers of the machine learning model 804. The process can be repeated until the loss function converges to a minimum value.The weights that yield the minimum value of the loss function can be stored as the weights included in a trained machine learning model 804. The training process for a single image 802 can be repeated multiple times for the images 802 included in the training dataset 710.

[0052] Fig. Figure 9 is a flowchart of a process 900 for operating a vehicle 110 based on a trained machine learning model 804. The process 900 can be implemented as hardware and software running on a server computer 120 to train the machine learning model 804, which is then transferred to a computing device 115 included in a vehicle 110 to operate the vehicle 110. The process 900 includes several blocks that can be executed in the illustrated sequence. Alternatively or additionally, the process 900 could include fewer blocks and could involve the blocks being executed in a different order.

[0053] In block 902, an initial software program running on server computer 120 generates a training dataset 710 based on a limited number of images captured at cardinal pendant angle positions, as described above in relation to Fig. 2-7 described.

[0054] In block 904, a second software program running on server computer 120 uses the training dataset 710 to train a machine learning model 804, as described above in relation to Fig. 8 described.

[0055] In block 906, the trained machine learning model 804 can be transferred from the server computer 120 to a computing device 115, which is located in a vehicle 110. The computing device 115 can acquire data from sensors located in the vehicle 110, including a video camera. The trained machine learning model 804 can receive images from the video camera and determine a prediction 806 regarding a trailer angle 614, which is included in the acquired image. The computing device 115 can determine a vehicle trajectory which, when operated by the computing device 115, can cause the vehicle 110 to position the coupling ball 608 below the coupling coupler 606 to allow the trailer 602 to be coupled to the vehicle 110.The vehicle trajectory can be determined by assuming a "bicycle" model for the vehicle 110, which can model the front steering wheels as the first independent wheel and the rear drive wheels as the second independent wheel. The computing device 115 can determine the steering angle of the front wheel while power is applied to the rear wheel to move the clutch ball 608 to position it below the clutch coupler 606. This technique can be modified as needed for front-wheel drive and all-wheel drive vehicles. The computing device 115 can operate the vehicle by determining commands to be transmitted to the controllers 112, 113, and 114 to control vehicle components, causing the vehicle 110 to operate along the determined vehicle trajectory. Following block 906, process 900 ends.

[0056] Every action taken by a vehicle or its user should comply with all rules and regulations specific to the vehicle's location and operation (e.g., federal, state, county, city, etc.). Furthermore, all operations disclosed herein are for illustrative purposes only. Certain operations may be modified or omitted depending on the context, situation, and applicable rules and regulations. Moreover, regardless of the operations or regulations, users should exercise good judgment and common sense when operating the vehicle. That is to say, all operations, whether standard or "extended," should be followed when appropriate and when complying with the rules and regulations specific to the vehicle's location and operation.

[0057] Computing devices, such as those described herein, generally each contain instructions that can be executed by one or more computing devices, such as those identified above, to perform blocks or steps of processes described above. For example, process blocks described above may be embodied as computer-executable instructions.

[0058] Computer-executable instructions can be compiled or interpreted by computer programs created using a variety of programming languages ​​and technologies, including, but not limited to, Java™, C, C++, Python, Julia, SCALA, Visual Basic, JavaScript, Perl, HTML, etc., either alone or in combination. Generally, a processor (e.g., a microprocessor) receives instructions, for example, from memory, a computer-readable medium, etc., and executes these instructions, thereby performing one or more processes that include one or more of the processes described herein. Such instructions and other data can be stored in files and transferred using a variety of computer-readable media. A file in a computing device is generally a collection of data stored on a computer-readable medium, such as a data storage medium, random-access memory, etc.

[0059] A computer-readable medium (also called a processor-readable medium) is any non-transient (i.e., physical) medium used to provide data (i.e., instructions) that can be read by a computer (i.e., by a computer's processor). Such a medium can take many forms, including but not limited to non-volatile and volatile media. Instructions can be transmitted through one or more transmission media, including optical fibers, wires, and wireless communication, as well as internal components that comprise a system bus connected to a computer's processor. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH EEPROM, any other memory chip, memory cartridge, or any other medium from which a computer can read.

[0060] All terms used in the patent claims shall have their general and ordinary meaning as understood by a person skilled in the art, unless expressly stated otherwise herein. In particular, the use of singular articles such as "a", "an", "the", "the", etc., shall be interpreted as referring to one or more of the listed elements, unless a patent claim expressly specifies otherwise.

[0061] The term “exemplary” is used herein in the sense that it means an example, i.e., a candidate for an “exemplary device” should simply be read as referring to an example of a device.

[0062] The adverb "approximately", which modifies a value or result, means that a form, structure, measurement, value, determination, calculation, etc., may deviate from a precisely described geometry, distance, measurement, value, determination, calculation, etc., due to deficiencies in materials, processing, manufacturing, sensor measurements, calculations, processing time, communication time, etc.

[0063] In the drawings, the same reference numerals denote the same elements. With regard to the media, processes, systems, procedures, etc., described herein, it is understood that although the steps or blocks of such processes, etc., have been described as being carried out according to a certain sequence, such processes could be implemented such that the described steps are carried out in a different order than that described herein. Furthermore, it is understood that certain steps could be carried out simultaneously, other steps could be added, or certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided to illustrate certain embodiments and should in no way be interpreted as limiting the claimed invention.According to the present invention, a system is provided comprising: a computer including a processor and memory, wherein the memory contains instructions that can be executed by the processor to: transform fisheye images containing objects at a first, second, and third angle into rectilinear images with a first image transformation; transform the rectilinear images into bird's-eye views with a second image transformation; transform the bird's-eye views into multiple images containing objects at multiple angles between the first, second, and third angles in order to generate a training dataset containing ground truth regarding the objects at multiple angles with a third image transformation; and train a machine learning model with the training dataset.

[0064] According to one embodiment, the first image transformation is based on intrinsic fisheye camera parameters, which include fisheye distortion parameters.

[0065] According to one embodiment, the second image transformation is based on intrinsic camera parameters that include focal length in x and y, optical center in x and y, magnification, optical center in x and y, and offset.

[0066] According to one embodiment, the second image transformation is based on extrinsic camera parameters, which include a camera pose with six degrees of freedom.

[0067] According to one embodiment, the second image transformation includes an affine transformation that places a coupling ball at a predetermined location in the images.

[0068] According to one embodiment, the first angle is 0 degrees, the second angle is 90 degrees, and the third angle is 180 degrees.

[0069] According to one embodiment, the third image transformation is based on generating intermediate angle images in 10-degree increments between 0 degrees and 180 degrees.

[0070] According to one embodiment, the machine learning model is a convolutional neural network. According to another embodiment, the objects include a pendant.

[0071] According to one embodiment, the first, second and third angles are based on an angle of a trailer drawbar in relation to a location of a coupling ball.

[0072] According to one embodiment, the machine learning model is trained to determine the location and angle of the trailer drawbar in relation to the location of the coupling ball.

[0073] According to one embodiment, the trained machine learning model is included in a second computer for a vehicle, wherein the second computer is programmed to operate the vehicle by determining a vehicle trajectory based on predictions output from the trained machine learning model.

[0074] According to one embodiment, the second computer is programmed to operate the vehicle on the vehicle trajectory by issuing commands to control vehicle components.

[0075] According to the present invention, a method comprises: transforming fisheye images containing objects at a first, second, and third angle into rectilinear images using a first image transformation; transforming the rectilinear images into bird's-eye views using a second image transformation; transforming the bird's-eye views into multiple images containing objects at multiple angles between the first, second, and third angles to generate a training dataset containing ground truth regarding the objects at multiple angles using a third image transformation; and training a machine learning model using the training dataset. According to one embodiment, the first image transformation is based on intrinsic fisheye camera parameters, which include fisheye distortion parameters.

[0076] According to one embodiment, the second image transformation is based on intrinsic camera parameters that include focal length in x and y, optical center in x and y, magnification, optical center in x and y, and offset.

[0077] According to one embodiment, the second image transformation is based on extrinsic camera parameters, which include a camera pose with six degrees of freedom.

[0078] According to one embodiment, the second image transformation includes an affine transformation that places a coupling ball at a predetermined location in the images.

[0079] According to one embodiment, the first angle is 0 degrees, the second angle is 90 degrees, and the third angle is 180 degrees.

[0080] According to one embodiment, the third image transformation is based on generating intermediate angle images in 10-degree increments between 0 degrees and 180 degrees. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] “A Geometric Approach to Obtain a Bird’s Eye View from an Image”, described by Ammar Abbas and Andrew Zisserman. This article is available at https: / / arxiv.org / abs / 1905.02231

[0038] as of the filing date of this application.

Claims

[1] Procedure, encompassing: Transforming fisheye images containing objects in a first, second, and third angle into rectilinear images using a first image transformation; Transforming the rectilinear images into bird's-eye views with a second image transformation; Transforming the bird's-eye views into multiple images containing objects at several angles between the first, second, and third angles to generate a training dataset that includes ground truth regarding the objects at multiple angles, with a third image transformation; and Training a machine learning model with the training dataset. [2] Method according to claim 1, wherein the first image transformation is based on intrinsic fisheye camera parameters that include fisheye distortion parameters. [3] Method according to claim 1, wherein the second image transformation is based on intrinsic camera parameters that include focal length in x and y, optical center in x and y, magnification, optical center in x and y and offset. [4] Method according to claim 3, wherein the second image transformation is based on extrinsic camera parameters that include a camera pose with six degrees of freedom. [5] Method according to claim 4, wherein the second image transformation includes an affine transformation that places a coupling ball at a predetermined location in the images. [6] Method according to claim 1, wherein the first angle is 0 degrees, the second angle is 90 degrees and the third angle is 180 degrees. [7] Method according to claim 1, wherein the third image transformation is based on generating intermediate angle images in steps of 10 degrees between 0 degrees and 180 degrees. [8] Method according to claim 1, wherein the machine learning model is a neural convolutional network. [9] Method according to claim 1, wherein the objects include a pendant. [10] Method according to claim 1, wherein the first, second and third angles are based on an angle of a trailer drawbar in relation to a location of a coupling ball. [11] Method according to claim 10, wherein the machine learning model is trained to determine a location and an angle of the trailer drawbar in relation to the location of the coupling ball. [12] Method according to claim 1, wherein the trained machine learning model is included in a second computer for a vehicle, the second computer being programmed to operate the vehicle by determining a vehicle trajectory based on predictions output from the trained machine learning model. [13] Method according to claim 12, wherein the second computer is programmed to operate the vehicle on the vehicle trajectory by commands that controllers operate vehicle components. [14] Method according to claim 8, wherein the neural convolutional network comprises multiple convolutional layers and multiple fully connected layers. [15] Method comprising a computer programmed to perform the methods according to any one of claims 1-14.

Images