Object pose generation via trained networks

Abstract

The present disclosure provides object pose generation via a trained network. A computer includes a processor and a memory, and the memory stores instructions executable by the processor to extract features of an object visible in a scene captured by a camera at a first perspective. The instructions may also transform the scene from the first perspective to a second perspective based on parameters generated in response to: (a) reconstructing the scene from the first perspective based on the scene from the second perspective, and (b) adjusting a parameter to minimize a difference between the extracted feature and a representation of the extracted feature reconstructed based on the second perspective. The instructions may also be to determine a depth of the extracted feature relative to the camera based on the parameter, and to determine a pose of the object based on the determined depth.

Description

Technical Field

[0001] The present disclosure relates to techniques for controlling or operating a vehicle via a trained network. Background Art

[0002] Modern vehicles typically include a variety of sensors. Some sensors detect static or moving objects outside the vehicle, such as other vehicles, lane markings of the road, traffic lights and / or signs, animals, natural objects, etc. Types of vehicle sensors may include radar sensors, ultrasonic sensors, scanning laser rangefinders, sensors for satellite positioning systems (e.g., GPS), and light detection and ranging (lidar) devices. Summary of the invention

[0003] The present disclosure describes techniques that can be provided for controlling or operating a machine, including techniques for calculating the pose of a static or moving object visible to a camera sensor (e.g., a camera for capturing a still or video scene) using a representation of a scene obtained via an output signal from a camera sensor. In an example, a scene can be captured by a camera sensor (such as a camera sensor mounted on a vehicle) from a first perspective. Features (such as static or moving objects) present in the captured scene can be detected, enhanced, and transformed from a first perspective to a second perspective, such as a top view perspective, a bird's eye view perspective, etc. In response to the transformation to the second perspective, a computer can execute instructions to determine spatial relationships among the features detected in the captured scene, such as angular orientation and depth relative to the camera sensor. Based on the spatial relationships among the detected features of the captured scene, the computer can execute instructions to: determine the depth of a static or moving object in the scene relative to the camera sensor; assign semantic tags to objects in the scene; and determine the pose of the static or moving object in the scene. The computer can additionally execute instructions to plan the movement of a vehicle, such as a camera sensor, for example. Furthermore, the computer may execute instructions to implement autonomous driving applications, which may include autonomously actuating steering components, actuating vehicle propulsion components, etc. based on the posture of static or moving objects in the scene.

[0004] In an example, rather than a computer (e.g., a computer on a vehicle) performing geometry-based calculations to transform a scene from a first (camera) perspective to a second (top view) perspective that utilizes, among other things, properties inherent to the camera sensor (e.g., the focal length of the camera sensor) and scene geometry (e.g., the vertical vanishing point of the scene), the computer may utilize parameters developed by an auxiliary network that trains the computer to perform the transformation. In this context, "transformation" means a translation or representation of points of a captured scene from a first perspective to a second perspective or from a second perspective to a first perspective, i.e., a transformation comprises a translation from a first coordinate system to a second coordinate system. In an example, a translation representing pixels of a scene captured from a first (camera) perspective to a second (top view) perspective may utilize a rotation matrix that computes the cosine of an angle relative to the horizon and represents the pixels in terms of linear distances from a reference point of the second (top view) perspective. In another example, for a vehicle computer implementing a neural network (e.g., a multi-layer perceptron), rather than the vehicle computer expending processing resources to develop weights at hidden layers of the neural network to represent pixels of a captured scene from a first (camera perspective) to a second (top view) perspective, the computer can utilize parameters derived from weights developed within an auxiliary network. Thus, in either case (i.e., the computer performs a geometry-based transformation or the computer implements a neural network), the parameters developed or derived from the auxiliary network can be used to train the computer to transform a scene captured at a first (camera) perspective to a second (top view) perspective. As the term is used herein, an "auxiliary network" means a network that operates external to a vehicle network that is connected or coupled to the vehicle network during a training, manufacturing, or calibration process.

[0005] As described herein, a transformation of a scene captured from a first perspective to a second perspective may be performed using one or more parameters stored in a memory accessed by, for example, a vehicle computer. In an example, one or more parameters may be generated or derived from a multilayer perceptron that operates to train or instruct the vehicle computer so that the transformation of the scene may be performed with improved accuracy and reduced computational resources. In this context, a "parameter" is a value used by the vehicle computer to control or affect the operation of the vehicle. A "parameter" herein means one or more numerical values ​​and may include a scalar, such as a floating point value with a magnitude between -1 and +1, between -100 and +100, etc., or may include a vector, such as a floating point value with a magnitude and a direction. Thus, for example, the programming of a vehicle computer (e.g., computer 104) may utilize parameters to transform a scene captured from a first perspective (e.g., a camera perspective) to a second (e.g., a top view) perspective.

[0006] In this context, a multilayer perceptron means a fully connected feedforward artificial neural network having at least three layers (an input layer, an output layer, and at least one hidden layer). In an example, an auxiliary network (e.g., 120) may include a multilayer perceptron. In an example, the input layer of the multilayer perceptron operates to receive a query, and the output layer operates to provide a decision or prediction in response to the received query. The hidden layers of the multilayer perceptron operate as a computing resource that is capable of approximating many continuous functions. In this context, a "continuous function" means a five-dimensional vector-valued function (e.g., F(x, y, z, θ, Φ)) expressing a three-dimensional position (e.g., x-coordinate, y-coordinate, and z-coordinate) and a two-dimensional viewing direction (e.g., pitch axis angle (θ) and yaw axis (Φ) angle), and the volume density of each point in the scene is rendered by the multilayer perceptron. In an example, the continuous function may be a novel posture (e.g., x-coordinate, y-coordinate, and z-coordinate, pitch axis angle (θ) and yaw axis (Φ) angle) relative to a camera sensor (e.g., camera sensor 108A).

[0007] In an example, a multilayer perceptron of an auxiliary network may generate parameters that may be used by a vehicle computer by reconstructing a scene from a first perspective based on a scene transformed to a second perspective. The multilayer perceptron may enforce consistency between scenes captured from a first perspective and reconstructed from a second perspective at a feature level, an image level, and a task output level utilizing a feature map of the multilayer perceptron. In this context, "enforcing consistency" means adjusting or modifying one or more weights of a hidden layer of the multilayer perceptron such that loss of feature level, image level, and / or feature layer may be reduced or minimized. The results of the operation for enforcing consistency at the feature level, image level, and task output level of the multilayer perceptron may include adjusting weights applied to a pixel map of a scene captured by a camera, coefficients of a polynomial utilized by an image processing programming of a vehicle computer, or entries in a lookup table utilized by a vehicle computer during the process of transforming a scene captured from a first perspective to a second perspective. In this context, the term "feature loss" means the difference between the position or geometry of a feature reconstructed from a second perspective view and the position or geometry of a feature extracted from a scene from a first (e.g., camera) perspective. Also in this context, the term "image loss" means the difference between the dimensions of the image of the scene reconstructed from the second perspective view and the dimensions of the image of the scene captured from the first (e.g., camera) perspective. Furthermore, in this context, the term "feature map loss" means the difference between the feature maps (e.g., corners, edges, center points) generated by the output layer of the multilayer perceptron based on the scene reconstructed from the second perspective view and the scene captured from the first (e.g., camera) perspective.

[0008] In an example, a system may include a computer having a processor and a memory, wherein the memory includes instructions executable by the processor to: extract features of an object visible in a scene captured by a camera at a first perspective; and transform the scene from the first perspective to a second perspective based on parameters generated in response to: (a) reconstructing the scene from the first perspective based on the scene from the second perspective, and (b) adjusting the parameters to minimize a difference between the extracted features and a representation of the extracted features reconstructed based on the second perspective. The instructions may additionally determine a depth of the extracted features relative to the camera based on the parameters, and determine a pose of the object based on the determined depth.

[0009] In an example, the instructions may additionally be to actuate a vehicle component based on the determined posture of the feature.

[0010] In an example, the vehicle component may be a steering component or a propulsion component.

[0011] In an example, the second viewing angle may be a top view.

[0012] In an example, the parameters may be derived from weights of a multi-layer perceptron learned in response to reconstructing the scene from the first perspective based on the scene from the second perspective.

[0013] In an example, the second perspective may be a bird's-eye view.In an example, the reconstruction of the scene may include instructions for converting pixels of the scene from the second perspective to a polar coordinate system.

[0014] The reconstruction of the scene may include wherein the reconstruction of the scene includes instructions for transmitting a query and a position key in the polar coordinate system to a multilayer perceptron.

[0015] The reconstruction of the scene may include instructions for transmitting a query and a position key in the polar coordinate system to a multilayer perceptron, and wherein the instructions further include instructions for calculating attention weights.

[0016] In an example, the adjusting of the parameters includes instructions for performing bilinear interpolation between the dimensions of the extracted features and the dimensions of the representation of the features extracted by the multilayer perceptron.

[0017] In an example, the adjustment of the parameters includes instructions for minimizing a difference between a scene captured at a first perspective and a reconstruction of the captured scene from a second perspective.

[0018] In an example, the adjusting of the parameters comprises instructions for minimizing a difference between a first task output feature map obtained for the scene captured at the first viewing angle and a second task output feature map obtained for the extracted features reconstructed based on the second viewing angle.

[0019] In an example, a method may include extracting features of an object visible in a scene captured by a camera at a first perspective. The method may additionally include transforming the scene from the first perspective to a second perspective based on parameters generated in response to: (a) reconstructing the scene from the first perspective based on the scene from the second perspective, and (b) adjusting the parameters to minimize a difference between the extracted features and a representation of the extracted features reconstructed from the second perspective. The method may additionally include determining a depth of the extracted features relative to the camera based on the parameters, and determining a pose of the object based on the determined depth.

[0020] In an example, the method may additionally include actuating a vehicle component based on the determined gesture.

[0021] In an example, the vehicle component may be a steering component or a steering component or a propulsion component.

[0022] In an example, the second viewing angle may be a top view.

[0023] In an example, the second perspective may be a bird's eye view.

[0024] In an example, the parameters may be derived from weights of a multi-layer perceptron learned in response to reconstructing the scene from the first perspective based on the scene from the second perspective.

[0025] In an example, adjusting the parameters comprises performing bilinear interpolation between dimensions of the extracted features from the captured scene and dimensions of the representation of the extracted features reconstructed by a multi-layer perceptron.

[0026] In an example, adjusting the parameters includes minimizing a difference between a scene captured at a first perspective and a reconstruction of the captured scene from a second perspective.

[0027] In an example, adjusting the parameter comprises minimizing a difference between a first task output feature map obtained for the scene captured at the first viewing angle and a second task output feature map obtained for the scene reconstructed based on the second viewing angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a block diagram of an example vehicle.

[0029] Figure 2 are example views of the scene from a first perspective and from a second perspective.

[0030] Figure 3 is a schematic diagram of a system for object pose generation via a trained network.

[0031] Figure 4 is a flow chart of a method for sensor object generation via a trained network. DETAILED DESCRIPTION

[0032] refer to Figure 1 , the vehicle 100 may be any passenger or commercial vehicle, such as a car, truck, recreational vehicle, sport utility vehicle, crossover, van, minivan, taxi, bus, etc. The vehicle 100 may include a computer 104, a communication network 106, a sensor group 108, vehicle components 110, a human-machine interface (HMI) 112, a communication interface 114 (e.g., to provide Wi-Fi communication, communication with a satellite or terrestrial network, communication with other vehicles, etc.), and an actuator 116. The sensor group 108 may include a lidar sensor, a long-range radar sensor, an ultrasonic sensor, and a camera sensor 108A. The sensor group 108 may additionally include navigation sensors, such as sensors of a satellite positioning system (e.g., GPS), sensors of an inertial measurement unit, etc.

[0033] In an example, the camera sensor 108A may be mounted on the body 102 of the vehicle 100. The camera sensor 108A may include a camera for capturing a still or video scene including objects located outside the vehicle 100, such as stationary or moving vehicles, animals, natural objects, lane markings, traffic signs, etc. In an example, the camera sensor 108A may be operable to detect electromagnetic radiation within a certain wavelength range. For example, the camera sensor 108A may detect visible light, infrared radiation, ultraviolet light, or a certain range of wavelengths including visible light, infrared light, and / or ultraviolet light. For example, the camera sensor 108A may include an image sensor such as a charge coupled device (CCD), an active pixel sensor such as a complementary metal oxide semiconductor (CMOS) sensor, etc.

[0034] The vehicle component 110 includes a propulsion system to convert stored energy (e.g., gasoline, diesel fuel, electric charge) into motion to propel the vehicle 100. The vehicle component 110 may include a conventional vehicle propulsion subsystem, such as a conventional powertrain system, which includes an internal combustion engine coupled to a transmission that transmits torque generated by the engine to the wheels of the vehicle 100. The vehicle component 110 may also include a hybrid powertrain system that utilizes elements of a conventional powertrain system and an electric powertrain system; or may include another type of powertrain system. The vehicle component 110 may include an electronic control unit (ECU) or the like that communicates with and receives input from the computer 104 and / or a human operator. The human operator may control the propulsion system and / or a gear shift lever.

[0035] The vehicle component 110 may include a conventional vehicle steering subsystem to control the rotation of the wheels of the vehicle 100. The steering subsystem may include a rack-and-pinion steering member with electric power steering, a steer-by-wire system, or another suitable system. The steering subsystem may include an electronic control unit (ECU) or the like that communicates with and receives input from the computer 104 and / or a human operator. The human operator may control the steering subsystem via, for example, a steering wheel.

[0036] The HMI 112 presents information to and receives information from an operator of the vehicle 100. The HMI 112 may include controls and displays located, for example, on an instrument panel in a passenger compartment of the vehicle 100, or may be located at another location accessible to an operator of the vehicle 100. The HMI 112 may include dials, digital readouts, screens, speakers, etc., for providing information to an operator of the vehicle 100. The HMI 112 may include buttons, knobs, keypads, microphones, etc., for receiving information from an operator.

[0037] The vehicle 100 may additionally include an actuator 116 that operates to apply a mechanical force or electromotive force to control an aspect of the vehicle 100. For example, the actuator 116 may include a steering actuator that operates to modify the orientation of the front wheels of the vehicle 100, such as during autonomous or semi-autonomous vehicle operation. In another example, the actuator 116 may include a propulsion actuator that operates to reposition a throttle control of the vehicle 100 to increase or decrease the speed of the vehicle 100.

[0038] The computer 104 of the vehicle 100 may include a microprocessor-based computing device, such as a general-purpose computing device (including a processor and memory, an electronic controller, etc.), a field programmable gate array (FPGA), a system on a chip, an application-specific integrated circuit (ASIC), a combination of the foregoing, and the like. In an example, a hardware description language such as VHDL (VHSIC (Very High Speed ​​Integrated Circuit) Hardware Description Language) can be used in electronic design automation to describe digital and mixed signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, and the logic components inside the FPGA can be configured based on, for example, VHDL programming stored in a memory coupled to the FPGA circuit. Therefore, the computer 104 may include a processor, a memory, and the like. The memory of the computer 104 may include a tangible medium for storing instructions executable by the processor and for electronically storing data and / or a database. Alternatively or in addition, the computer 104 may include a structure such as the aforementioned structure that provides programming. In an example, the computer 104 may be a plurality of computers connected together to operate as a single computing resource.

[0039] The computer 104 can transmit and receive data via the communication network 106. The communication network 106 can include, for example, a controller area network (CAN) bus, Ethernet, WiFi, a local interconnect network (LIN), an on-board diagnostic connector (OBD-II), and / or another wired or wireless communication network. The computer 104 can be communicatively coupled to the camera sensor 108A, the vehicle component 110, the HMI 112, the communication interface 114, and other vehicle systems and / or subsystems via the communication network 106.

[0040] The computer 104 may execute instructions to perform image processing of the scene captured via the camera sensor 108A. Figure 2 and Figure 3Described in more detail, the computer 104 may be operable to extract features from an image of a scene and identify and / or classify such features, such as features representing static or moving objects outside the vehicle 100. In this context, a "feature" means one or more data describing the content of an image of a scene or identifying an attribute of the content of an image of a scene, typically data about physical properties or characteristics included (or not included) in a specified area of ​​the image. In an example, a feature may include information about each facet (such as a point, edge, or object) of an image of a scene. A first feature may also be determined in response to a point, edge, or object that detects a second feature that may be adjacent to or near the first feature in the image. Thus, for example, the computer 104 may execute instructions to identify a moving object as another vehicle traveling along the path 50 in the direction ahead of the vehicle 100. The computer 104 may additionally execute instructions to enhance the detected object, such as by rendering the detected object inside a bounding box, highlighting various aspects of the detected object with a descriptor (e.g., arrow, line, distance indicator, etc.). Furthermore, the computer 104 may additionally execute instructions to transform an image of a scene captured from a first perspective (e.g., the perspective of the camera sensor 108A) to a second perspective, such as a perspective overhead of the vehicle 100. In an example, the computer 104 may utilize parameters stored in a memory accessible to the computer 104, which may include constants, adjustable weights, coefficients of a polynomial, entries of a lookup table, etc.

[0041] exist Figure 1 In the example of FIG. 1 , the auxiliary network 120 may be coupled to the communication network 106 during the manufacturing or testing phase of the vehicle 100. Figure 2 , Figure 3 and Figure 4As described, the parameters stored in the memory accessible to the computer 104 can be developed and refined by the auxiliary network 120. In the example, the auxiliary network 120 includes the multilayer perceptron described above. The multilayer perceptron of the auxiliary network 120 is operable to receive the output data from the camera sensor 108A and the programming of the computer 104 to obtain a representation of the scene captured via the camera sensor 108A and transformed into a scene from a second perspective. The multilayer perceptron of the auxiliary network 120 is operable to reconstruct the scene from the perspective of the camera sensor 108A using the top view of the transformed scene. The auxiliary network 120 can then calculate the loss and enforce consistency so as to reduce or minimize the difference between the reconstructed scene and the original image captured via the camera sensor 108A. The auxiliary network 120 can then upload the parameters generated or derived in response to enforcing the consistency between the reconstructed scene and the original scene for use by the computer 104 during actual use of the vehicle 100. By uploading parameters generated or derived by the auxiliary network 120, the computer 104 may be trained to reduce the computational resources involved in transforming a scene captured from a first (camera sensor) perspective to a second (top view) perspective.

[0042] Exemplary System Operation

[0043] Figure 2 A first example view and a second example view of an area are shown, the first view being shown by scene 200 from a first perspective, and the second view being shown by view 250 from a second perspective. Scene 200 represents a scene captured via camera sensor 108A. Scene 200 includes an image of building 202 and parking lot features 215 and 220. Scene 200 additionally includes open area feature 210, which may represent a grass-covered field adjacent to road 204. Scene 200 includes many additional features that are not labeled, such as trees, additional parking lot features, features within building 202, etc. View 250 is a top-down (or top-view) view of the area depicted in scene 200.

[0044] As reference Figure 1 As described above, the programming of computer 104 may implement top view (OV) transformation component 104A to translate the image of scene 200 into top view 250. Top view 250 may be used to calculate the depth of objects detected in scene 200. Figure 22, building 202 has been transformed from a first (camera) perspective to a second (top view) perspective shown as building 252. Parking lot features 215 and 220 have been transformed to a top view as represented by parking lot features 265 and 270 (respectively) of top view 250. Road 204 has been transformed to a top view represented by road 254, and open area 210 has been transformed to a top view represented by open area 260 of top view 250. In performing the transformations of building 202, parking lot features 215 and 220, open area features 210, and road 204, top view transformation component 104A may be trained by auxiliary network 120 so that transformations of features present in scene 200 (including depths of features from camera sensor 108A) may be performed using reduced computational resources of computer 104.

[0045] Figure 3 is a schematic diagram of a system 300 for object pose generation via a trained network. Figure 3 As an overview, the computer 104 executes instructions to implement the image feature extractor component 305, the image feature enhancer component 310, and the top view transform component 104A. The top view encoder 320 utilizes the output signal from the top view transform component 104A to perform 3D object detection (via the 3D object detection component 325), 3D map segmentation (via the 3D map segmentation component 330), and 3D road feature detection (via the 3D road feature detection component 335). The post-processor component 350 executes instructions to obtain the output from the 3D object detection component 325, the 3D map segmentation component 330, and the 3D road feature detection component 335 for post-processing, which applies confidence thresholding and non-maximum suppression to remove artifacts that may be generated during the 3D object detection, map segmentation, and 3D road feature detection components 325-335. The output from the post-processor component 350, which may include lateral position (x-coordinate), height (y-coordinate), and depth (z-coordinate), is sent to a pose generator 355, which generates a pose of an object observed in the scene by the camera sensor 108A. In response to the output signal from the pose generator 355, the computer 104 may implement motion planning via a motion planner component 360. The motion planner component 360 includes processing steps that operate as part of an autonomous or semi-autonomous driving application, which may cause the actuator 116 to apply, for example, steering commands, propulsion commands, etc. to the vehicle based on the lateral position, height, and depth of an object external to the vehicle 100.

[0046] continue Figure 3As discussed above, the auxiliary network 120 may implement a multi-layer perceptron that operates to provide training parameters to the top view transformation component 104A, which is operable to transform an image of a scene captured at a first (camera) perspective to a second (top view) perspective. The auxiliary network 120 may include a camera perspective transformation component 370 that may receive and process an input signal representing the output of the top view transformation component 104A. The output from the camera perspective transformation component 370 may be sent to an image feature backbone network 375. The output signal from the image feature backbone network 375 may be sent to a scene reconstructor 380. The auxiliary network 120 may be operable to enforce image-level consistency by minimizing the loss between the image captured by the camera sensor 108A and the image reconstructed via the scene reconstructor 380. The auxiliary network 120 may additionally operate to enforce consistency between the enhanced features from the image feature enhancer component 310 and the features of the image decoded via the image feature backbone network 375. The auxiliary network 120 may also be operable to enforce consistency between or among output feature maps within the multilayer perceptrons of the auxiliary network 120. The auxiliary network 120 may be connected or coupled to the communication network 106 during the manufacturing, training, or calibration phases of the vehicle 100.

[0047] exist Figure 3 In the example of , camera sensor 108A is operable to capture an image in a first (camera) perspective of a scene within the field of view of camera sensor 108A. The digitized image from camera sensor 108A may be sent to image feature extractor component 305. Image feature extractor component 305 is operable to extract features of the image of scene 200. In this context, "extracting features" means applying pixel processing techniques that are operable to locate or identify points on an image that can be registered with similar points on other images. Figure 3 In the example of , image feature extractor component 305 is operable to detect edges, corners, spots, ridges, etc. Image feature extractor component 305 may operate through template matching, Hough transform, and / or any other suitable technique that operates to detect low-level features within an image present in scene 200.

[0048] The image features extracted by the image feature extractor component 305 may be sent to an image feature enhancer component 310 that operates to apply, for example, color space enhancement, kernel filtering, image blending, scaling, enhancing brightness and / or contrast, etc. In an example, the image feature enhancer component 310 may operate as a feature learning component that may be trained to recognize features of images present in the scene 200.

[0049] The top view transformation component 104A is operable to translate an image of the scene 200 captured from a first (camera) perspective into a second perspective, such as in Figure 2 250. In an example, the top view transformation component 104A may represent a bird's eye view or another type of geometric image modification representing an elevated view of the scene 200. As previously described herein, the top view transformation component 104A may utilize programming to perform geometry-based transformation calculations that take into account, among other things, properties inherent to the camera sensor (e.g., the focal length of the camera sensor 108A) and scene geometry (e.g., the vertical vanishing point of the scene 200). Alternatively or in addition, the top view transformation component 104A may utilize parameters developed by the auxiliary network 120 that is operable to train the top view transformation component 104A to represent pixels of a scene captured from a first perspective (camera perspective) to a second (top view) perspective.

[0050] The output from the top view transform component 104A, which represents pixels of the scene 200 rendered in the second (top view) perspective, may be sent to a top view encoder 320. The top view encoder 320 may be operable to detect and / or label features present in the second (top view) perspective. Thus, the output of the top view encoder 320 may be sent to a 3D object detection component 325, which may assign semantic labels to objects detected in the second (top view) perspective. Such semantic labels may include categories such as road markings, street or traffic signs, street lights, crosswalks, static or moving vehicles, etc.

[0051] The output of the top view encoder 320 may additionally include a 3D map segmentation component 330 operable to modify the representation of the image within the second (top view) of the scene 200 by identifying the image of the top view 250 via bounding boxes, lines, curves, etc. The image of the top view 250 may be segmented by thresholding the pixels within the top view 250, pixel clustering, histogram-based methods, etc. In an example, the 3D map segmentation component 330 assigns semantic labels to corresponding pixels in the image such that pixels with the same semantic label share one or more characteristics.

[0052] The output of the top view encoder 320 may additionally include a 3D road feature detection component 335 that operates to identify, for example, points or features on the surface of the road 204 and determine the location and orientation of lanes of the road 204. Alternatively or in addition, the 3D road feature detection component 335 may operate by detecting edges or corners in the road 204 and / or may utilize a pre-existing model of the road surface. When detecting road features such as lane lines, the 3D road feature detection component 335 may operate to project virtual lines between the various lane lines so as to present a comprehensive picture of the road features of the road 204.

[0053] The outputs of the 3D object detection component 325, the 3D map segmentation component 330, and the 3D road feature detection component 335 may be sent to a post-processor component 350, which applies confidence thresholding and non-maximum suppression to remove artifacts caused by the detection and segmentation components 325-335. In an example, the post-processor component 350 may utilize dilation and erosion algorithms, feature smoothing, heuristic filtering, screening and closing, median filtering, clustering, etc.

[0054] The output of the post-processor component 350 may be sent to a pose generator 355, which operates to determine the pose of the object in the top view 250. The pose generator 355 may obtain signals from other sensors of the sensor group 108, such as sensors such as an inertial measurement unit, a satellite positioning system (e.g., GPS), a lidar sensor, a long-range radar sensor, an ultrasonic sensor, etc., to determine a motion vector of the vehicle 100 relative to the object in the global reference frame.

[0055] The output of the pose generator 355 may be sent to a motion planner component 360 that operates to plan the motion of the vehicle 100 based on the poses of the objects in the top view 250. The motion planner component 360 of the vehicle 100 may provide steering commands to modify the motion vector of the vehicle 100, provide propulsion commands, etc. In an example, based on the poses of static objects such as road features (e.g., lane markings, road signs, bridge abutments, etc.), the motion planner component 360 may provide steering commands to adjust the path of the vehicle 100. The output of the motion planner component 360 may be sent to the actuator 116 that operates to modify the direction of travel of the vehicle 100, increase or decrease the speed of the vehicle 100, or actuate another vehicle control involved in moving the vehicle 100.

[0056] exist Figure 3 In the example of FIG. 1 , the auxiliary network 120 includes a camera perspective (CP) transformation component 370. The camera perspective transformation component 370 operates to receive output data from the top view transformation component 104A and perform the inverse operation of the operation performed by the top view transformation component 104A. Figure 3 In the example of FIG. 3 , the camera perspective transformation component 370 executes programming to translate the top view 250 from the second (top view) perspective to the first (camera) perspective.

[0057] In one example, the instructions executed by the camera view transform component 370 may be operable to transform the pixels of the top view 250 from a first (Cartesian) coordinate system to a polar coordinate system. Thus, each pixel of the top view 250 may be represented as (C, r, θ), where θ represents an angular coordinate, r represents a distance from a center point of the top view 250, and C represents a feature dimension at a given pixel of the top view 250. The transformation of Cartesian coordinates (e.g., x, y, z pixel locations) to a polar coordinate system may be performed according to the following expressions (1) and (2):

[0058]

[0059] θ=arctan(y,x) (2)

[0060] After converting the top view 250 to the polar coordinate system, the top view 250 can be divided into a plurality of sectors, wherein each sector measures, for example, a portion of the 360 ​​degree circumference of a circle in the polar coordinate system. Thus, the sectors can measure, for example, 1° radian, 2° radian, 3° radian, or any other suitable subdivision of the polar coordinate representation of the top view 250.

[0061] The pixels of the top view 250 may be arranged to have a shape C xh p The reconstruction of the top view 250 can be obtained by (C,h p ,w p ) is represented by this arrangement. In this arrangement, the polar coordinate subdivision (p s ) may include a shape C xr. Then two matrices may be defined: W Q , which includes a query vector having shape cx c′; and W K , which may include a position key vector having a shape of cx c′. Then, the key matrix for each sector of the polar coordinate representation of each pixel of the top view 250 may be defined according to the following expression (3):

[0062] K(p s )=(p s ) T W k (3)

[0063] Where K(p s ) is the position key vector, and where ((p sp ) T W k ) represents polar coordinate subdivision (p s). Then, the sinusoidal position encoding can be defined as p for each column in the first (camera) view: e Therefore, the query (Q(p e )) can be calculated by the multilayer perceptron of the auxiliary network 120 according to the following expression (4):

[0064] Q(p e )=(p e ) T W Q (4)

[0065] It should be noted that although sinusoidal position encoding is used for each column in the first (camera) perspective, another suitable encoding technique may be applied in expressions (3) and (4). Another suitable encoding technique may be any encoding technique that produces a value between 0 and 1 for each pixel in the column of view 250 of the second (top) perspective.

[0066] After submitting the query to the multilayer perceptron of the auxiliary network 120, each query (Q(p e )) and all key vectors (K(p s )), which produces a vector of scalars. The scalars can then be normalized according to the following expression (5):

[0067]

[0068] Then, a softmax function may be applied to the weights of the multilayer perceptron of the auxiliary network 120, which results in each weight comprising a value between 0 and 1. In the example, applying the softmax results in an attention weight for each pixel in a column of the segmentation of the polar coordinate representation of the second (top view) perspective shown in the view 250. The softmax function may be according to the following expression (6):

[0069]

[0070] Where β[i][j] of expression (6) represents a pixel having dimension ixj (where i represents the pixels in each column and where j represents the number of columns) and having shape h p In this example, β[i][j] may represent the attention weights of the pixels in the polar coordinate representation of the second (top) perspective shown in view 250. In this context, “attention weights” refers to the mapping of a query and a set of key-value pairs to an output, where the query (Q(p e ))、Key(K(p s )) and values ​​(e.g., floating point values ​​between 0 and 1) and the outputs (attention weights β[i][j]) are vectors.

[0071] In the example, each of the sinusoidal position encoding values ​​(p ) of each column of the first camera view may be denoted by an appropriate scalar vector e ) is weighted to obtain a feature vector at a pixel position represented by the polar coordinates of the second (top) perspective shown in view 250. Therefore, the process blocks of expressions (1) to (6) may be repeated until the attention weights of all columns of the polar coordinates of the second (top) perspective shown in view 250 are obtained. For each pixel (i) of column (j), the reconstructed feature vector (F) of each pixel at a distance (r) from the center point of view 250 is calculated according to the following expression (7): reconst feat [i][j]):

[0072]

[0073] Among them, F reconst feat [i][j] denote the reconstructed features in the first (camera) perspective view.

[0074] In an example, learning the relationship between columns of the reconstructed first (camera) perspective view can utilize a feature pyramid network operating on different scale feature maps from the reconstructed first view. In an example, bilinear interpolation is used to transform the dimensions (F reconst img ) and the image dimension (F original img ). In bilinear interpolation, the dimension of the reconstructed image of the first (camera) view (f reconst img ) is modified to approximate the image dimension (F original img ) dimension.

[0075] In the example, the reconstructed feature map and image of the scene 200 captured from the first (camera) perspective can be used to learn the 3D-2D spatial relationship (e.g., angular position and depth) between the reconstructed features and the features of the scene 200 captured by the camera sensor 108A. In the example, the three-dimensional features are learned from the second (top view) perspective, and the auxiliary network 120 and the image feature backbone network 375 can be used to perform two-dimensional object detection and two-dimensional segmentation tasks. In this context, "image feature backbone network" means a feature extractor network that calculates features based on the input image and then upsamples such features through a decoder programming component to generate a segmented mask. In the example, the image feature backbone network 375 includes a feature encoder using a basic residual network (ResNet) block, in which the weights of the hidden layers of the multilayer perceptron learn the residual function of the reference layer input. In this context, a residual network means a type of neural network that uses skip layer connections or shortcuts to exclude hidden layers of the neural network. In the example, the task output feature map for determining the center point of a two-dimensional image in object detection can utilize a convolution-based foreground / background segmentation mask. The output of the image feature backbone network 375 is then sent to a scene reconstructor 380 so that the detected image features can be assembled into a reconstructed or recreated scene representing the first (camera) perspective view of the scene 200 .

[0076] In an example, the auxiliary network 120 may enforce consistency between the two-dimensional features reconstructed from the second (top) perspective shown in the view 250 and the original two-dimensional feature map obtained from the image feature enhancer component 310. The feature map loss function may be calculated according to the following expression (8):

[0077]

[0078] where F reconst feat represents the features reconstructed from the first (camera) view, and where F original feat Represents features transmitted from image feature enhancer component 310 via signal path 395.

[0079] In an example, auxiliary network 120 may enforce consistency between an image reconstructed from a second (top) perspective shown in view 250 and an original image from a first (camera) perspective of scene 200. Enforcement of the consistency loss may be calculated according to the following expression (9):

[0080]

[0081] Where I represents the input image from camera sensor 108A transmitted via signal path 385.

[0082] The auxiliary network 120 may additionally enforce a task consistency loss between the tasks performed by the multilayer perceptrons of the auxiliary network. For example, the task consistency loss may include a loss between a detection task and a segmentation task, as described in the following expression (10):

[0083] L c task =L c det +L c seg (10)

[0084] Where L c det Represents the heat map task (F aux heatmap -F feat heatmap )), the heat map task can render areas of the image in different colors or different tones relative to other areas of the image. c det The total value of may additionally include the regression task of the auxiliary network 120 and the image feature center point network (F aux reg -F feat reg ), as depicted in the following expression (11):

[0085]

[0086] In the example, as indicated in the following expression (12), L c seg Includes the mean squared error loss between the image feature segmentation tasks performed by the auxiliary network 120.

[0087]

[0088] Then, the total loss calculated by the auxiliary network 120 can be expressed as the following expression (13):

[0089] L total =L 3D BEV +L 2D视角 +L c feat +L c img +L c task (13)

[0090] Where L 3D BEV is the 3D top view task loss, and where L 2D视角 is the 2D first (camera) view task loss. L 3D BEV Loss and L 2D视角 The loss is a generic loss function and varies depending on the task being performed.

[0091] Figure 4 is a flow chart of a process 400 for object pose generation via a trained network. Figure 4 In the example process, using Figure 3The image feature extractor component 305 extracts and enhances features of the scene 200 captured from the first (camera) perspective. The extracted image is enhanced, such as by applying color space enhancement, kernel filtering, image blending, scaling, etc. The image may be transformed to the second (top) perspective shown in the view 250. The transformed image may be transmitted to the camera perspective transformation component 370 of the auxiliary network 120, where the transformed image may be converted to polar coordinates. The polar coordinate transformed image may be subdivided into columns, where, for example, each column may represent a 1° sector of the image in the second (top view) perspective. The values ​​of the pixels in each column may be encoded and assigned a key to query the multi-layer perceptron of the auxiliary network 120. Attention weights may be calculated in response to queries submitted to the multi-layer perceptron of the auxiliary network 120, where the attention weights represent pixels transformed from the second (top view) perspective to the first (camera) perspective. The pixels may be sent along the image feature backbone network 375 to perform two-dimensional object detection and / or two-dimensional segmentation tasks. The output of the image feature backbone network 375 may be sent to a scene reconstructor 380 that operates to reconstruct the scene captured from the first (camera) perspective based on the second (top view) perspective.

[0092] The multi-layer perceptron of the auxiliary network 120 can enforce consistency between the reconstructed scene at the image level, the image feature level, and the task level of the task performed by the multi-layer perceptron. The multi-layer perceptron can calculate training parameters to upload to the top view transformation component 104A that operates via the execution of instructions on the computer 104. The uploaded training parameters are operable to allow the top view transformation component 104A to operate with reduced latency when converting a scene image captured from a first (camera) perspective to a second (top view) perspective. Therefore, the computer 104 can, for example, during autonomous or semi-autonomous operation of the vehicle 100, simultaneously operate driver assistance features or advanced driver assistance system (ADAS) features of the vehicle 100 such as lane keeping, automatic steering, cruise control, etc., reducing the processing resources consumed by calculating the pose of objects external to the vehicle 100, planning movements, and actuating vehicle controls.

[0093] Process 400 begins at block 405 , which includes obtaining an image of scene 200 captured from a first perspective from camera sensor 108A.

[0094] Process 400 continues at block 410, which includes extracting image features via image feature extractor component 305. Block 410 may include computer 104 performing image processing to detect edges, corners, spots, ridges, etc. that may be present in the image of scene 200. Block 410 may additionally include executing programming to perform scale-invariant feature transforms, template matching, Hough transforms, and / or any other suitable technique that operates to detect low-level features within the image present in scene 200.

[0095] Process 400 continues at box 415, which includes the computer 104 executing programming to enhance detected features, such as color space enhancement, kernel filtering, image blending, scaling, enhancing brightness and / or contrast, etc., which is operable to emphasize details in the image of the scene 200 captured from the first (camera) perspective.

[0096] Process 400 continues at block 420 , which includes computer 104 executing programming to transform (eg, via top view transformation component 104A) a scene captured at a first (camera) perspective to a scene from a second (top view) perspective, such as depicted in view 250 .

[0097] Process 400 continues at block 425 where computer 104 transmits the transformed scene to auxiliary network 120. Auxiliary network 120 may include a multilayer perceptron or another type of neural network operable to reconstruct scene 200 captured from a first (camera) perspective based on a second (top view) perspective (e.g., view 250).

[0098] Process 400 continues at block 430, where auxiliary network 120 may execute instructions (e.g., via camera view transform component 370). Block 430 may include subdividing the pixels of top view 250 into a polar coordinate system representing sectors (e.g., 1° sectors, 2° sectors, 3° sectors, etc.), and arranging the pixel values ​​of the scene or each pixel into columns. The subdivision of the scene in the top view may be according to expressions (1) and (2).

[0099] Process 400 continues at block 435, where camera view transform component 370 generates queries to the multilayer perceptron of auxiliary network 120. Each query may include an encoded pixel value and a key value, such as a floating point value between 0 and 1. In an example, the auxiliary network may utilize a sinusoidal encoding of each pixel in a column. The encoding of the pixel value may be according to expression (3).

[0100] Process 400 continues at block 440, which includes auxiliary network 120 querying the multilayer perceptron using the query generated at block 435. The query may take a form similar to expression (4). The query (Q(p)) of expression (4) may be queried according to expression (5). e )) is normalized.

[0101] Process 400 continues at block 445, which includes auxiliary network 120 obtaining attention weights from the multilayer perceptron of auxiliary network 120. Computing attention weights may be obtained for all columns of pixels in the polar coordinate representation of the second (top view) perspective shown in view 250 according to expressions (6) and (7).

[0102] Process 400 continues at block 450, which includes enforcing a consistency loss between the weights calculated by the multilayer perceptron of auxiliary network 120 and the feature maps representing the features detected by the image feature enhancer component 310 and the reconstructed features detected by the image feature backbone network 375 (e.g., via expression (8)). Block 450 may additionally include enforcing a consistency loss between the image reconstructed by the scene reconstructor 380 and the scene 200 captured by the camera sensor 108A (e.g., via expression (9)). Block 450 may additionally include enforcing a consistency loss between the output feature maps within the multilayer perceptron of auxiliary network 120 (e.g., via expressions (10), (11), (12)). The total loss calculated by auxiliary network 120 may then be expressed according to expression (13).

[0103] Process 400 continues at box 455, which includes assisting network 120 to transmit training parameters for uploading to computer 104. Thus, top view transformation component 104A can perform a transformation from a first (camera) perspective to a second (top view) perspective while consuming reduced processing resources. Performing such transformations can help generate poses of objects visible to camera sensor 108A mounted on vehicle body 102. Motion planner component 360 can utilize object pose generation to actuate one or more of a propulsion system of vehicle component 110, a steering system of component 110, or HMI 112. For example, computer 104 can actuate one or more of component 110 that performs an advanced driver assistance system (ADAS). ADAS is an electronic technology that assists drivers in achieving driving functions and parking functions. Examples of ADAS include lane departure detection, blind spot detection, adaptive cruise control, and lane keeping assistance. Computer 104 can actuate the system of vehicle 100 according to an algorithm operating without human input to stop the vehicle before reaching a static or moving object in a scene captured by camera sensor 108A. Computer 104 may autonomously operate vehicle 100 , ie, the propulsion system and / or the steering system, based on output signals from motion planner component 360 .

[0104] Generally speaking, the computing systems and / or devices described may employ any of a variety of computer operating systems, including but not limited to the following versions and / or varieties: Ford Application; AppLink / Smart Device Link middleware; Microsoft Operating system; Microsoft Operating system; Unix operating system (for example, distributed by Oracle Corporation of Redwood Shores, California operating systems); AIX UNIX operating system distributed by International Business Machines Corporation of Armonk, New York; Linux operating system; Mac OSX and iOS operating systems distributed by Apple Inc. of Cupertino, California; BlackBerry operating system distributed by BlackBerry Ltd. of Waterloo, Canada; and Android operating system developed by Google Inc. and the Open Handset Alliance; or provided by QNX Software Systems CAR Infotainment Platform. Examples of computing devices include, but are not limited to, an in-vehicle computer, a computer workstation, a server, a desktop, notebook, laptop or handheld computer, or some other computing system and / or device.

[0105] Computing devices typically include computer-executable instructions, wherein the instructions are executable by one or more computing devices such as those listed above. Computer-executable instructions may be compiled or interpreted from a computer program created using a variety of programming languages ​​and / or technologies, including, alone or in combination, but not limited to Java. TM , C, C++, Matlab, Simulink, Stateflow, Visual Basic, Java Script, Python, Perl, HTML, etc. Some of these applications can be compiled and executed on virtual machines such as Java virtual machines, Dalvik virtual machines, etc. Typically, a processor (e.g., a microprocessor) receives instructions from, for example, a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using various computer-readable media. Files in a computing device are typically a collection of data stored on a computer-readable medium (such as a storage medium, a random access memory, etc.).

[0106] Computer-readable media (also referred to as processor-readable media) include any non-transitory (e.g., tangible) media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including optical fiber, wires, wireless communications, including internals that make up a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, flash EEPROM, any other memory chip or cassette tape or any other medium from which a computer can read.

[0107] The database, data repository or other data storage described herein may include various mechanisms for storing, accessing and retrieving various data, including hierarchical databases, file sets in file systems, application databases in special formats, relational database management systems (RDBMS), non-relational databases (NoSQL), graphic databases (GDB), etc. Each such data storage is typically included in a computing device using a computer operating system such as one of the above-mentioned, and can be accessed via a network in any one or more of a variety of ways. The file system can be accessed from the computer operating system and can include files stored in various formats. In addition to the language (such as the above-mentioned PL / SQL language) for creating, storing, editing and executing stored programs, RDBMS also typically uses structured query language (SQL).

[0108] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) stored on computer-readable media associated therewith (e.g., disks, memories, etc.). A computer program product may include such instructions stored on a computer-readable medium for performing the functions described herein.

[0109] In the accompanying drawings, the same reference numerals indicate the same elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring in a certain ordered sequence, such processes may be practiced by performing the steps in an order different from that described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable owner / user manuals and / or safety guidelines.

[0110] The present disclosure has been described in an illustrative manner, and it should be understood that the terms that have been used are intended to be descriptive in nature, rather than restrictive. The adjectives "first" and "second" are used throughout this document as identifiers, and are not intended to indicate importance, order, or quantity. The use of "in response to" and "after determining..." indicates a causal relationship, not just a temporal relationship. In view of the above teachings, many modifications and variations of the present disclosure are possible, and the present disclosure can be practiced in other ways than those specifically described.

[0111] According to the present invention, a system is provided, comprising: a computer having a processor and a memory, the memory comprising instructions executable by the processor to: extract features of an object visible in a scene captured by a camera at a first perspective; transform the scene from the first perspective to a second perspective based on parameters generated in response to the following items: (a) reconstructing the scene from the first perspective based on the scene from the second perspective, and (b) adjusting parameters to minimize the difference between the extracted features and a representation of the extracted features reconstructed based on the second perspective; determining the depth of the extracted features relative to the camera based on the parameters; and determining the pose of the object based on the determined depth.

[0112] According to an embodiment, the instructions comprise instructions for actuating a vehicle component based on the determined pose of the object.

[0113] According to an embodiment, the vehicle component is a steering component or a propulsion component.

[0114] According to an embodiment, the second viewing angle is a top view.

[0115] According to an embodiment, the parameters are derived from weights of a multi-layer perceptron learned in response to reconstructing the scene from the first perspective based on the scene from the second perspective.

[0116] According to an embodiment, the second perspective is a bird's eye view, and wherein the reconstruction of the scene comprises instructions for converting pixels of the scene from the second perspective into a polar coordinate system.

[0117] According to an embodiment, said reconstruction of said scene comprises instructions for transmitting a query and a position key in said polar coordinate system to a multilayer perceptron.

[0118] According to an embodiment, said reconstruction of said scene comprises instructions for transmitting a query and a position key in said polar coordinate system to a multilayer perceptron, and wherein said instructions further comprise instructions for calculating attention weights.

[0119] According to an embodiment, the adjusting of the parameters includes instructions for performing bilinear interpolation between the dimensions of the extracted features and the dimensions of the representation of the features extracted by the multi-layer perceptron.

[0120] According to an embodiment, the adjustment of the parameters comprises instructions for minimizing the difference between the scene captured at the first perspective and the reconstruction of the captured scene from the second perspective.

[0121] According to an embodiment, the adjusting of parameters comprises instructions for minimizing the difference between a first task output feature map obtained for the scene captured at the first viewing angle and a second task output feature map obtained for the extracted features reconstructed based on the second viewing angle.

[0122] According to the present invention, a method includes: extracting features of an object visible in a scene captured by a camera at a first perspective; transforming the scene from the first perspective to a second perspective based on parameters generated in response to the following items: (a) reconstructing the scene from the first perspective based on the scene from the second perspective, and (b) adjusting parameters to minimize the difference between the extracted features and the representation of the extracted features reconstructed based on the second perspective; determining the depth of the extracted features relative to the camera based on the parameters; and determining the pose of the object based on the determined depth.

[0123] In one aspect of the invention, the method further comprises actuating a vehicle component based on the determined posture of the feature.

[0124] In one aspect of the invention, the vehicle component is a steering component or a propulsion component.

[0125] In one aspect of the invention, the second viewing angle is a top view.

[0126] In one aspect of the invention, the second viewing angle is a bird's eye view.

[0127] In one aspect of the invention, the parameters are derived from weights of a multi-layer perceptron learned in response to reconstructing the scene from the first perspective based on the scene from the second perspective.

[0128] In one aspect of the invention, adjusting said parameters comprises performing bilinear interpolation between the dimensions of said extracted features from the captured scene and the dimensions of said representation of said extracted features reconstructed by a multi-layer perceptron.

[0129] In one aspect of the invention, adjusting the parameters includes minimizing a difference between a scene captured at a first viewing angle and a reconstruction of the captured scene from a second viewing angle.

[0130] In one aspect of the invention, adjusting the parameters comprises minimizing a difference between a first task output feature map obtained for the scene captured at the first viewing angle and a second task output feature map obtained for the scene reconstructed based on the second viewing angle.

Claims

1. A method comprising: Extracting features of objects visible in a scene captured by a camera in a first-person perspective; transforming the scene from the first view to a second view based on parameters generated in response to: (a) reconstructing the scene from the first view based on the scene from the second view, and (b) adjusting parameters to minimize differences between extracted features and representations of the extracted features reconstructed from the second view; determining a depth of the extracted feature relative to the camera based on the parameter; and A pose of the object is determined based on the determined depth.

2. The method of claim 1, wherein the method further comprises: A vehicle component is actuated based on the determined pose of the object.

3. The method of claim 2, wherein the vehicle component is a steering component or a propulsion component. The method of claim 1 , wherein the second viewing angle is a top view. The method of claim 1 , wherein the second perspective is a bird's eye view.

6. The method of claim 1, wherein the parameters are derived from weights of a multilayer perceptron learned in response to reconstructing the scene from the first perspective based on the scene from the second perspective.

7. The method of claim 1, wherein adjusting the parameters comprises performing bilinear interpolation between dimensions of the extracted features from the captured scene and dimensions of the representation of the extracted features reconstructed by a multi-layer perceptron.

8. The method of claim 1, wherein adjusting the parameters comprises minimizing a difference between the scene captured at the first perspective and the captured scene reconstructed from the second perspective.

9. The method of claim 1, wherein adjusting the parameters comprises minimizing a difference between a first task output feature map obtained for the scene captured at the first perspective and a second task output feature map obtained for the scene reconstructed based on the second perspective.

10. The method of claim 1, wherein the second perspective is a bird's eye view, and wherein reconstructing the scene further comprises converting pixels of the scene from the second perspective to a polar coordinate system.

11. The method of claim 10, wherein reconstructing the scene further comprises transmitting the query and the position key in the polar coordinate system to a multi-layer perceptron.

12. The method of claim 10, wherein reconstructing the scene further comprises: transmitting the query and the position key in the polar coordinate system to a multilayer perceptron; as well as Calculate attention weights.

13. The method of claim 1, wherein adjusting the parameters further comprises minimizing a difference between the scene captured at the first perspective and the captured scene reconstructed from the second perspective.

14. A computer programmed to perform the method of any one of claims 1-13.

15. A vehicle comprising the computer according to claim 14.

Images