Systems and methods for three-dimensional (3D) object detection

Abstract

Disclosed herein is a system and method for three-dimensional (3D) object detection. Particular embodiments can be configured to: receive image data from at least one camera associated with an autonomous vehicle, the image data representing at least one image frame; use a trained deep learning module to determine pixel coordinates of a two-dimensional (2D) bounding box around an object detected in the image frame; use the trained deep learning module to determine vertices of a three-dimensional (3D) bounding box around the object; use a fitting module to obtain geologic information related to a particular environment associated with the image frame and to obtain camera calibration information associated with the at least one camera; and use the fitting module to determine 3D properties of the object using the 3D bounding box, the geologic information, and the camera calibration information.

Description

[0001] Priority Claims and Related Patent Applications

[0002] This PCT application claims priority to and the benefit of U.S. Patent Application No. 16 / 129,040, filed September 12, 2018. The entire disclosure of the above application is incorporated by reference as part of the disclosure of this patent document.

[0003] Copyright Notice

[0004] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to the disclosure herein and to the drawings that form a part of this document: Copyright © 2017-2018, TuSimple, All Rights Reserved. TECHNICAL FIELD

[0005] This patent document generally relates to tools (systems, apparatuses, methods, computer program products, etc.) for image processing, vehicle control systems, and autonomous driving systems, and more particularly but not exclusively to a system and method for three-dimensional (3D) object detection. BACKGROUND

[0006] Object detection is a fundamental problem for many vision tasks, including object tracking, semantic instance segmentation, and object behavior prediction. Detecting all objects (such as cars, buses, pedestrians, and bicycles) in a traffic environment is crucial for building autonomous driving systems. Failure to detect an object (e.g., a car or a person) can cause malfunction of the motion planning module of an autonomous driving car, leading to catastrophic accidents. As such, object detection for autonomous driving vehicles is an important operational and safety concern.

[0007] Deep learning-based 2D object detection models have been successfully applied to various computer vision tasks, including face detection, instance segmentation, point cloud processing, and autonomous driving. Given an input image, the goal of 2D object detection is to output the class labels and locations (using rectangular bounding boxes) of all objects of interest. However, because all operations are performed on the 2D image plane, conventional models can only obtain relative location information (in pixels), not absolute values (in meters). This behavior produced by traditional 2D models is undesirable for modern autonomous driving systems because losing precise locations (and potentially car sizes) can severely compromise the output quality of the perception module, affecting the execution of subsequent motion planning and control modules and creating potential hazards. SUMMARY

[0008] The technology disclosed in this PCT application can be implemented to provide systems and methods for three-dimensional (3D) object detection.

[0009] In one aspect, the disclosed technology can be implemented to provide a system comprising a data processor and a 3D image processing system. The image processing system is configured to: receive image data from at least one camera associated with an autonomous vehicle, the image data representing at least one image frame; use a trained deep learning module to determine pixel coordinates of a two-dimensional (2D) bounding box around an object detected in the image frame; use the trained deep learning module to determine vertices of a three-dimensional (3D) bounding box around the object; use a fitting module to obtain geologic information related to a particular environment associated with the image frame and to obtain camera calibration information associated with the at least one camera; and use the fitting module to determine 3D properties of the object using the 3D bounding box, the geologic information, and the camera calibration information.

[0010] In another aspect, the disclosed technology can be implemented to provide a method comprising: receiving image data from at least one camera associated with an autonomous vehicle, the image data representing at least one image frame; using a trained deep learning module to determine pixel coordinates of a two-dimensional (2D) bounding box around an object detected in the image frame; using the trained deep learning module to determine vertices of a three-dimensional (3D) bounding box around the object; using a fitting module to obtain geologic information related to a particular environment associated with the image frame and to obtain camera calibration information associated with the at least one camera; and using the fitting module to determine 3D properties of the object using the 3D bounding box, the geologic information, and the camera calibration information.

[0011] In yet another aspect, the disclosed technology can be implemented to provide a non-transitory machine-useable storage medium embodying instructions that, when executed by a machine, cause the machine to: receive image data from at least one camera associated with an autonomous vehicle, the image data representing at least one image frame; use a trained deep learning module to determine pixel coordinates of a two-dimensional (2D) bounding box around an object detected in the image frame; use the trained deep learning module to determine vertices of a three-dimensional (3D) bounding box around the object; use a fitting module to obtain geologic information related to a particular environment associated with the image frame and to obtain camera calibration information associated with the at least one camera; and use the fitting module to determine 3D properties of the object using the 3D bounding box, the geologic information, and the camera calibration information. In some implementations, the non-transitory machine-useable storage medium can be further configured to provide the 3D properties of the object to an autonomous driving perception system. In some implementations, the at least one camera includes a camera lens of a type from the group consisting of: a wide or close-up lens, a mid-range lens, and a long-range lens. In some implementations, the vertices of the three-dimensional (3D) bounding box around the object are determined in pixel coordinates. In some implementations, the geologic information is obtained from a topographic map including global positioning system (GPS) locations with heights of terrain. In some implementations, the 3D properties of the object include an object length, a height, a width, a 3D spatial location, and a direction of travel of the object.

[0012] Further, in some implementations, the disclosed technology can be used to provide an example system and method for 3D object detection to include a 3D image processing system configured to receive image data from at least one camera associated with an autonomous vehicle. Example embodiments can be configured to output a location of a 2D bounding box around a detected object and a location of eight corners that depict the size and direction (direction of travel) of the object. This is an improvement over conventional systems that do not provide real-world 3D information. With geologic information related to a particular environment (e.g., road or terrain information) and a camera calibration matrix, example embodiments can calculate the exact size and location of an object imaged by a camera in 3D coordinates. Example embodiments run in real-time and serve as a key component in an autonomous driving perception system.

[0013] In various embodiments described herein, a 3D image processing module is configured to address the aforementioned issues. As described herein, the 3D image processing module can be used to obtain 3D attributes of objects, including length, height, width, 3D spatial location in camera coordinate space (all in meters), and moving direction (heading) of the objects. In example embodiments, the 3D image processing module includes two sub-modules, namely: 1) a deep learning module that learns pixel coordinates of 2D bounding boxes in image plane and all vertices of 3D bounding boxes; and 2) a fitting module that uses geological information from a topographic map and camera information (including camera calibration matrix with extrinsic and intrinsic matrices of the camera) to solve for 3D attributes. The 3D object detection module can run in real-time across multiple cameras and can significantly contribute to the perception pipeline and improve the robustness and safety level of autonomous driving systems.

[0014] Details of various example embodiments are provided below. BRIEF DESCRIPTION OF DRAWINGS

[0015] In the drawings accompanying hereof, various embodiments are illustrated by way of example and not by way of limitation, in which:

[0016] Figure 1 a block diagram illustrating an example ecosystem of an in-vehicle image processing module that can implement example embodiments is illustrated;

[0017] Figure 2 an example image showing two-dimensional (2D) and three-dimensional (3D) bounding boxes of a vehicle in an image plane is illustrated;

[0018] Figure 3 and Figure 4 a first image sample set including images from a wide-angle camera is illustrated; Figure 3 a wide-angle image of the first image set as processed by a deep learning module of an example embodiment is illustrated; Figure 4 a wide-angle image of the first image set as processed by a fitting module of an example embodiment is illustrated;

[0019] Figure 5 and Figure 6 a first image sample set including images from a mid-range camera is illustrated; Figure 5 a mid-range image of the first image set as processed by a deep learning module of an example embodiment is illustrated; Figure 6 a mid-range image of the first image set as processed by a fitting module of an example embodiment is illustrated;

[0020] Figure 7 and Figure 8 a first image sample set including images from a long-range camera is illustrated; Figure 7illustrates tele images of the first image set as processed by the deep learning module of the example embodiment; Figure 8 illustrates tele images of the first image set as processed by the fitting module of the example embodiment;

[0021] Figure 9 and Figure 10 illustrates a second image sample set including images from a wide camera; Figure 9 illustrates wide images of the second image set as processed by the deep learning module of the example embodiment; Figure 10 illustrates wide images of the second image set as processed by the fitting module of the example embodiment;

[0022] Figure 11 and Figure 12 illustrates a second image sample set including images from a mid camera; Figure 11 illustrates mid images of the second image set as processed by the deep learning module of the example embodiment; Figure 12 illustrates mid images of the second image set as processed by the fitting module of the example embodiment;

[0023] Figure 13 and Figure 14 illustrates a second image sample set including images from a tele camera; Figure 13 illustrates tele images of the second image set as processed by the deep learning module of the example embodiment; Figure 14 illustrates tele images of the second image set as processed by the fitting module of the example embodiment;

[0024] Figure 15 and Figure 16 illustrates a third image sample set including images from a wide camera; Figure 15 illustrates wide images of the third image set as processed by the deep learning module of the example embodiment; Figure 16 illustrates wide images of the third image set as processed by the fitting module of the example embodiment;

[0025] Figure 17 and Figure 18 illustrates a third image sample set including images from a mid camera; Figure 17 illustrates mid images of the third image set as processed by the deep learning module of the example embodiment; Figure 18 illustrates mid images of the third image set as processed by the fitting module of the example embodiment;

[0026] Figure 19 and Figure 20 illustrates a third image sample set including images from a tele camera; Figure 19illustrates a telephoto image of the third image set as processed by a deep learning module of an example embodiment; Figure 20 illustrates a telephoto image of the third image set as processed by a fitting module of an example embodiment;

[0027] Figure 21 illustrates an example embodiment as used in the context of a 3D image processing system for autonomous vehicles;

[0028] Figure 22 is a process flow diagram illustrating example embodiments of systems and methods for 3D object detection; and

[0029] Figure 23 shows an illustrative diagram of a machine in the example form of a computer system within which instructions sets can be executed to cause the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0030] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. It will be apparent, however, to one skilled in the art that the various embodiments can be practiced without

[0031] Disclosed herein is a system and method for three-dimensional (3D) object detection. Example systems and methods for 3D object detection can include a 3D image processing system configured to receive image data from at least one camera associated with an autonomous vehicle. As described herein, the 3D image processing system can be used to obtain 3D properties of objects detected in the image data, including the length, height, width, 3D spatial location in camera coordinate space (all in meters), and direction of movement (heading) of the objects. Details of various example embodiments are provided below.

[0032] Example embodiments disclosed herein can be used in the context of an in-vehicle control system 150 in a vehicle ecosystem 101. In one example embodiment, the in-vehicle control system 150 with a 3D image processing module 200 resident in a vehicle 105 can be configured like the architecture and ecosystem 101 illustrated in Figure 1 However, it will be apparent to those of ordinary skill in the art that the 3D image processing module 200 described and claimed herein can also be implemented, configured, and used in various other applications and systems.

[0033] Reference is now made to Figure 1, a block diagram illustrates an example ecosystem 101 in which the in-vehicle control system 150 and the 3D image processing module 200 of the example embodiments can be implemented. These components will be described in greater detail below. The ecosystem 101 includes various systems and components that can generate one or more information / data sources and related services, and / or deliver the information / data sources and related services to the in-vehicle control system 150 and the 3D image processing module 200 that can be installed in the vehicle 105. For example, one or more cameras installed in or on the vehicle 105 (as one of the devices of the vehicle subsystems 140) can generate images and timing data that can be received by the in-vehicle control system 150. The one or more cameras installed in or on the vehicle 105 can be equipped with various types of camera lenses (e.g., wide or close-up lenses, mid-range lenses, and long-range lenses) to capture images of the environment surrounding the vehicle 105. The in-vehicle control system 150 and the 3D image processing module 200 executing therein can receive this image and timing data input. As described in greater detail below, the 3D image processing module 200 can process the image input and enable the generation of 3D information associated with object features in the images that can be used by an autonomous vehicle control subsystem as another one of the subsystems of the vehicle subsystems 140. The autonomous vehicle control subsystem, for example, can use real-time 3D information associated with object features to safely and efficiently navigate and control the vehicle 105 in a real-world driving environment while avoiding obstacles and safely controlling the vehicle.

[0034] In example embodiments as described herein, the in-vehicle control system 150 can be in data communication with a plurality of vehicle subsystems 140, all of which can reside in the user’s vehicle 105. A vehicle subsystem interface 141 is provided to facilitate data communication between the in-vehicle control system 150 and the plurality of vehicle subsystems 140. The in-vehicle control system 150 can be configured to include a data processor 171 to execute a 3D image processing module 200 for processing image data received from one or more vehicle subsystems 140. The data processor 171 can be combined with a data storage device 172 as part of a computing system 170 in the in-vehicle control system 150. The data storage device 172 can be used to store data, processing parameters, camera parameters, terrain data, and data processing instructions. A processing module interface 165 can be provided to facilitate data communication between the data processor 171 and the 3D image processing module 200. In various example embodiments, a plurality of processing modules similar to the 3D image processing module 200 configured can be provided for execution by the data processor 171. As Figure 1As shown by the dashed lines in FIG. 1, the 3D image processing module 200 can be integrated into the in-vehicle control system 150, the 3D image processing module 200 can optionally be downloaded to the in-vehicle control system 150, or deployed separately from the in-vehicle control system 150.

[0035] The in-vehicle control system 150 can be configured to receive data from or transmit data to the wide area network 120 and network resources 122 connected to the wide area network 120. The in-vehicle web-enabled device 130 and / or the user mobile device 132 can be used to communicate via the network 120. The web-enabled device interface 131 can be used by the in-vehicle control system 150 to facilitate data communication between the in-vehicle control system 150 and the network 120 via the in-vehicle web-enabled device 130. Similarly, the user mobile device interface 133 can be used by the in-vehicle control system 150 to facilitate data communication between the in-vehicle control system 150 and the network 120 via the user mobile device 132. In this manner, the in-vehicle control system 150 can obtain real-time access to network resources 122 via the network 120. The network resources 122 can be used to obtain processing modules for execution by the data processor 171, data content to train internal neural networks, system parameters, or other data.

[0036] The ecosystem 101 can include a wide area data network 120. The network 120 represents one or more conventional wide area data networks, such as the Internet, a cellular telephone network, a satellite network, a pager network, a wireless broadcast network, a gaming network, a Wi-Fi network, a peer-to-peer network, a Voice over IP (VoIP) network, and the like. One or more of these networks 120 can be used to connect user or client systems with network resources 122, such as websites, servers, central control sites, and the like. The network resources 122 can generate and / or distribute data that can be received in the vehicle 105 via the in-vehicle web-enabled device 130 or the user mobile device 132. The network resources 122 can also host network cloud services that can support functionality used to compute or assist in processing image inputs or image input analysis. Antennae can be used to connect the in-vehicle control system 150 and the 3D image processing module 200 with the data network 120 via cellular, satellite, radio, or other conventional signal reception mechanisms. Such cellular data networks are currently available (e.g., Verizon TM , AT&T TM , T-Mobile TM , and the like). Such satellite-based data or content networks are also currently available (e.g., SiriusXM TM , HughesNet TMConventional broadcast networks (such as AM / FM radio networks, pager networks, UHF networks, gaming networks, WiFi networks, peer-to-peer networks, Voice over IP (VoIP) networks, and the like) are also well known. Thus, as described in greater detail below, the in-vehicle control system 150 and the 3D image processing module 200 can receive web-based data or content via the in-vehicle web-enabled device interface 131, which can be used to connect with the in-vehicle web-enabled device receiver 130 and the network 120. In this manner, the in-vehicle control system 150 and the 3D image processing module 200 can receive data from a variety of network-enabled in-vehicle devices and systems within the vehicle 105.

[0037] As shown in Figure 1 , the in-vehicle control system 150 and the 3D image processing module 200 can also receive data, image processing control parameters, and training content from user mobile devices 132, which can be located within or near the vehicle 105. The user mobile devices 132 can represent standard mobile devices, such as cellular phones, smart phones, personal digital assistants (PDAs), MP3 players, tablet computing devices (e.g., iPad TM ), laptop computers, CD players, and other mobile devices that can generate, receive, and / or communicate data, image processing control parameters, and content for the in-vehicle control system 150 and the 3D image processing module 200. As shown in Figure 1 , the mobile devices 132 can also be in data communication with the network cloud 120. The mobile devices 132 can seek data and content from internal memory components of the mobile devices 132 themselves or from network resources 122 via the network 120. Additionally, the mobile devices 132 themselves can include GPS data receivers, accelerometers, WiFi triangulation, or other geopositioning sensors or components in the mobile device that can be used to determine the real-time geopositioning of the user (via the mobile device) at any moment in time. In any case, as shown in Figure 1 , the in-vehicle control system 150 and the 3D image processing module 200 can both receive data from the mobile devices 132.

[0038] Still referring to Figure 1For embodiments implemented in a vehicle 105, many standard vehicles include operational subsystems such as electronic control units (ECUs), support monitoring / control subsystems for engines, brakes, transmissions, electrical systems, exhaust systems, interior environments, etc. For example, data signals communicated from the vehicle operational subsystems 140 (e.g., ECUs of the vehicle 105) to the in-vehicle control system 150 via the vehicle subsystem interface 141 can include information regarding the status of one or more of the components or subsystems of the vehicle 105. In particular, data signals that can be communicated from the vehicle operational subsystems 140 to a controller area network (CAN) bus of the vehicle 105 can be received and processed by the in-vehicle control system 150 via the vehicle subsystem interface 141. Embodiments of the systems and methods described herein can be used substantially with any mechanized system that uses a CAN bus or similar data communication bus as defined herein, including but not limited to industrial equipment, boats, trucks, machines, or cars; thus, the term "vehicle" as used herein can include any such mechanized system. Embodiments of the systems and methods described herein can also be used with any system that employs some form of network data communication; however, such network communication is not necessary.

[0039] Still referring to Figure 1 Example embodiments of the ecosystem 101 and the vehicle operational subsystems 140 therein can include various vehicle subsystems to support operation of the vehicle 105. Generally, the vehicle 105 can take the form of, for example, an automobile, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a lawn mower, a bulldozer, a snowmobile, an aircraft, a recreational vehicle, an amusement park vehicle, agricultural equipment, construction equipment, a trolley, a golf cart, a train, and a handcart. Other vehicles are possible. The vehicle 105 can be configured to operate completely or partially in an autonomous mode. For example, the vehicle 105 can self-control while in the autonomous mode and can be operable to determine a current state of the vehicle and an environment of the vehicle; determine a predicted behavior of at least one other vehicle in the environment; determine a confidence that can correspond to a likelihood that the at least one other vehicle will perform the predicted behavior; and control the vehicle 105 based on the determined information. While in the autonomous mode, the vehicle 105 can be configured to operate without human interaction.

[0040] The vehicle 105 can include various vehicle subsystems, such as a vehicle drive subsystem 142, a vehicle sensor subsystem 144, a vehicle control subsystem 146, and a passenger interface subsystem 148. As described above, the vehicle 105 can also include an in-vehicle control system 150, a computing system 170, and a 3D image processing module 200. The vehicle 105 can include more or fewer subsystems, and each subsystem can include multiple elements. Further, each of the subsystems and elements of the vehicle 105 can be interconnected. Thus, one or more of the described functions of the vehicle 105 can be divided into additional functions or physical components, or combined into fewer functions or physical components. In some other examples, additional functions and physical components can be added to those described by the vehicle 105, or some of those described can be removed. Figure 1 The illustrated example.

[0041] The vehicle drive subsystem 142 can include components operable to provide motive movement for the vehicle 105. In example embodiments, the vehicle drive subsystem 142 can include an engine or motor, wheels / tires, a transmission, electrical subsystems, and a power source. The engine or motor can be any combination of an internal combustion engine, an electric motor, a steam engine, a fuel cell engine, a propane engine, or other types of engines or motors. In some example embodiments, the engine can be configured to convert the power source into mechanical energy. In some example embodiments, the vehicle drive subsystem 142 can include multiple types of engines or motors. For example, a gasoline-electric hybrid vehicle can include a gasoline engine and an electric motor. Other examples are possible.

[0042] The wheels of the vehicle 105 can be standard tires. The wheels of the vehicle 105 can be configured in various formats, including, for example, single, double, triple, or four wheel formats such as on a car or truck. Other wheel geometries are possible, such as those including six or more wheels. Any combination of wheels of the vehicle 105 can be operable to rotate differently relative to the other wheels. The wheels can represent at least one wheel fixedly attached to a transmission and at least one tire coupled to a rim of the wheel, which can be in contact with a driving surface. The wheels can include a combination of metal and rubber or another combination of materials. The transmission can include elements operable to transmit mechanical power from an engine to the wheels. To this end, the transmission can include a gear box, a clutch, a differential, and a drive shaft. The transmission can also include other elements. The drive shaft can include one or more shafts that can be coupled to one or more wheels. The electrical system can include elements operable to carry and control electrical signals in the vehicle 105. These electrical signals can be used to activate the lights, servos, electric motors, and other electrically driven or controlled devices of the vehicle 105. The power source can represent an energy source that can provide power, in whole or in part, to the engine or electric motor. That is, the engine or electric motor can be configured to convert the power source into mechanical energy. Examples of power sources include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, fuel cells, solar panels, battery packs, and other sources of electrical power. The power source can additionally or alternatively include any combination of fuel tanks, battery packs, capacitors, or flywheels. The power source can also provide energy for other subsystems of the vehicle 105.

[0043] The vehicle sensor subsystem 144 can include a plurality of sensors configured to sense information about the environment or conditions of the vehicle 105. For example, the vehicle sensor subsystem 144 can include an inertial measurement unit (IMU), a global positioning system (GPS) transceiver, a RADAR unit, a laser rangefinder / LIDAR unit, and one or more cameras or image capture devices. The vehicle sensor subsystem 144 can also include sensors configured to monitor internal systems of the vehicle 105 (e.g., 02 monitor, fuel gauge, engine oil temperature). Other sensors are possible. One or more of the sensors included in the vehicle sensor subsystem 144 can be configured to be actuated individually or collectively in order to modify the position, orientation, or both of the one or more sensors.

[0044] The IMU can include any combination of sensors (e.g., accelerometers and gyroscopes) configured to sense changes in position and orientation of the vehicle 105 based on inertial acceleration. The GPS transceiver can be any sensor configured to estimate the geographic location of the vehicle 105. To this end, the GPS transceiver can include a receiver / transmitter operable to provide information about the position of the vehicle 105 relative to the earth. The RADAR unit can represent a system that utilizes radio signals to sense objects within the local environment of the vehicle 105. In some embodiments, in addition to sensing objects, the RADAR unit can additionally be configured to sense the speed and direction of travel of objects approaching the vehicle 105. The laser rangefinder or LIDAR unit can be any sensor configured to use laser light to sense objects in the environment in which the vehicle 105 is located. In example embodiments, the laser rangefinder / LIDAR unit can include one or more laser sources, a laser scanner, and one or more detectors, among other system components. The laser rangefinder / LIDAR unit can be configured to operate in either a coherent (e.g., using heterodyne detection) or non-coherent detection mode. The camera can include one or more devices configured to capture a plurality of images of the environment of the vehicle 105. The camera can be a still image camera or a motion video camera.

[0045] The vehicle control subsystem 146 can be configured to control the operation of the vehicle 105 and its components. Accordingly, the vehicle control subsystem 146 can include various elements such as a steering unit, a throttle, a braking unit, a navigation unit, and an autonomous control unit.

[0046] The steering unit can represent any combination of mechanisms that can be operable to adjust the direction of travel of the vehicle 105. The throttle can be configured to control the operational speed of, for example, an engine, and in turn, the speed of the vehicle 105. The braking unit can include any combination of mechanisms configured to slow the vehicle 105. The braking unit can use friction to slow the wheels in a standard manner. In other embodiments, the braking unit can convert the kinetic energy of the wheels into electrical current. The braking unit can also take other forms. The navigation unit can be any system configured to determine a driving path or route for the vehicle 105. The navigation unit can additionally be configured to dynamically update the driving path as the vehicle 105 is in operation. In some embodiments, the navigation unit can be configured to incorporate data from the 3D image processing module 200, a GPS transceiver, and one or more predetermined maps in order to determine a driving path for the vehicle 105. The autonomous control unit can represent a control system configured to identify, evaluate, and avoid or otherwise navigate around potential obstacles in the environment of the vehicle 105. Generally, the autonomous control unit can be configured to control the vehicle 105 to operate without a driver, or to provide driver assistance in controlling the vehicle 105. In some embodiments, the autonomous control unit can be configured to incorporate data from the 3D image processing module 200, a GPS transceiver, RADAR, LIDAR, cameras, and other vehicle subsystems to determine a driving path or trajectory for the vehicle 105. The vehicle control subsystem 146 can additionally or alternatively include components other than those shown and described.

[0047] The occupant interface subsystem 148 can be configured to allow interaction between the vehicle 105 and external sensors, other vehicles, other computer systems, and / or occupants or users of the vehicle 105. For example, the occupant interface subsystem 148 can include standard visual display devices (e.g., plasma displays, liquid crystal displays (LCDs), touch screen displays, heads-up displays, etc.), speakers or other audio output devices, microphones or other audio input devices, navigation interfaces, and interfaces for controlling the interior environment of the vehicle 105 (e.g., temperature, fans, etc.).

[0048] In example embodiments, occupant interface subsystem 148 can provide components for a user / occupant of vehicle 105 to interact with other vehicle subsystems, for example. A visual display device can provide information to a user of vehicle 105. The user interface device can also be operable to accept input from a user via a touchscreen. The touchscreen can be configured to sense at least one of position and movement of a user's finger via a capacitive sensing, resistive sensing, or surface acoustic wave process, among other possibilities. The touchscreen can be capable of sensing finger movement in a direction parallel or planar to the touchscreen surface, in a direction normal to the touchscreen surface, or both, and can also be capable of sensing a level of pressure applied to the touchscreen surface. The touchscreen can be formed from one or more semi-transparent or transparent insulating layers and one or more semi-transparent or transparent conductive layers. The touchscreen can also take other forms.

[0049] In other instances, occupant interface subsystem 148 can provide components for vehicle 105 to communicate with devices within the environment of vehicle 105. A microphone can be configured to receive audio (e.g., voice commands or other audio input) from a user of vehicle 105. Similarly, a speaker can be configured to output audio to a user of vehicle 105. In one example embodiment, occupant interface subsystem 148 can be configured to wirelessly communicate with one or more devices, either directly or via a communication network. For example, the wireless communication system can use 3G cellular communication (such as CDMA, EVDO, GSM / GPRS), or 4G cellular communication, such as WiMAX or LTE. Alternatively, the wireless communication system can communicate with a wireless local area network (WLAN), for example, using an infrared link, In some embodiments, vehicle control subsystem 146 can communicate directly with devices, for example, using a Bluetooth® link, or other wireless protocol. Other wireless protocols, such as various vehicle communication systems, are possible within the context of the present disclosure. For example, the wireless communication system can include one or more dedicated short range communication (DSRC) devices, which can include public or private data communication between vehicles and / or roadside stations.

[0050] Many or all of the functions of the vehicle 105 can be controlled by the computing system 170. The computing system 170 can include at least one data processor 171 (which can include at least one microprocessor) that executes processing instructions stored in a non-transitory computer-readable medium, such as a data storage device 172. The computing system 170 can also represent a plurality of computing devices that can be used to control various components or subsystems of the vehicle 105 in a distributed manner. In some embodiments, the data storage device 172 can contain processing instructions (e.g., program logic) that are executable by the data processor 171 to perform various functions of the vehicle 105, including those described herein in connection with the figures. The data storage device 172 can also contain additional instructions, including instructions to transmit data to, accept data from, interact with, or control one or more of the vehicle drive subsystem 142, the vehicle sensor subsystem 144, the vehicle control subsystem 146, and the occupant interface subsystem 148.

[0051] In addition to processing instructions, the data storage device 172 can also store data, such as image processing parameters, training data, road maps and path information, and other information. Such information can be used by the vehicle 105 and the computing system 170 during operation of the vehicle 105 in autonomous, semi-autonomous, and / or manual modes.

[0052] The vehicle 105 can include a user interface for providing information to or receiving input from a user or occupant of the vehicle 105. The user interface can control or enable control of the content and layout of interactive images that can be displayed on a display device. Further, the user interface can include one or more input / output devices within the collection of occupant interface subsystems 148, such as a display device, a speaker, a microphone, or a wireless communication system.

[0053] The computing system 170 can control functions of the vehicle 105 based on inputs received from various vehicle subsystems, such as the vehicle drive subsystem 142, the vehicle sensor subsystem 144, and the vehicle control subsystem 146, as well as from the occupant interface subsystem 148. For example, the computing system 170 can use inputs from the vehicle control subsystem 146 in order to control a steering unit, to avoid obstacles detected by the vehicle sensor subsystem 144 and the 3D image processing module 200, to move in a controlled manner, or to follow a path or trajectory based on outputs generated by the 3D image processing module 200. In example embodiments, the computing system 170 can be operable to provide control over many aspects of the vehicle 105 and its subsystems.

[0054] Although Figure 1 While various components of the vehicle 105, such as the vehicle subsystems 140, the computing system 170, the data storage device 172, and the 3D image processing module 200, are shown as being integrated into the vehicle 105, one or more of these components can be installed or associated separately from the vehicle 105. For example, the data storage device 172 can exist partially or entirely separate from the vehicle 105. Thus, the vehicle 105 can be provided in the form of device elements that can be located separately or together. The device elements that make up the vehicle 105 can be communicatively coupled together in a wired or wireless manner.

[0055] Additionally, other data and / or content, denoted herein as auxiliary data, can be obtained by the in-vehicle control system 150 from local and / or remote sources as described above. The auxiliary data can be used to augment, modify, or train the operation of the 3D image processing module 200 based on a variety of factors, including the context in which the user is operating the vehicle (e.g., the location of the vehicle, a designated destination, a direction of travel, a speed, a time of day, a state of the vehicle, etc.) and various other data that can be obtained from various sources (local and remote) as described herein.

[0056] In particular embodiments, the in-vehicle control system 150 and the 3D image processing module 200 can be implemented as in-vehicle components of the vehicle 105. In various example embodiments, the in-vehicle control system 150 and the 3D image processing module 200 in data communication with the in-vehicle control system 150 can be implemented as integrated components or separate components. In example embodiments, software components of the in-vehicle control system 150 and the 3D image processing module 200 can be dynamically updated, modified, and / or augmented using data connections with the mobile device 132 and / or the network resources 122 via the network 120. The in-vehicle control system 150 can periodically query the mobile device 132 or the network resources 122 for updates, or updates can be pushed to the in-vehicle control system 150.

[0057] Systems and methods for three-dimensional (3D) object detection

[0058] Disclosed herein is a system and method for three-dimensional (3D) object detection. An example system and method for 3D object detection can include a 3D image processing system 210 configured to receive image data from at least one camera associated with an autonomous vehicle. Example embodiments can be configured to output the locations of 2D bounding boxes around detected objects and the locations of eight corners that depict the size and orientation (heading direction) of the objects. As such, example embodiments can obtain 3D properties of objects detected in image data. This is an improvement over conventional systems that do not provide real-world 3D information. With geology information (e.g., road or terrain information) related to a particular environment and a camera calibration matrix, example embodiments can accurately compute the precise size and location of objects imaged by a camera in 3D coordinates. The camera calibration matrix can correspond to the way a particular camera is mounted on a vehicle and the configuration and orientation of images produced by the camera. Example embodiments run in real-time and serve as a key component in an autonomous driving perception system.

[0059] In various embodiments described herein, a 3D image processing module 200 (see Figure 21 ) of a 3D image processing system 210 is configured to address the aforementioned problems. As described herein, the 3D image processing module 200 can be used to obtain 3D properties of objects, including the length, height, width, 3D spatial location in camera coordinate space (all in meters), and moving direction (heading direction) of the objects. In example embodiments, the 3D image processing module 200 can include two sub-modules, namely: 1) a deep learning module 212 that learns the pixel coordinates of 2D bounding boxes and all vertices of 3D bounding boxes in image planes; and 2) a fitting module 214 that uses geology information from a terrain map and camera information (including a camera calibration matrix with extrinsic and intrinsic matrices of the camera) to solve for the 3D properties. The camera extrinsic matrix represents a coordinate system transformation from 3D world coordinates to 3D camera coordinates. The camera intrinsic matrix represents a coordinate system transformation from 3D camera coordinates to 2D image coordinates. Details are provided below and in connection with Figure 21 The deep learning module 212 and the fitting module 214 are described in more detail. The 3D image processing module 200 can run in real-time across multiple cameras and can significantly contribute to the perception pipeline and improve the robustness and safety level of autonomous driving systems. Details of various example embodiments are provided below.

[0060] In example embodiments, the 3D object detection problem can be defined as follows:

[0061] Given an input image I, for each object O n in a list of objects O = {01, 02,..., 0 i , output the following vector:

[0062] O i = {x top , y top , x bottom , y bottom , x 3d,1 , y 3d,1 ,..., x 3d,8 , y 3d,8 , h, w, l, X, Y, Z, Θ},

[0063] where xsand ysare pixel values in the image plane; top and bottom represent the top-left and bottom-right corners that define the 2D bounding box; x 3d s and y 3d s are the eight vertices of the 3D bounding box projected on the 2D image plane. The remaining values are simply 3D properties of the bounding box, including the height (h), width (w), length (l), location in the 3D world relative to the camera (X, Y, Z), and the heading direction orientation of the bounding box (Θ).

[0064] Figure 2 illustrates sample images showing the two-dimensional (2D) and three-dimensional (3D) bounding boxes of an object (e.g., a vehicle) in the image plane as produced by the 3D image processing module 200 of example embodiments. In the example of FIG. 3, an object (e.g., a vehicle) is shown with a 2D bounding box (yellow) and a 3D bounding box (green) as generated by the 3D image processing module 200 as described herein. Figure 2

[0065] In example embodiments of the 3D image processing module 200, a deep learning module 212 is used to learn the projected 3D bounding boxes in the image plane. A fitting module 214 uses the output of the deep learning module 212 together with the input of the corresponding camera matrix and terrain map data to produce the 3D properties of the objects in the input image set. The deep learning module 212 and the fitting module 214 will be described in more detail below.

[0066] Deep learning module for 3D bounding box generation

[0067] ​Deep learning based methods for learning 2D bounding boxes are well established and have been the state-of-the-art methods for many years. Typically, machine learning systems or neural networks are used to implement these deep learning based methods. However, such deep learning based methods for 3D object bounding boxes have not been employed in conventional systems. In the example embodiments described herein, the deep learning module 212 increases the number of points to be regressed (e.g., from 2 corners in a 2D model to 8 vertices in a 3D model). In example embodiments of the deep learning module 212, a new branch is added to learn the x and y coordinates of all the projected vertices of the 3D bounding box. The new branch can be trained together with the original 2D object detection architecture. In example embodiments, the task weights for classification, 2D bounding box regression, and 3D bounding box regression are set to (1:1:1) by default. All ground truths can be obtained by human annotation, and the order of the eight points can be predefined to facilitate feature learning. During the inference training of the deep learning module 212, a training image set can be input to the network (e.g., neural network) of the deep learning module 212, and all the x and y coordinates of the 2D and 3D bounding boxes of each object in the image can be obtained. After the training of the deep learning module 212, non-maximum suppression (NMS) is also applied to refine the bounding boxes and improve the prediction quality. The deep learning module 212 can run in real time at 40 fps (frames per second) for a single image, which meets the requirements of autonomous driving systems.

[0068] Fitting module

[0069] In example embodiments, the goal of the fitting module 214 is to lift the bounding box on the 2D image plane to the 3D space and obtain the 3D absolute properties of the object in the camera 3D coordinate space, including the height, width, length, distance to the camera, and orientation of the object. In an autonomous driving environment, it is highly challenging to directly lift the 2D information to the 3D coordinate space; this is because 1) the fitting module 214 cannot use the assumption of a flat ground since in a typical driving environment, the road always has a slope (up and down), and 2) the initial camera extrinsic matrix is not always reliable due to the possibility of severe vibration during driving. To address the first issue, the fitting module 214 can use a pre-computed or previously obtained terrain map that contains accurate global positioning system (GPS) locations with the height of the terrain from which the fitting module 214 can obtain accurate geographic information associated with the input image. To address the second issue, the fitting module 214 can use the output of the online calibration module, and thus can correct the camera matrix in real time. In this way, severe vibrations experienced by the autonomous vehicle can be corrected with respect to the camera matrix.

[0070] The 3D bounding box fitting process performed by the fitting module 214 of the example embodiments is described below:

[0071] 3D bounding box fitting process:

[0072] 1: Step fitting (image, bboxes):

[0073] 2: Obtain camera extrinsic matrix T and intrinsic matrix K.

[0074] 3: For each bbox in bboxes, do the following

[0075] 4: Obtain terrain value v.

[0076] 5: Set the origin to the bottom center of the bbox, get the coordinates of all eight points.

[0077] 6: Transform the bbox to camera coordinates using T.

[0078] 7: Project the eight points to the image plane using K.

[0079] 8: Solve the fitting problem using least square algorithm and prior v.

[0080] 9: End for

[0081] 10: Return 3D bboxes

[0082] 11: End step

[0083] In the 3D bounding box fitting process described above, the camera matrix (T and K) and terrain value v are obtained through calibration data source (e.g., online calibration module) and terrain map data source, respectively. Essentially, the fitting module 214 of the example embodiments is configured to minimize the difference between the output values of the eight points corresponding to the object from the deep learning module 212 and the projected values of the eight points from the 3D world as produced by the fitting module 214, while maintaining the assumption that the object is represented as a cuboid in 3D space. By solving the fitting problem, the fitting module 214 can obtain the optimal values of the unknown 3D properties of the object in the image and recover the 3D properties of the object in 3D space. To improve the robustness of the least square algorithm used in the example embodiments, the fitting module 214 can use predefined bounds for some variables (e.g., the height, width, and length of the vehicle object). For example, the predefined bounds for the variables of the vehicle object can all be greater than 1 meter and less than 5 meters and 50 meters. The solution of the fitting process as described herein is highly accurate and can also be applied in multi-camera tracking scenarios. In the example embodiments, the average processing speed of the fitting module 214 is about 2 milliseconds per image, which only adds a little overhead to the overall system and can meet the requirements of autonomous driving systems.

[0084] A sample illustration of 3D object detection data results 220 produced by example embodiments is shown in Figures 3-20 and described below.

[0085] Figure 3 and Figure 4 illustrates a first image sample set including images from a wide-angle camera; Figure 3 illustrates wide-angle images of the first image set as processed by the deep learning module 212 of example embodiments; Figure 4 illustrates wide-angle images of the first image set as processed by the fitting module 214 of example embodiments.

[0086] Figure 5 and Figure 6 illustrates a first image sample set including images from a mid-range camera; Figure 5 illustrates mid-range images of the first image set as processed by the deep learning module 212 of example embodiments; Figure 6 illustrates mid-range images of the first image set as processed by the fitting module 214 of example embodiments.

[0087] Figure 7 and Figure 8 illustrates a first image sample set including images from a long-range camera; Figure 7 illustrates long-range images of the first image set as processed by the deep learning module 212 of example embodiments; Figure 8 illustrates long-range images of the first image set as processed by the fitting module 214 of example embodiments.

[0088] In the first image sample set shown in Figures 3-8 , the cameras are facing in the same direction. Because different cameras of an autonomous vehicle can have different fields of view and detection ranges, the same object can appear at different locations in the image plane. For example, the vehicle in the middle of the mid-range camera image (see Figure 5 and Figure 6 ) appears in the upper half of the wide-angle camera image (see Figure 3 and Figure 4 ) and in the lower left corner of the long-range camera image (see Figure 7 and Figure 8 ). Figure 3 , Figure 5 and Figure 7 show the results of the processing performed by the deep learning module 212. Figure 4 , Figure 6 and Figure 8 show the projection results generated by the fitting module 214. Figure 3 , Figure 5 and Figure 7 each show green bounding boxes, which represent the results of 2D object detection.Figures 3-8 Each shows a blue cube around the detected object, where the blue cube represents the result of 3D detection produced by the 3D image processing module 200 as described herein. In Figures 3-8 In, the numbers written in yellow illustrate the order of the eight vertices used for 3D detection. The illustrated example only shows the bottom four vertices, and the top vertices are omitted for simplicity. In Figure 4 , Figure 6 and Figure 8 In, the fitting results are obtained by projecting the computed 3D properties back to the 2D image plane. For each bounding box, the red text describes the computed 3D object properties in the following order: vehicle height, width, length, distance (in z-axis), and orientation. For example, the middle vehicle shown in Figure 6 has a height of 1.6 meters, a width of 2.0 meters, a length of 3.7 meters, a distance of 28.4 meters, and an orientation of -88.5 degrees. The orientation of an object is measured as the angle between the object’s heading direction and the camera x-axis. If an object is driving forward in the same direction as the autonomous vehicle 105, the orientation angle will be -90 degrees. It is clear from the illustrated example images that even though the object is more than 200 meters away from the autonomous vehicle 105, the 3D image processing module 200 as described herein can still obtain accurate measurement data for 3D object properties. The 3D image processing module 200 can also effectively handle cases such as severe occlusion (see Figure 3 and Figure 4 ) and partial observation (see Figure 3 , Figure 4 , Figure 7 and Figure 8 ).

[0089] Figure 9 and Figure 10 illustrates a second image sample set including images from a wide-angle camera; Figure 9 illustrates a wide-angle image of the second image set as processed by the deep learning module 212 of the example embodiment; Figure 10 illustrates a wide-angle image of the second image set as processed by the fitting module 214 of the example embodiment.

[0090] Figure 11 and Figure 12 illustrates a second image sample set including images from a mid-range camera; Figure 11 illustrates a mid-range image of the second image set as processed by the deep learning module 212 of the example embodiment; Figure 12 illustrates a mid-range image of the second image set as processed by the fitting module 214 of the example embodiment.

[0091] Figure 13 and Figure 14a second image sample set including images from a tele camera is illustrated; Figure 13 a tele image of the second image set as processed by the deep learning module 212 of the example embodiment is illustrated; Figure 14 a tele image of the second image set as processed by the fitting module 214 of the example embodiment is illustrated.

[0092] Figure 15 and Figure 16 a third image sample set including images from a wide camera is illustrated; Figure 15 a wide image of the third image set as processed by the deep learning module 212 of the example embodiment is illustrated; Figure 16 a wide image of the third image set as processed by the fitting module 214 of the example embodiment is illustrated.

[0093] Figure 17 and Figure 18 a third image sample set including images from a mid camera is illustrated; Figure 17 a mid image of the third image set as processed by the deep learning module 212 of the example embodiment is illustrated; Figure 18 a mid image of the third image set as processed by the fitting module 214 of the example embodiment is illustrated.

[0094] Figure 19 and Figure 20 a third image sample set including images from a tele camera is illustrated; Figure 19 a tele image of the third image set as processed by the deep learning module 212 of the example embodiment is illustrated; Figure 20 a tele image of the third image set as processed by the fitting module 214 of the example embodiment is illustrated.

[0095] Reference is now made to Figure 21The example embodiments disclosed herein can be used in the context of a 3D image processing system 210 for autonomous vehicles. The 3D image processing system 210 can include, be included in, be executed by, or be executed with the 3D image processing module 200 as described above. The 3D image processing system 210 can include a deep learning module 212 and a fitting module 214 as described above. These modules can be implemented as processing modules, software or firmware elements, processing instructions, or other processing logic that implement any one or more of the methods or functions described and / or claimed herein. The 3D image processing system 210 and the 3D image processing module 200 therein can receive one or more image streams or image data sets from one or more cameras (block 205). As described above, the image data sets corresponding to raw image frames from the cameras are provided to the deep learning module 212 of the 3D image processing module 200. The deep learning module 212 can learn pixel coordinates of 2D bounding boxes of objects in the image plane and all vertices of 3D bounding boxes. The fitting module 214 can use geological information from a topographic map and camera information including camera calibration matrices with extrinsic and intrinsic matrices of the cameras to produce 3D properties of objects in the input images. The fitting module 214 can produce projection values of eight points of an object in 3D space. The 3D properties of the object can be provided as 3D object detection data 220 as output from the 3D image processing system 210 and the 3D image processing module 200 therein. Details of the processing performed by the 3D image processing module 200 are provided above.

[0096] Reference is now made to Figure 22 , a flowchart illustrates an example embodiment of a system and method 1000 for image processing. The example embodiment can be configured to receive image data from at least one camera associated with an autonomous vehicle, the image data representing at least one image frame (processing block 1010); use a trained deep learning module to determine pixel coordinates of a two-dimensional (2D) bounding box around an object detected in the image frame (processing block 1020); use the trained deep learning module to determine vertices of a three-dimensional (3D) bounding box around the object (processing block 1030); use a fitting module to obtain geological information related to a particular environment associated with the image frame and to obtain camera calibration information associated with the at least one camera (processing block 1040); and use the fitting module to determine 3D properties of the object using the 3D bounding box, the geological information, and the camera calibration information (processing block 1050).

[0097] As used herein and unless otherwise specified, the term "mobile device" includes any computing or communication device that can communicate with the in-vehicle control system 150 and / or 3D image processing module 200 described herein to obtain read or write access to data signals, messages, or content conveyed via any data communication mode. In many cases, the mobile device 130 is a handheld, portable device such as a smartphone, mobile phone, cellular phone, tablet computer, laptop computer, display pager, radio frequency (RF) device, infrared (IR) device, global positioning device (GPS), personal digital assistant (PDA), handheld computer, wearable computer, portable game console, other mobile communication and / or computing device, or an integrated device that combines one or more of the foregoing devices, etc. Additionally, the mobile device 130 can be a computing device, personal computer (PC), multi-processor system, microprocessor-based or programmable consumer electronic device, network PC, diagnostic device, system operated by a vehicle 119 manufacturer or service technician, etc., and is not limited to portable devices. The mobile device 130 can receive and process data in any of a variety of data formats. The data formats can include or be configured to operate with any programming format, protocol, or language, including but not limited to JavaScript, C++, iOS, Android, etc.

[0098] As used herein and unless otherwise specified, the term "network resource" includes any device, system, or service that can communicate with the in-vehicle control system 150 and / or 3D image processing module 200 described herein to obtain read or write access to data signals, messages, or content conveyed via any inter-process or networked data communication mode. In many cases, the network resource 122 is a data network accessible computing platform, including a client or server computer, a website, a mobile device, a peer-to-peer (P2P) network node, etc. Additionally, the network resource 122 can be a web appliance, a network router, switch, bridge, gateway, diagnostic equipment, a system operated by a vehicle 119 manufacturer or service technician, or any machine capable of executing a set of instructions (sequential or otherwise) that specifies actions to be taken by that machine. Further, while a single machine is illustrated, the term "machine" can also be understood to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. The network resource 122 can include any of a variety of providers or processors of network deliverable digital content. Typically, the file format employed is Extensible Markup Language (XML), however, the various embodiments are not so limited and other file formats can be used. For example, the various embodiments can support data formats other than HyperText Markup Language (HTML) / XML or formats other than open / standard data formats. The various embodiments described herein can support any electronic file format such as Portable Document Format (PDF), audio (e.g., Moving Picture Experts Group Audio Layer-3 - MP3, etc.), video (e.g., MP4, etc.), and any proprietary exchange format defined by a particular content site.

[0099] A wide area data network 120 (also denoted as a network cloud) used with network resources 122 can be configured to couple one computing or communication device to another. The network can be enabled to employ any form of computer-readable data or media for communication or transfer of information from one electronic device to another. In addition to other wide area networks (WAN), cellular telephone networks, metropolitan area networks, local area networks (LAN), other packet-switched networks, circuit-switched networks, direct data connections (such as through a universal serial bus (USB) or Ethernet port), other forms of computer-readable media or any combination thereof, the network 120 can also include the Internet. In addition to other wide area networks (WAN), cellular telephone networks, satellite networks, over-the-air broadcast networks, AM / FM radio networks, pager networks, UHF networks, other broadcast networks, gaming networks, WiFi networks, peer-to-peer networks, Voice over Internet Protocol (VoIP) networks, metropolitan area networks, local area networks (LAN), other packet-switched networks, circuit-switched networks, direct data connections (such as through a universal serial bus (USB) or Ethernet port), other forms of computer-readable media or any combination thereof, the network 120 can also include the Internet. On a group of interconnected networks, including networks based on different architectures and protocols, routers or gateways can act as links between networks, enabling messages to be sent between computing devices on different networks. Further, communication links within the network can typically include twisted pair

[0100] The network 120 can also include any of a variety of wireless sub-networks that can also cross over into independent ad hoc networks and the like to provide infrastructure-oriented connectivity. Such sub-networks can include mesh networks, wireless LAN (WLAN) networks, cellular networks, and the like. The network can also include autonomous systems of terminals, gateways, routers, and the like connected by wireless radio links or wireless transceivers. These connectors can be configured to move freely and randomly and organize themselves arbitrarily so that the topology of the network can change quickly. The network 120 can also employ one or more of a number of standards wireless and / or cellular protocols or access technologies, including the standards wireless and / or cellular protocols or access technologies set forth herein in connection with the network interface 712 and network 714 described in the figures herein.

[0101] In particular embodiments, mobile devices 132 and / or network resources 122 can act as client devices that enable users to access and use the in-vehicle control system 150 and / or the 3D image processing module 200 to interact with one or more components of a vehicle subsystem. These client devices 132 or 122 can actually include any computing device configured to send and receive information over a network, such as network 120 as described herein. Such client devices can include mobile devices such as cellular phones, smartphones, tablet computers, display pagers, radio frequency (RF) devices, infrared (IR) devices, global positioning devices (GPS), personal digital assistants (PDAs), handheld computers, wearable computers, game consoles, integrated devices combining one or more of the preceding devices, and the like. Client devices can also include other computing devices such as personal computers (PCs), multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, and the like. As such, client devices can vary widely in their capabilities and features. For example, a client device configured as a cellular phone can have a numeric keypad and a few rows of monochrome LCD display on which only text can be displayed. In another example, a web-enabled client device can have a touch sensitive screen, a stylus, and a color LCD screen on which both text and graphics can be displayed. Further, a web-enabled client device can include a browser application, which is able to receive and to send wireless application protocol messages (WAP) and / or wirelessly play multimedia TM messages with relevant information.

[0102] The client device can also include at least one client application configured to receive content or messages from another computing device via a network transmission. The client application can include the ability to provide and receive text content, graphical content, video content, audio content, alerts, messages, notifications, etc. Further, the client device can also be configured to communicate and / or receive messages between another computing device, such as through short message service (SMS), direct messaging (e.g., Twitter), email, multimedia message service (MMS), instant messaging (IM), Internet relay chat (IRC), mIRC, Jabber, enhanced messaging service (EMS), text messaging, smart messaging, over-the-air (OTA) messaging, etc. The client device can also include a wireless application device on which the client application is configured to enable a user of the device to send and receive information wirelessly to and from a network resource via a network.

[0103] The in-vehicle control system 150 and / or the 3D image processing module 200 can be implemented using a system that enhances the security of the execution environment, thereby improving security and reducing the likelihood that the in-vehicle control system 150 and / or the 3D image processing module 200 and related services can be compromised by viruses or malware. For example, the in-vehicle control system 150 and / or the 3D image processing module 200 can be implemented using a trusted execution environment, which can ensure that sensitive data is stored, processed, and communicated in a secure manner.

[0104] Figure 23 Shown is a diagrammatic representation of the machine in the example form of a computing system 700 within which instructions sets can be executed and / or processing logic can be launched, when executed, to cause the machine to perform any one or more of the methodologies described and / or claimed herein. In alternative embodiments, the machine operates as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate in the capacity of a server or a client machine in server-client network environments, or as a peer machine in peer-to-peer (or distributed) network environments. The machine can be a personal computer (PC), a laptop computer, a tablet computing system, a personal digital assistant (PDA), a cellular telephone, a smart phone, a web appliance, a set-top box (STB), a network router, switch or bridge, or any machine capable of executing instructions sets (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute instructions sets (or multiple sets of instructions) to perform any one or more of the methodologies described and / or claimed herein.

[0105] The example computing system 700 can include a data processor 702 (e.g., a system on a chip (SoC), general purpose processing core, graphics core, and optional other processing logic) and a memory 704 that can communicate with each other via a bus or other data transfer system 706. The mobile computing and / or communication system 700 can also include various input / output (I / O) devices and / or interfaces 710, such as a touchscreen display, audio jacks, voice interface, and optional network interface 712. In example embodiments, the network interface 712 can include one or more radio transceivers configured to operate with any one or more standard wireless and / or cellular protocols or access technologies, such as second generation (2G), 2.5 generation, third generation (3G), fourth generation (4G), and next generation wireless radio access of cellular systems, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), LTE, CDMA2000, WLAN, Wireless Router (WR) mesh, etc. The network interface 712 can also be configured to operate with various other wired and / or wireless communication protocols, including TCP / IP, UDP, SIP, SMS, RTP, WAP, CDMA, TDMA, UMTS, UWB, WiFi, WiMax, IEEE 802.1 lx, etc. In essence, the network interface 712 can include or support virtually any wired and / or wireless communication and data processing mechanisms by which information / data can be communicated between the computing system 700 and another computing or communication system via a network 714.

[0106] Memory 704 can be a machine-readable medium on which is stored one or more sets of instructions, software, firmware, or other processing logic embodying any one or more of the methodologies or functions described and / or claimed herein (e.g., logic 708). The logic 708 or portions thereof can also reside within the processor 702 during execution thereof, such that the processor 702 and the memory 704 can constitute a machine-readable medium embodying the functionality or logic 708. Accordingly, the memory 704 and the processor 702 can also constitute a machine-readable medium. Logic 708 or portions thereof can also be configured to operate as processing logic or logic that is at least partially embodied in hardware. Logic 708 or portions thereof can further be transmitted or received, e.g., over the network 714 via the network interface 712. Although the machine-readable medium of an example embodiment can be a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and processing systems) that store the one or more sets of instructions. The term "machine-readable medium" shall also be taken to include any non- transitory medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present embodiments, or that is capable of storing, encoding or carrying data structures utilized by or associated with such instructions. The term "machine-readable medium" shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0107] The abstract of the disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that the abstract will not be used to interpret or limit the scope or the meaning of the claims. Additionally, in the foregoing Detailed Description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are explicitly recited in each claim. Rather, as the following claims reflect, inventive subject matter can lie in fewer than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, where each claim can stand as a separate embodiment.

Claims

1. A system comprising: Data processor; as well as A 3D image processing system, which can be executed by the data processor, is configured to: Image data is received from at least one camera associated with the autonomous vehicle, the image data representing at least one image frame; A trained deep learning module is used to determine the pixel coordinates of the two-dimensional (2D) bounding boxes around the objects detected in the image frame; The trained deep learning module is used to determine the vertices of the three-dimensional (3D) bounding box around the object; The fitting module is used to obtain geological information related to a specific environment associated with the image frame and to obtain camera calibration information associated with the at least one camera. as well as The fitting module is used to determine the 3D properties of the object using the 3D bounding box, the geological information, and the camera calibration information.

2. The system of claim 1 is further configured to provide the 3D attributes of the object to the autonomous driving perception system.

3. The system of claim 1, wherein the at least one camera comprises a camera lens of the type comprising the group consisting of: wide-angle or close-up lenses, medium-range lenses, and telephoto lenses.

4. The system of claim 1, wherein the vertices of the three-dimensional (3D) bounding box surrounding the object are determined in pixel coordinates.

5. The system of claim 1, wherein the geological information is obtained from a topographic map, the topographic map including GPS locations having the elevation of the terrain.

6. The system of claim 1, wherein the camera calibration information includes a camera calibration matrix having camera non-inherent and inherent matrices.

7. The system of claim 1, wherein the 3D attributes of the object include object length, height, width, 3D spatial location, and the direction of travel of the object.

8. A method comprising: Image data is received from at least one camera associated with the autonomous vehicle, the image data representing at least one image frame; A trained deep learning module is used to determine the pixel coordinates of the two-dimensional (2D) bounding boxes around the objects detected in the image frame; The trained deep learning module is used to determine the vertices of the three-dimensional (3D) bounding box around the object; The fitting module is used to obtain geological information related to a specific environment associated with the image frame and to obtain camera calibration information associated with the at least one camera. as well as The fitting module is used to determine the 3D properties of the object using the 3D bounding box, the geological information, and the camera calibration information.

9. The method of claim 8, comprising: The 3D attributes of the object are provided to the autonomous driving perception system.

10. The method of claim 8, wherein the at least one camera comprises a camera lens of the type comprising the group consisting of: wide-angle or close-up lenses, medium-range lenses, and telephoto lenses.

11. The method of claim 8, wherein the vertices of the three-dimensional (3D) bounding box surrounding the object are determined in pixel coordinates.

12. The method of claim 8, wherein the geological information is obtained from a topographic map, the topographic map including GPS locations having the elevation of the terrain.

13. The method of claim 8, wherein the camera calibration information includes a camera calibration matrix having camera non-inherent and inherent matrices.

14. The method of claim 8, wherein the 3D attributes of the object include object length, height, width, 3D spatial location, and the direction of travel of the object.

Images