Positioning application for ADAS calibration target

Abstract

A method for calibrating an advanced driver-assistance system (ADAS) for a vehicle comprises: receiving first information by a mobile device having a camera and a light detection and ranging (LiDAR) device, the first information including three-dimensional (3D) specifications of the physical dimensions of the vehicle having the ADAS; scanning physical targets for the vehicle and ADAS by the mobile device, the scan being performed using the camera and LiDAR device after receiving the first information, and the scan generating frame data; tracking the vehicle in the frame data using the first information by the mobile device; generating an output by the mobile device indicating whether the physical targets are positioned at the target location for the ADAS; and starting the ADAS calibration process using the physical targets after the output has been generated.

Description

Technical Field

[0004] , , ,

[0003] ,

[0001] [Cross - Reference to Related Applications] This application claims priority to U.S. Provisional Patent Application No. 63 / 495,872, filed Apr. 13, 2023, entitled "Positioning Application for ADAS Calibration Target", and is a continuation of, and claims priority to, U.S. Non - Provisional Patent Application No. 18 / 328,511, filed Jun. 2, 2023, entitled "Positioning Application for ADAS Calibration Target", the entire disclosures of which are incorporated herein by reference.

[0002] This application also claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 495,872, filed Apr. 13, 2023, the entire disclosure of which is incorporated herein by reference.

[0003] This document relates to a positioning application for an advanced driver assistance system (ADAS) calibration target.

Background Art

[0004] Some vehicles manufactured in recent years are equipped with an advanced driver assistance system (ADAS) that can at least partially handle operations related to driving the vehicle. The ADAS can automatically survey the surrounding of the vehicle and take actions regarding the detected vehicles, pedestrians, or objects. Calibration is performed to configure the ADAS. However, existing calibration systems are expensive, complex, inefficient, inaccurate, and / or occupy a large floor area in a service center. In one previous approach, a target for ADAS calibration is placed on a stand that is positioned relative to the vehicle. The positioning is done by placing another target on the vehicle's wheel, capturing an image of this other target using a camera on the stand, and re - positioning the stand to the target for ADAS calibration.

Summary of the Invention

[0005] In a first embodiment, a method for calibrating an advanced driver-assistance system (ADAS) for a vehicle comprises: receiving first information by a mobile device having a camera and a light detection and ranging (LiDAR) device, the first information including three-dimensional (3D) specifications of the physical dimensions of a vehicle having the ADAS; scanning a physical target for the vehicle and the ADAS by the mobile device, the scan being performed using the camera and LiDAR device after receiving the first information, and the scan generating frame data; tracking the vehicle in the frame data using the first information by the mobile device; generating an output by the mobile device indicating whether the physical target is positioned at a target location for the ADAS; and, after the output has been generated, initiating a calibration process for the ADAS using the physical target.

[0006] An implementation may include any or all of the following features: The step of generating an output includes presenting a camera feed on the display of a mobile device and adding a virtual target in the camera feed, the virtual target being positioned according to a calibration specification for the ADAS. If multiple virtual targets are specified by the calibration specification, the method further comprises the step of selecting a virtual target from among the multiple virtual targets to add to the camera feed. The method further comprises the step of rendering a virtual overlay on the vehicle in the camera feed, the virtual overlay being positioned based on the tracking of the vehicle in the frame data. The method further comprises the step of receiving second information, including a 3D specification for a physical target, by the mobile device; and the step of also tracking the physical target in the frame data by the mobile device, the physical target being tracked using the second information, wherein the output indicates whether the physical target is positioned at a target location for the ADAS based on the tracking of the physical target. The method further comprises the step of the mobile device performing a determination, based on the tracking of the physical target, that the position of the physical target substantially coincides with a target location for the ADAS, wherein the output is generated based on the determination. The mobile device is a handheld device or a wearable device. The method is a one-person process performed on the mobile device, where the person holds the mobile device and observes (i) a camera feed including the physical target and (ii) a virtual target on the display while maneuvering a physical target. The first information includes a computer-aided design model of the vehicle.

[0007] In a second embodiment, a method for inspecting a vehicle comprises: receiving first and second information by a mobile device having a camera and a light detection and ranging (LiDAR) device, the first information including three-dimensional (3D) specifications of the physical dimensions of the vehicle, and the second information including 3D specifications of the physical components of the vehicle; scanning the vehicle by the mobile device, the scan being performed using the camera and LiDAR device after receiving the first and second information, and the scan generating frame data; tracking the vehicle and physical components in the frame data using the first and second information by the mobile device; and generating a presentation using the frame data by the mobile device, the presentation including camera feeds of at least a portion of the vehicle, and 3D representations of the physical components overlaid on the camera feeds, the 3D representations being positioned based on tracking.

[0008] The implementation may include any or all of the following features: The method further comprises the step of rendering a virtual overlay on a vehicle in a camera feed, the virtual overlay being positioned based on tracking of the vehicle in frame data; both the vehicle and the virtual overlay are visible in the presentation; the virtual overlay includes a perimeter for a portion of the vehicle in the camera feed, and further comprises the step of performing alignment verification between the virtual overlay and the vehicle in the camera feed. The mobile device is a handheld or wearable device. The method is a one-person process performed on a mobile device, where a person holds the mobile device and observes on the mobile device's display (i) a camera feed including physical components, and (ii) a 3D representation of the physical components. The first information includes a computer-aided design model of the vehicle. [Brief explanation of the drawing]

[0009] [Figure 1]A schematic example of a calibration assistant used to facilitate ADAS calibration using mobile devices such as handheld devices is shown.

[0010] [Figure 2] This shows an example of a presentation that can be performed by a calibration assistant to facilitate ADAS calibration.

[0011] [Figure 3] This example demonstrates the use of a calibration assistant to facilitate ADAS calibration. [Figure 4] This example demonstrates the use of a calibration assistant to facilitate ADAS calibration. [Figure 5] This example demonstrates the use of a calibration assistant to facilitate ADAS calibration. [Figure 6] This example demonstrates the use of a calibration assistant to facilitate ADAS calibration.

[0012] [Figure 7] Here is an example of how to perform ADAS calibration.

[0013] [Figure 8] This document provides an example of a data flow and technology stack for facilitating ADAS calibration using a calibration assistant.

[0014] [Figure 9A] This example demonstrates how to position an ADAS calibration target in a single-person process. [Figure 9B] This example demonstrates how to position an ADAS calibration target in a single-person process.

[0015] [Figure 10A] This shows an example of vehicle inspection. [Figure 10B] This shows an example of vehicle inspection. [Figure 11] This shows an example of vehicle inspection. [Figure 12] This shows an example of vehicle inspection. [Figure 13] An example of inspecting a vehicle is shown. [Figure 14] An example of inspecting a vehicle is shown.

[0016] [Figure 15] An example of a vehicle is shown.

[0017] [Figure 16] An exemplary architecture of a computing device that can be used to implement aspects of the present disclosure is illustrated.

[0018] Like reference numerals in the various drawings indicate like elements.

Mode for Carrying Out the Invention

[0019] This document describes examples of systems and techniques for calibrating the ADAS of a vehicle. In some implementations, a technician aims a mobile device at a vehicle and uses a calibration assistant application running on the mobile device for guidance in orienting one or more physical ADAS targets towards the vehicle. By placing the targets, the vehicle can perform a calibration procedure for its ADAS.

[0020] Generally, during an ADAS calibration procedure, the ADAS activates some or all of its sensors and registers the signals detected thereby in relation to one or more physical ADAS targets near the vehicle. For example, a passive sensor such as a camera can capture an image of a physical ADAS target. In another example, an active sensor such as a light detection and ranging (LiDAR) device or radar emits a signal from the vehicle and can detect the resulting reflection. The detected signals can then be processed to generate calibration settings to be applied to that particular vehicle.

[0021] ADAS systems are typically calibrated first during the vehicle manufacturing process. A manufacturing plant may have physical ADAS targets positioned within tunnels through which vehicles pass during manufacturing, allowing the vehicle's ADAS to detect the targets and perform calibration procedures while inside the tunnel. Since the tunnels are part of the assembly line, the spatial location of vehicles within the tunnels is controlled and does not experience significant variation between vehicles. For this reason, it is not necessary to reposition the physical ADAS targets for every vehicle in the manufacturing plant. Rather, each vehicle is transported to a stationary physical ADAS target for calibration.

[0022] After a vehicle leaves the manufacturing plant, the ADAS may require subsequent calibration (recalibration) during service sessions. For example, ADAS calibration may be performed after the ADAS has been removed or cut off from the vehicle; after the removal of a rearview mirror which may contain a camera; after the removal of a bumper which may contain a camera and / or LiDAR; after the removal of the windshield or rear window; and / or after the removal of the rear bumper. Furthermore, at a service center, the spatial position of a vehicle may not be as precisely controlled as on an assembly line; therefore, a physical ADAS target must be transported to its position relative to the specific vehicle before each calibration session. If the physical ADAS target is not in the exact position according to the tolerance specifications for a particular ADAS when the calibration procedure is attempted, the calibration may fail and an error status may occur. This subject provides improvements to the positioning of the physical ADAS target relative to the vehicle for the success of the calibration procedure. For example, the process according to this subject can be less expensive to implement, easier to execute, occupy less space, and / or be more precise compared to previous approaches.

[0023] The examples herein refer to vehicles. A vehicle is a machine that transports passengers, cargo, or both. A vehicle may have one or more motors that use at least one type of fuel or other energy source (e.g., electricity). Examples of vehicles include, but are not limited to, cars, trucks, and buses. The number of wheels may vary between types of vehicles, and one or more (e.g., all) of the wheels may be used for propulsion of the vehicle, or the vehicle may not have power (e.g., if a trailer is attached to another vehicle). A vehicle may include a passenger compartment that accommodates one or more persons. At least one passenger of a vehicle may be considered the driver; in that case, various tools, instruments, or other devices may be provided to the driver. In the examples herein, any person transported by a vehicle may be referred to as the “driver” or “passenger” of the vehicle, regardless of whether that person is driving the vehicle, or whether that person has access to the controls for driving the vehicle, or whether that person lacks the controls for driving the vehicle. The vehicles in this example are shown for illustrative purposes only, and are depicted as being similar to or identical to one another.

[0024] The examples herein refer to ADAS. In some implementations, ADAS can perform driver assistance and / or autonomous driving. ADAS can automate at least one or more dynamic driving tasks. ADAS can operate in part on the outputs of one or more sensors typically located on, under, or inside the vehicle. ADAS can plan one or more trajectories for the vehicle before and / or while controlling the vehicle's motion. The planned trajectories can define the path for the vehicle to travel. Thus, propelling the vehicle according to the planned trajectories may correspond to controlling one or more aspects of the vehicle's operating behavior, such as the vehicle's steering angle, gear (e.g., forward or reverse), speed, acceleration, and / or braking.

[0025] Autonomous vehicles are an example of ADAS, but not all ADAS are designed to provide fully autonomous vehicles. SAE International has defined several levels of driver automation, typically referred to as Level 0, 1, 2, 3, 4, and 5. For example, a Level 0 system or driving mode does not require continuous vehicle control by the system. For example, a Level 1 system or driving mode may include adaptive cruise control, emergency braking assist, automatic emergency braking assist, lane keeping, and / or lane centering. For example, a Level 2 system or driving mode may include highway assistance, autonomous obstacle avoidance, and / or autonomous parking. For example, a Level 3 or 4 system or driving mode may include incremental control of the vehicle by the driver assistance system. For example, a Level 5 system or driving mode may not require human intervention in the driver assistance system.

[0026] The examples herein refer to sensors. A sensor is configured to detect one or more aspects of its environment and to output a signal that reflects this detection. The detected aspects can be static or dynamic at the time of detection. As merely illustrative examples, a sensor may indicate one or more of the following: the distance between the sensor and an object, the speed of a vehicle holding the sensor, the trajectory of the vehicle, or the acceleration of the vehicle. A sensor may also generate an output without probing its surroundings using anything (e.g., passive detection, such as an image sensor that captures electromagnetic radiation), or the sensor may probe its surroundings (e.g., active detection by transmitting electromagnetic radiation and / or sound waves) and detect a response to the probe. Examples of sensors that may be used with one or more embodiments include, but are not limited to, optical sensors (e.g., cameras); light-based sensing systems (e.g., optical ranging and detection (LiDAR) devices); radio wave-based sensors (e.g., radar); acoustic sensors (e.g., ultrasonic devices and / or microphones); inertial measuring units (e.g., gyroscopes and / or accelerometers); speed sensors (e.g., for vehicles or their components); position sensors (e.g., for vehicles or their components); orientation sensors (e.g., for vehicles or their components); torque sensors; thermal sensors, temperature sensors (e.g., primary or secondary thermometers); pressure sensors (e.g., for the ambient air or components of a vehicle); humidity sensors (e.g., rain detectors); or seating sensors.

[0027] Figure 1 schematically shows an example 100 of a calibration assistant used to facilitate ADAS calibration using a mobile device such as a handheld device. Example 100 may be used in conjunction with one or more other examples described elsewhere in this specification. Example 100 involves a vehicle 102 having ADAS to be recalibrated at a service center. A technician uses a calibration assistant running on a mobile device 104 to obtain guidance on positioning one or more physical ADAS targets. The mobile device 104 may be a handheld device (e.g., a smartphone or tablet) or a wearable device, to give only two examples. The mobile device 104 presents a camera feed 106 on its display, which includes an image 108 of the vehicle 102. The mobile device 104 can add one or more virtual targets 110 to the camera feed 106. In some implementations, the virtual targets 110 are augmented reality (AR) content to guide the technician in positioning the physical ADAS targets relative to the vehicle.

[0028] In some implementations, the calibration assistant is specific to a particular operating system (including but not limited to Android® or iOS®) and uses computer vision to detect the vehicle 102 and track it in real-world three-dimensional (3D) space, for example, using cameras and LiDAR. The calibration assistant can track the vehicle 102 using information about the physical vehicle (e.g., a 3D model such as a computer-aided design (CAD) file). For example, such information / CAD file may include 3D specifications of physical dimensions. The calibration assistant can use virtual tools to map both the 3D space inside and around the vehicle 102, and the vehicle 102 itself. The calibration assistant can provide AR guidance for the technician in the placement and positioning of physical ADAS targets relative to the vehicle 102. These physical ADAS targets are then used to calibrate multiple different ADAS sensors on the vehicle 102. Example 100 illustrates a much simpler and less costly method for positioning physical ADAS targets. In contrast, previous approaches may require expensive hardware, the placement of specialized targets on the vehicle itself (different from ADAS targets), and / or the use of large floor spaces that may not be available in all service centers.

[0029] Figure 2 shows an example of a presentation 200 that may be given by a calibration assistant to facilitate ADAS calibration. The presentation 200 can be used with one or more other examples described elsewhere in this specification. For example, the mobile device 104 in Figure 1 can generate a presentation 200 for a technician as AR guidance for positioning a physical ADAS target.

[0030] Presentation 200 includes a camera feed 202 showing an image 204 of a vehicle based on 3D tracking of the physical vehicle in a real 3D space. The camera feed 202 is generated as a live stream of video content when the technician points a mobile device (running the calibration assistant) at the vehicle. One or more virtual targets 206 can be added to the camera feed 202 as AR content. The virtual targets 206 can be positioned on the screen according to the calibration specifications for the ADAS. The virtual targets 206 can have any shape, including but not limited to 3D boxes, two-dimensional (2D) planes, and / or lines or points. The calibration assistant can provide one or more controls 208 and / or 210 for operating Presentation 200. For example, control 208 is used to select which virtual targets 206 appear in Presentation 200 by switching them on or off, and / or to control their appearance. In some implementations, control 210 can modify the guide view to assist the user in aligning the vehicle according to user convenience. An example of a guide view is shown below with reference to Figure 10A.

[0031] Figures 3 to 6 illustrate examples of using a calibration assistant to facilitate ADAS calibration. The illustrated examples can be used in conjunction with one or more other examples described elsewhere in this specification. In Figure 3, a camera feed 300 is captured using a mobile device and currently shows an image 302 of the vehicle. The technician points the mobile device at the vehicle and receives AU guidance to position at least one physical target 304 relative to the vehicle. The physical target 304 can have any shape or size that fits the vehicle's ADAS. Here, the physical target 304 includes a 3x3 pattern of alternating dark and bright areas of the same size. For example, the physical target 304 can be mounted on a stand 306. The stand 306 can have one or more physical targets 304, and in this example, there are two instances.

[0032] Figure 4 shows that a virtual target 400 has been added to the camera feed 300 based on vehicle tracking in 3D space. The camera feed 300 and the virtual target 400 can be maintained as frame data by the mobile device and continuously updated. Any of the controls 208 can be used to control which virtual target 400 is added to the camera feed 300 and / or to control the appearance of any of the virtual targets 400 (for example, to select a transparent or opaque texture for the virtual target 400). The vehicle image 302 shown in Figure 3 can here be completely or partially hidden by a virtual overlay 302'. For example, the virtual overlay 302' can cover the entire shape of the vehicle or only a part of the shape. For example, the virtual overlay 302' can have the same shape as the vehicle (part) or a different shape.

[0033] Figure 5 shows that the camera feed 300 currently includes a virtual overlay 302' for the tracked vehicle, and among the virtual targets 400, virtual target 402 in particular. For ADAS calibration to be performed successfully, the physical target 304, which is also visible in the camera feed 300, should be positioned at the location of the virtual target 402 (for example, so that they substantially coincide with each other). Currently, the physical target 304 is not precisely positioned at the location of the virtual target 402. Therefore, the stand 306 can be repositioned under the AR guidance of the calibration assistant. In some implementations, the calibration assistant tracks only the vehicle in 3D space and does not track the physical target 304. The technician can then visually observe the camera feed 300 and virtual target 402 on the display to determine where the physical target 304 should be positioned. In other implementations, the calibration assistant can track at least both the vehicle and the physical target 304 in 3D space, and the calibration assistant can then generate an output indicating whether the physical target 304 is currently positioned in the correct location. For example, the output may be visual and / or auditory, or the absence of a particular output may itself serve as such an indication.

[0034] Figure 6 shows that the physical target 304 has been repositioned relative to the tracked vehicle so that it is at the location of the virtual target 402 (for example, so that they substantially coincide with each other). The calibration assistant can guide the technician to position the stand 306 so that the physical target 304 is precisely located. In implementations where the physical target 304 is also tracked in 3D space, the calibration assistant can generate an output indicating whether the physical target 304 is now precisely located. For example, the output may be visual and / or auditory, or the absence of a particular output itself may be such an indication. Thus, the calibration assistant can track the physical target 304 using information about the physical target 304 (for example, a 3D model such as a CAD file). Based on the tracking of the physical target 304, the calibration assistant can make a determination that the position of the physical target 304 substantially coincides with the target position of the virtual target 402.

[0035] The calibration process can be initiated when it is determined (for example, by a technician through visual inspection or automatically by a calibration assistant based on tracking) that the position of the physical target 304 substantially coincides with the target position of the virtual target 402. In some implementations, the calibration process can be initiated from the same mobile device running the calibration assistant.

[0036] Figure 7 shows an example of method 700 for performing ADAS calibration. Method 700 may be used in conjunction with one or more other examples described elsewhere in this specification. More or fewer operations than shown may be performed. Unless otherwise indicated, two or more operations may be performed in a different order.

[0037] In operation 702, the vehicle can be scanned using a calibration assistant on a mobile device. For example, the technician can point the mobile device 104 in Figure 1 at the vehicle 102 and identify the virtual targets 206 (Figure 2) and / or 400 (Figure 4) and / or 402 (Figures 5-6).

[0038] In operation 704, the technician can position a physical target relative to the vehicle based on guidance from the calibration assistant. For example, the technician positions the virtual target 304 using the stand 306 in Figure 5. Positioning can be completed when the technician determines that the physical target is correctly positioned, or when instructed to do so by the calibration assistant.

[0039] In operation 706, the vehicle's ADAS calibration process can be initiated. This may involve running a diagnostic application on the vehicle's computer system. For example, the vehicle attempts to locate a physical target using its sensors. The tolerance for the placement of the physical target can be specified by the ADAS settings. In previous approaches, calibration target placement was based on locations determined using a tape measure. In that case, the ADAS settings for target position tolerances were sometimes set with relatively lenient limits to account for variations in the target placement procedure. On the other hand, this subject can provide substantially more precise placement. As a result, the ADAS settings for target position tolerances can be made stricter.

[0040] The example above illustrates that a method for calibrating an ADAS for a vehicle may include a step of receiving first information (e.g., a CAD file) by a mobile device having a camera and a LiDAR device (e.g., mobile device 104 in Figure 1). The first information includes 3D specifications of the physical dimensions of the vehicle having the ADAS. The method may include a step of scanning a physical target (e.g., physical target 402 in Figure 5) for the vehicle and ADAS by the mobile device, the scanning step being performed using the camera and LiDAR device after receiving the first information. The scan generates frame data. The method may include a step of tracking the vehicle in the frame data using the first information by the mobile device (e.g., calibration tracks an image 302 of the vehicle in 3D space based on its CAD file). The method may include a step of generating an output by the mobile device indicating whether the physical target is positioned at the target location for the ADAS. For example, a video / audio output may be generated. The output can guide the placement of the physical target at its precise location. The method may include a step in which, after the output is generated, a calibration process for the ADAS using a physical target is initiated. For example, the physical target 402 in Figure 6 can be used to calibrate the ADAS.

[0041] Figure 8 shows an example data flow and technology stack 800 for facilitating ADAS calibration using a calibration assistant. Example 800 can be used in conjunction with one or more other examples described elsewhere in this specification. To initiate ADAS calibration for a vehicle 102, a calibration assistant on a mobile device 104 can be used. The mobile device 104 comprises a component 802 including at least a camera (e.g., a smartphone camera) and a LiDAR. Operation 804 instructs the camera to generate frame data and the LiDAR to create a depth map. In operation 806, pixel format conversion and / or flattening can be performed. For example, the color data of the frame data can be combined with the depth data of the depth map to generate converted frame data 808.

[0042] The trained object database 810 can contain information about one or more objects that can be tracked. In some implementations, the trained object database 810 specifies at least the physical dimensions (or other visually recognizable characteristics) of a vehicle. Physical targets can also be specified for tracking. Objects can be specified using one of several different types of files. In some implementations, at least one of the vehicle or physical target is specified by a CAD file. The vehicle is tracked in terms of where the vehicle image is located in the camera feed, and as a result, virtual targets can be added to the camera feed at a specific distance and orientation relative to the vehicle image. Physical targets can be tracked so that the calibration assistant can determine whether and when the location of the physical target substantially coincides with the location of the virtual target.

[0043] The object tracker 812 can perform tracking of one or more objects based on a trained object database 810. The object tracker 812 may be an object recognition library. The object tracker 812 can track objects by comparing the transformed frame data 808 with the definitions of one or more objects in the trained object database 810. The object tracker 812 can generate a virtual map of the tracked object in 3D space. An engineer can move a mobile device relative to a vehicle (or another tracked object) to generate a more complete virtual map in the cache. Thus, operations performed by the object tracker 812 may include detecting objects similar to those found in the trained object database 810, and then tracking the detected objects. To track more than one object (e.g., both a vehicle and a physical target), the object tracker 812 can perform continuous tracking to maintain simultaneous tracking of both objects. For example, another thread can be run in the object tracker 812 to handle one or more tracked objects.

[0044] In the rendering stage 814, one or more virtual features are rendered using the converted frame data 808 and the tracking information from the object tracker 812. For example, one of the virtual targets 206 (Figure 2), 400 (Figure 4), and / or 402 (Figures 5-6) can be rendered. As another example, a virtual overlay can be rendered.

[0045] The Unity-based ADAS Assistant Calibration Assistant application 816 can receive and use event-based signals from the converted frame data 808 and object tracker 812, and can provide multiple handles or features using the processed frame data. The ADAS Calibration Assistant application 816 is built using Unity, a 3D graphics application development framework for building graphics on top of the processed frame data. The Unity-based ADAS Assistant Calibration Assistant application 816 can query tracking status objects (e.g., whether their status is tracked or partially tracked), update application logic, render graphics on top of the frame data containing tracking data, and render virtual targets for tracked objects in 3D space. Screens 818 and 820 are screenshots from a particular user session. For example, screen 818 shows a user session with all actively tracked vehicles and enabled virtual targets. As another example, screen 820 shows a user session with all enabled virtual targets except the actively tracked vehicles and virtual floor mat targets.

[0046] Figures 9A and 9B illustrate an example of positioning an ADAS calibration target in a one-person process. The example involves a service center 900 (viewed from above) and a mobile device 104, and can be used in conjunction with one or more other examples described elsewhere in this specification. In these illustrations, the mobile device 104 is shown both in the service center 900 and separately, clarifying how the technician uses the mobile device 104 and observes its output.

[0047] Vehicle 902 is located at the service center 900, and ADAS calibration is to be performed for vehicle 902. A physical target 304 mounted on a stand 306 is presented at the service center 900. The technician holds a mobile device 104 containing a camera 904 and a LiDAR 906. As an alternative example, the mobile device 104 may be mounted on a tripod or other stand. For example, both the camera 904 and the LiDAR 906 may be mounted in front of or behind the mobile device 104. The camera 904 and LiDAR 906 now have views of both vehicle 902 and physical target 304. The mobile device 104 presents a screen 908 containing the camera feed from camera 904 and one or more virtual features. Screen 908 includes an image 910 of vehicle 902 and an image 912 of physical target 304. The calibration assistant can track vehicle 902 and optionally track physical target 304. The calibration assistant can add a virtual target 402 to the screen 908 at a specific distance and orientation relative to the image 910 of the vehicle 902, based on tracking. Here, the virtual target 402 is also shown to the service center 900 for clarity regarding the spatial relationship. The mobile device 104 can provide an output (e.g., a screen 908 observable by the technician) indicating that the physical target 304 is not currently located at the location of the virtual target 402.

[0048] The technician can move the stand 306 to reposition the physical target 304 based on the output from the mobile device 104. The mobile device 104 can present a screen 914 in which the image 912 of the physical target 304 substantially matches the virtual target 402. For example, if the physical target 304 has a handle long enough to be operated, or if the physical target 304 is wirelessly controllable, the technician can observe screens 908 and 914 while holding the mobile device 104 in one hand and moving the stand 306 with the other. As another example, the technician can remotely operate the stand 306 while viewing the screen on the mobile device 104 mounted on the stand. As yet another example, the screens 908 and 914 of the mobile device 104 mounted on the stand themselves can be mirrored to another display device that is visible to the technician during operation. If the mobile device 104 tracks both the vehicle 902 and the physical target 304, the calibration assistant can determine when the physical target 304 was positioned in its current location. Therefore, the calibration assistant on the mobile device 104 can facilitate a one-person process for ADAS calibration.

[0049] Figures 10A–10B and 11–14 illustrate examples 1000–1008 of inspecting vehicle 1010. Such inspections can enable a technician to gain insights into the vehicle for the purpose of repairing, evaluating, or otherwise investigating vehicle 1010. In example 1000, vehicle 1010 is visible in a camera feed captured using a mobile device which also has LiDAR. The mobile device can access object information about vehicle 1010 (e.g., a CAD file) and begin tracking vehicle 1010 in 3D space as the mobile device moves. In example 1000, a virtual overlay 1012' is added to the camera feed to act as a guide for aligning the detected vehicle. Herein, the virtual overlay 1012' has the same spatial orientation as vehicle 1010 (e.g., front to right, rear to left), but is not yet fitted to or precisely aligned with vehicle 1010. Referring again briefly to Figure 2, the virtual overlay 1012' may be a guide view presented in response to the user activating one of the controls 210 to select from among the available guide views, where the guide view is a side view of the vehicle. Other guide views, including but not limited to a front view or a rear view, may also be available. For example, if there is not enough physical space at the current location to align the vehicle from a side view, the user may instead select a front view guide view or a rear view guide view to align the vehicle.

[0050] The view displayed on the device may include photographic content, augmented reality (AR) content, or a combination of both. Figure 10B shows example 1001 (i.e., photographic content) where the hood 1003, wheels 1005, and trunk lid 1007 are parts of an actual vehicle. On the other hand, the body structure 1009 and wheel perimeter 1011 are AR content added as overlays on the photographic image. Thus, both photographic and AR content can be displayed together. For example, in an area presented as a door opening by AR content, the outside of the door 1013 of the photographic content is visible.

[0051] Here, the wheel circumference 1011 is aligned with the circumference of the wheel 1005 in the photographic content. For example, this allows for verification that the alignment of the AR content with the photographic content is performed correctly. In addition or alternatively, one or more other parts of the vehicle may provide the circumference of the AR content (e.g., the hood 1003 or the trunk lid 1007).

[0052] In Example 1002, the virtual overlay 1012 is instead added to the camera feed based on tracking the vehicle using camera frame data and a LiDAR depth map. For example, the virtual overlay 1012 can give the vehicle a different appearance (e.g., color, shape, or other texture) than in Example 1000. Controls 1014 can be provided. For example, each of the controls 1014 is associated with one or more physical components of the vehicle 1010. Such physical components may include, but are not limited to, the body structure, high-voltage systems, low-voltage systems, suspension, or powertrain. The controls 1014 can be operated to overlay a 3D representation of the physical component on the camera feed at a precise spatial position based on tracking the vehicle.

[0053] In Example 1004, the control was activated to add a virtual overlay 1016 representing its body structure to the vehicle. Next, Example 1006 illustrates that the virtual overlay 1016 can represent the vehicle's door openings 1018 and structural members 1020 (e.g., crash rails). The door openings 1018 and structural members 1020 of the virtual overlay 1016 are now positioned relative to the vehicle based on tracking in 3D space. Finally, Example 1008 indicates that the structural members 1020 are visible in the AR view on a mobile device. If the vehicle is now disassembled so that the physical members (whose structural members 1020 are virtual 3D representations) are visible, the engineer can observe both the physical members and the structural members 1020 on the screen. If these objects do not match, the engineer can conclude that the physical members are not in their design position and may be damaged or improperly installed. Thus, an inspection can be performed to detect physical damage on the vehicle.

[0054] Figure 15 shows an example of vehicle 1500. Vehicle 1500 may be used in conjunction with one or more other examples described elsewhere in this specification. Vehicle 1500 includes ADAS 1502 and vehicle control unit 1504. ADAS 1502 may be implemented using some or all of the components described below with reference to Figure 16. ADAS 1502 includes sensors 1506 and planning algorithm 1508. Other embodiments of vehicle 1500, including but not limited to other components of vehicle 1500 in which ADAS 1502 may be implemented, are omitted here for simplicity.

[0055] Sensor 1506 is described herein, including appropriate circuitry and / or executable programming for processing the sensor output and performing detections based on that processing. Sensor 1506 may include radar 1510. In some implementations, radar 1510 may include any object detection system that is at least partially based on radio waves. For example, radar 1510 may be oriented forward relative to the vehicle and may be used to detect at least the distance to one or more other objects (e.g., another vehicle). Radar 1510 can detect the surrounding conditions of vehicle 1500 by detecting the presence of objects in relation to vehicle 1500.

[0056] Sensor 1506 may include an active light sensor 1512. In some implementations, the active light sensor 1512 may include an arbitrary object detection system that is at least partially based on laser light. For example, the active light sensor 1512 may be oriented in any direction relative to the vehicle and may be used to detect the distance to at least one or more other objects (e.g., lane boundaries). The active light sensor 1512 can detect the surrounding conditions of the vehicle 1500 by detecting the presence of objects in relation to the vehicle 1500. To give just two examples, the active light sensor 1512 may be a scanning LiDAR or a non-scanning LiDAR (e.g., a flash LiDAR).

[0057] Sensor 1506 may include camera 1514. In some implementations, camera 1514 may include any image sensor whose signal the vehicle 1500 considers. For example, camera 1514 may be oriented in any direction relative to the vehicle and may be used to detect the vehicle, lanes, lane markings, curbs and / or road signs. Camera 1514 may detect the surrounding conditions of vehicle 1500 by visually recording the situation in relation to vehicle 1500.

[0058] Sensor 1506 may include an ultrasonic sensor 1516. In some implementations, the ultrasonic sensor 1516 may include any transmitter, receiver, and / or transceiver used to detect the proximity of at least an object based on ultrasound. For example, the ultrasonic sensor 1516 may be located on or near the exterior of the vehicle. The ultrasonic sensor 1516 can detect the surrounding conditions of the vehicle 1500 by detecting the presence of an object in relation to the vehicle 1500.

[0059] Regardless of whether ADAS 1502 controls the movement of vehicle 1500, any of the sensors 1506, individually or collectively, two or more of the sensors 1506, can detect the surrounding conditions of vehicle 1500. In some implementations, at least one of the sensors 1506 may produce an output that is considered when providing alerts or other prompts to the driver and / or when controlling the movement of vehicle 1500. For example, the outputs of two or more sensors (e.g., radar 1510, active light sensor 1512, and camera 1514) may be combined. In some implementations, one or more other types of sensors may be included in addition to or instead of sensors 1506.

[0060] The planning algorithm 1508 may plan whether the ADAS 1502 will perform one or more actions or no actions at all, in response to monitoring of the surrounding environment of the vehicle 1500 and / or input from the driver. One or more outputs of the sensor 1506 may be taken into consideration. In some implementations, the planning algorithm 1508 may perform a motion plan for the vehicle 1500 and / or plan its trajectory.

[0061] The vehicle control unit 1504 may include a steering control unit 1518. In some implementations, the ADAS 1502 and / or another driver of the vehicle 1500 control the trajectory of the vehicle 1500 by operating the steering control unit 1518 to adjust the steering angle of at least one wheel. The steering control unit 1518 may be configured to control the steering angle through a mechanical connection between the steering control unit 1518 and an adjustable wheel, or it may be part of a steer-by-wire system.

[0062] The vehicle control unit 1504 may include a gear control unit 1520. In some implementations, the ADAS 1502 and / or another driver of the vehicle 1500 use the gear control unit 1520 to select from several operating modes of the vehicle (e.g., driving mode, neutral mode, or parking mode). For example, the gear control unit 1520 may be used to control automatic transmission in the vehicle 1500.

[0063] The vehicle control unit 1504 may include a signal control unit 1522. In some implementations, the signal control unit 1522 may control one or more signals that the vehicle 1500 may generate. For example, the signal control unit 1522 may control the vehicle 1500's headlights, turn signals, and / or horn.

[0064] The vehicle control unit 1504 may include a brake control unit 1524. In some implementations, the brake control unit 1524 may control one or more types of braking systems designed to decelerate the vehicle, stop the vehicle, and / or keep the vehicle stopped if it is stopped. For example, the brake control unit 1524 may be actuated by an ADAS 1502. In another example, the brake control unit 1524 may be actuated by the driver using the brake pedal.

[0065] The vehicle control unit 1504 may include a vehicle dynamics system 1526. In some implementations, the vehicle dynamics system 1526 may control one or more functions of the vehicle 1500 in addition to, in the absence of, or on behalf of, the driver's control. For example, if the vehicle comes to a stop on a hill and the driver does not activate the brake control unit 1524 (e.g., by pressing the brake pedal), the vehicle dynamics system 1526 may keep the vehicle stopped.

[0066] The vehicle control unit 1504 may include an acceleration control unit 1528. In some implementations, the acceleration control unit 1528 may control one or more types of propulsion motors of the vehicle. For example, the acceleration control unit 1528 may control the electric motor and / or internal combustion motor of the vehicle 1500.

[0067] The vehicle control unit may further include one or more additional control units, collectively shown here as control unit 1530. Control unit 1530 may provide vehicle control of one or more functions or components. In some implementations, control unit 1530 may adjust one or more sensors of the vehicle 1500. For example, the vehicle 1500 may adjust sensor settings (e.g., frame rate and / or resolution) based on ambient environment data measured by these sensors and / or any other sensors of the vehicle 1500.

[0068] The vehicle 1500 may include a user interface 1532. The user interface 1532 may include an audio interface 1534 that can be used to generate alerts regarding detections. In some implementations, the audio interface 1534 may include one or more speakers located within the passenger compartment. For example, the audio interface 1534 may operate at least partially in conjunction with an infotainment system in the vehicle.

[0069] The user interface 1532 may include a visual interface 1536 that can be used to generate alerts regarding detection. In some implementations, the visual interface 1536 may include at least one display device in the passenger compartment of the vehicle 1500. For example, the visual interface 1536 may include a touchscreen device and / or an instrument cluster display.

[0070] Figure 16 shows an exemplary architecture of a computing device 1600 that can be used to implement aspects of the present disclosure, including any of the systems, apparatus, and / or techniques described herein, or any other systems, apparatus, and / or techniques that may be used in various possible embodiments.

[0071] The computing device shown in Figure 16 can be used to run the operating systems, application programs, and / or software modules (including software engines) described herein.

[0072] In some embodiments, the computing device 1600 comprises at least one processing device 1602 (e.g., a processor), such as a central processing unit (CPU). Various processing devices are available from various manufacturers (e.g., Intel or Advanced Micro Devices). In this example, the computing device 1600 also includes system memory 1604 and a system bus 1606 that connects various system components, including the system memory 1604, to the processing device 1602. The system bus 1606 is one of any number of bus structures that can be used, including, but are not limited to, a memory bus, or a memory controller; peripheral bus; and local bus, using any of various bus architectures.

[0073] Examples of computing devices that can be implemented using computing device 1600 include desktop computers, laptop computers, tablet computers, mobile computing devices (such as smartphones, touchpad mobile digital devices, or other mobile devices), or other devices configured to process digital instructions.

[0074] The system memory 1604 includes a read-only memory 1608 and a random-access memory 1610. A basic input / output system 1612, which includes basic routines that function to transfer information within the computing device 1600 during startup, etc., can be stored in the read-only memory 1608.

[0075] In some embodiments, the computing device 1600 also includes an auxiliary storage device 1614 (e.g., a hard disk drive) for storing digital data. The auxiliary storage device 1614 is connected to the system bus 1606 by an auxiliary storage interface 1616. The auxiliary storage device 1614 and its associated computer-readable medium provide non-volatile and non-temporary storage for computer-readable instructions, data structures, and other data (including application programs and program modules) for the computing device 1600.

[0076] In the examples of environments described herein, a hard disk drive is used as an auxiliary storage device, but in other embodiments, other types of computer-readable storage media are used. Examples of these other types of computer-readable storage media include magnetic cassettes, flash memory cards, solid-state drives (SSDs), digital video discs, Bernoulli cartridges, compact disk read-only memory, digital multipurpose disk read-only memory, random access memory, or read-only memory. Some embodiments include non-temporary media. For example, a computer program product can be tangibly embodied in a non-temporary storage medium. Furthermore, such computer-readable storage media may include local storage or cloud-based storage.

[0077] Multiple program modules can be stored in the auxiliary storage device 1614 and / or in the system memory 1604, which includes the operating system 1618, one or more application programs 1620, other program modules 1622 (such as a software engine as described herein), and program data 1624. The computing device 1600 can utilize any suitable operating system.

[0078] In some embodiments, the user provides input to the computing device 1600 through one or more input devices 1626. Examples of input devices 1626 include a keyboard 1628, a mouse 1630, a microphone 1632 (for, e.g., voice and / or other audio input), a touch sensor 1634 (such as a touchpad or touch-sensitive display), and a gesture sensor 1635 (for, e.g., gesture input). In some implementations, the input devices 1626 provide detection based on presence, proximity, and / or motion. Other embodiments include other input devices 1626. The input devices can be connected to the processing device 1602 through an input / output interface 1636 coupled to the system bus 1606. These input devices 1626 can be connected by any number of input / output interfaces, such as parallel ports, serial ports, game ports, or a universal serial bus. Wireless communication between the input device 1626 and the input / output interface 1636 is also possible, and in some possible embodiments, to name just a few, includes infrared, Bluetooth® wireless technology, 802.11a / b / g / n, cellular, ultra-wideband (UWB), ZigBee®, or other radio frequency communication systems.

[0079] In this exemplary embodiment, a display device 1638, such as a monitor, liquid crystal display device, light-emitting diode display device, projector, or touch-sensitive display device, is also connected to the system bus 1606 via an interface such as a video adapter 1640. In addition to the display device 1638, the computing device 1600 may include various other peripheral devices (not shown), such as speakers or printers.

[0080] The computing device 1600 can connect to one or more networks through the network interface 1642. The network interface 1642 can provide wired and / or wireless communication. In some implementations, the network interface 1642 may include one or more antennas for transmitting and / or receiving wireless signals. When used in a local area networking environment or a wide area networking environment (such as the Internet), the network interface 1642 may include an Ethernet® interface. In other possible embodiments, other communication devices are used. For example, some embodiments of the computing device 1600 include a modem for communication over a network.

[0081] The computing device 1600 may include at least some form of computer-readable media. Computer-readable media include any available media that the computing device 1600 can access. Examples of computer-readable media include computer-readable storage media and computer-readable communication media.

[0082] Computer-readable storage media include volatile and non-volatile, removable and fixed media configured to store information such as computer-readable instructions, data structures, program modules, or other data, and which are implemented in any device. Computer-readable storage media include, but are not limited to, random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory, or other memory technologies, compact disk read-only memory, digital versatile disk, or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or any other magnetic storage device, or other media that can be used to store desired information and are accessible by computing device 1600.

[0083] Computer-readable communication media typically embody computer-readable instructions, data structures, program modules, or other data in modulated data signals (e.g., carrier waves or other means of transport), and this includes all information distribution media. The term “modulated data signal” refers to a signal in which one or more of its characteristics are set or modified in a manner that encodes information into a signal. Examples of computer-readable communication media include wired media (e.g., wired networks or direct wired connections) and wireless media (e.g., acoustic media, radio frequency media, infrared media, and other wireless media). Any combination of the above also falls within the scope of computer-readable media.

[0084] The computing device shown in Figure 16 is also an example of a programmable electronic device, which may include one or more such computing devices. If multiple computing devices are included, such computing devices may be connected to one another using an appropriate data communication network in order to jointly perform the various functions, methods, or operations disclosed herein.

[0085] In some implementations, the computing device 1600 can be characterized as an ADAS computer. For example, the computing device 1600 may include one or more components that are sometimes used to handle tasks arising in the field of artificial intelligence (AI). In that case, the computing device 1600 includes sufficient processing power and the necessary supporting architecture for the demands of ADAS or AI in general. For example, the processing device 1602 may include a multi-core architecture. As another example, the computing device 1600 may include one or more coprocessors in addition to, or as part of, the processing device 1602. In some implementations, at least one hardware accelerator can be coupled to the system bus 1606. For example, a graphics processing unit can be used. In some implementations, the computing device 1600 may implement neural network-specific hardware to handle one or more ADAS tasks.

[0086] The terms “substantially” and “about” as used throughout this specification are used to describe and account for small variations, such as those resulting from processing variability. For example, they may mean less than or equal to ±5%, less than or equal to ±2%, less than or equal to ±1%, less than or equal to ±0.5%, less than or equal to ±0.2%, less than or equal to ±0.1%, less than or equal to ±0.05%. Also, as used herein, indefinite articles such as “a” or “an” mean “at least one.”

[0087] It should be understood that all combinations of the aforementioned concepts and any additional concepts discussed in more detail below (provided that such concepts are not mutually contradictory) are intended to be part of the subject matter of the invention disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are intended to be part of the subject matter of the invention disclosed herein.

[0088] Several implementations have been described. Nevertheless, it should be understood that various modifications may be made without deviating from the intent and scope of this specification.

[0089] Furthermore, the logical flow shown in the diagram does not require a specific or sequential order to achieve the desired result. In addition, other processes may be provided, or processes may be excluded from the described flow; other components may be added to or removed from the described system. Therefore, other implementations fall within the scope of the following claims.

[0090] While specific features of the described implementations have been shown as described herein, many modifications, substitutions, alterations, and equivalents will now come to mind for those skilled in the art. It should be understood that the appended claims are intended to encompass all such modifications and alterations that fall within the scope of these implementations. They are presented merely as examples and not as limitations, and it should be understood that various modifications in form and detail are possible. Any part of the apparatus and / or method described herein may be combined in any combination, except for mutually exclusive combinations. The implementations described herein may include various combinations and / or partial combinations of the functions, components, and / or features of the different implementations described herein.

Claims

1. A method for calibrating an advanced driver assistance system (ADAS) for a vehicle, comprising: The step of receiving first information by a mobile device having a camera and a light detection and ranging (LiDAR) device, wherein the first information includes three-dimensional (3D) specifications of the physical dimensions of the vehicle having the ADAS; The mobile device performs the step of scanning for physical targets of the vehicle and the ADAS, the scanning step is performed using the camera and the LiDAR device after receiving the first information, and the scanning step generates frame data; The step of tracking the vehicle in the frame data using the first information via the mobile device; The steps include: generating an output by the mobile device indicating whether the physical target is positioned at the target location for the ADAS; and After the output is generated, the ADAS calibration process using the physical target is initiated. A method for providing this.

2. The method according to claim 1, wherein the step of generating the output includes presenting a camera feed on the display of the mobile device, adding a virtual target in the camera feed, and positioning the virtual target according to a calibration specification for the ADAS.

3. The method according to claim 2, wherein a plurality of virtual targets are specified by the calibration specification, and the method further comprises the step of selecting a virtual target from the plurality of virtual targets in order to add the virtual target to the camera feed.

4. The method according to claim 2, further comprising the step of rendering a virtual overlay on the vehicle in the camera feed, wherein the virtual overlay is positioned based on the tracking of the vehicle in the frame data.

5. The steps include: receiving second information, including 3D specifications for the physical target, via the mobile device; and In the step where the mobile device also tracks the physical target in the frame data, the physical target is tracked using the second information, where the output indicates whether the physical target is positioned at the target position for the ADAS based on the tracking of the physical target. The method according to claim 2, further comprising:

6. The mobile device performs a determination, based on the tracking of the physical target, that the position of the physical target substantially coincides with the target position for the ADAS, wherein the output is generated based on the determination. The method according to claim 5, further comprising:

7. The method according to claim 2, wherein the mobile device is a handheld device or a wearable device.

8. The method according to claim 7, wherein the method is a one-person process performed on the mobile device, wherein the person holds the mobile device and, while manipulating the physical target, observes on the display (i) the camera feed including the physical target and (ii) the virtual target.

9. The method according to any one of claims 1 to 8, wherein the first information includes a computer-aided design model of the vehicle.

10. A method for inspecting a vehicle: The steps include receiving first and second information by a mobile device having a camera and a light detection and ranging (LiDAR) device, wherein the first information includes three-dimensional (3D) specifications of the physical dimensions of the vehicle, and the second information includes 3D specifications of the physical components of the vehicle; The mobile device performs the step of scanning the vehicle, the scanning step is performed using the camera and the LiDAR device after receiving the first and second information, and the scanning step generates frame data; The steps include: tracking the vehicle and physical components in the frame data using the first and second information via the mobile device; and The mobile device generates a presentation using the frame data, the presentation includes camera feeds of at least a portion of the vehicle, and a 3D representation of the physical components overlaid on the camera feeds, the 3D representation being positioned based on the tracking. A method for providing this.

11. The method according to claim 10, further comprising the step of rendering a virtual overlay on the vehicle in the camera feed, wherein the virtual overlay is positioned based on the tracking of the vehicle in the frame data.

12. The method according to claim 11, wherein both the vehicle and the virtual overlay are visible in the presentation.

13. The method according to claim 12, wherein the virtual overlay includes the outer perimeter of the portion of the vehicle in the camera feed, and further comprises the step of performing a verification of the alignment between the virtual overlay and the vehicle in the camera feed.

14. The method according to any one of claims 10 to 13, wherein the mobile device is a handheld or wearable device.

15. The method according to claim 14, wherein the method is a one-person process performed on the mobile device, wherein the person holds the mobile device and observes on the display of the mobile device (i) the camera feed including the physical component, and (ii) the 3D representation of the physical component.

16. The method according to any one of claims 10 to 13, wherein the first information includes a computer-aided design model of the vehicle.

Images