System and method for trailer posture estimation

Abstract

The present application relates to systems and methods for trailer pose estimation. The present teachings relate to methods, systems, media and embodiments for trailer pose estimation. In a case where a vehicle and a trailer have corresponding relative surfaces facing each other, dimensions of at least a trailer connected to the vehicle are obtained. A first set of feature points is identified from a first image of a target present on one of the relative surfaces acquired at a first moment by a sensor deployed on the other relative surface. A corresponding second set of feature points is identified from a second image of the target acquired by the sensor at a subsequent moment. Based on the first set of feature points and the second set of feature points, a first transformation is determined and used to calculate a second transformation experienced by the trailer from the first moment to the second moment. The pose of the trailer at the second moment is estimated based on the second transformation and the dimensions.

Description

Technical Field

[0001] The present teachings generally relate to autonomous driving. More specifically, the present teachings relate to autonomous driving of a fleet of vehicles. Background Art

[0002] With the development of sensing and artificial intelligence (AI) technologies, autonomous driving has gradually become a potentially meaningful alternative to conventional manual driving in recent years. Through modern sensing technologies and advanced sensing information processing deployed on mobile vehicles, mobile vehicles can "see" their surroundings and obstacles that need to be avoided by analyzing the on-the-fly information received from sensors. Figure 1A An example of what the ego vehicle sees while moving is shown. As can be seen, there are several small hills in the distance ahead. Close up ahead, there are fields and a road with lane markings. On the road, there is a string of moving vehicles entering in the opposite direction, and some vehicles driving in the same lane ahead of the ego vehicle. This is an image provided by a vision sensor mounted, for example, on the front of the ego vehicle. More vision sensors can be mounted on both sides of the ego vehicle, providing surrounding visual information such as can be observed from both sides of the vehicle. There may also be other types of sensors, such as LiDAR and / or radar, which can be used to sense depth information, which is crucial for avoiding obstacles.

[0003] The sensor information thus obtained can then be analyzed in real time to sense the surroundings. This is necessary to enable the vehicle to decide how to avoid obstacles. Figure 1B An example is shown in FIG. Visual information captured in front of the host vehicle is analyzed to detect obstacles, including several other moving vehicles 100, 105, 110, and 115 in different lanes in the forward direction and a roadblock 102 on one side of the road. To facilitate avoidance, relevant features associated with each detected obstacle may also be detected via different means. For example, for each detected moving vehicle, different measurements may be made based on its distance from the host vehicle, its position relative to the road, and its dimensions. Figure 1B As shown in , vehicle 100 is measured 12 meters away in the front lane with an estimated dimension of 4 feet by 7 feet. These measurements are presented on the detected vehicle as 12m / FL / 4X7. Similarly, for the further vehicle 110, its measurement is 15m / RL / 5X8, i.e., 15 meters away in the right lane with an estimated dimension of 5 feet by 8 feet (SUV). Vehicle 115 is closer at 3m / RL / 4X7 (3 meters away in the right lane with an estimated dimension of 4 feet by 7 feet). This semantics of the surrounding provides the basis for the autonomous vehicle to determine how to safely drive without hitting any obstacles in the lane on the road.

[0004] The technology required for autonomous driving may be different in different traffic environments. For example, the technology developed for autonomous driving of, for example, a car or SUV may not be sufficient for autonomous driving of a truck or a convoy of trucks. For a truck or a convoy, there may be special challenges that require different solutions. For example, since a truck may not be able to break as quickly as a car or turn as agilely as a car, it needs to "see" farther so that it can plan each action earlier in time. Trucks often need to know more about their surroundings in a larger geographic area so that they can plan ahead. Most of the traditional technologies for autonomous driving cannot solve the unique problems associated with trucks and convoys of trucks. Therefore, there is a need to develop technical solutions that can enable autonomous driving of trucks and convoys of trucks. Summary of the invention

[0005] The teachings disclosed herein relate to methods, systems and programming for data processing. More particularly, the teachings relate to methods, systems and programs related to modeling a scene to generate scene modeling information and its utilization.

[0006] In one example, a method for pose estimation implemented on a machine, the machine comprising at least one processor, a storage device, and a communication platform capable of connecting to a network. In the case where a vehicle and a trailer have corresponding relative surfaces facing each other, at least the dimensions of the trailer connected to the vehicle are obtained. A first set of feature points is identified from a first image of a target present on one of the relative surfaces acquired at a first moment by a sensor deployed on the other relative surface. A second set of feature points is identified from a second image of the target acquired by the sensor at a subsequent moment. Based on the first set of feature points and the second set of feature points, a first transformation is determined and used to calculate a second transformation experienced by the trailer from the first moment to the second moment. The pose of the trailer at the second moment is estimated based on the second transformation and the dimensions.

[0007] In a different example, the present teaching discloses a system for pose estimation, the system comprising a configuration unit, a target image feature extractor and a transformation generation unit, and a trailer pose estimator. The configuration unit is configured to obtain at least the dimensions of a trailer connected to a vehicle, wherein the vehicle and the trailer have corresponding relative surfaces facing each other. The target image feature extractor is configured to identify a first set of feature points from a first image of a target present on a first surface in the relative surface acquired at a first moment by a sensor deployed on a second surface in the relative surface, and to identify a second set of feature points from a second image of the target acquired by the sensor at a second moment, wherein each feature point in the second set of feature points corresponds to a feature point in the first set of feature points. The transformation generation unit is configured to determine a first transformation based on the first set of feature points and the second set of feature points. The trailer pose estimator is configured to calculate a second transformation experienced by the trailer from the first moment to the second moment according to the first transformation, and to estimate the pose of the trailer at the second moment for autonomous driving of the vehicle based on the second transformation and the dimensions.

[0008] Other concepts relate to software for implementing the present teachings. A software product according to this concept includes at least one machine-readable non-transient medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters associated with the executable program code, and / or user-related information, requests, content, or other additional information.

[0009] In one example, a machine-readable non-transitory and tangible medium having data recorded thereon for pose estimation, wherein when the medium is read by a machine, the machine is caused to perform a series of steps. In a case where a vehicle and a trailer have corresponding relative surfaces facing each other, at least the dimensions of the trailer connected to the vehicle are obtained. A first set of feature points is identified from a first image of an object present on one of the relative surfaces acquired at a first moment by a sensor deployed on the other relative surface. A second set of feature points is identified from a second image of the object acquired by the sensor at a subsequent moment. Based on the first set of feature points and the second set of feature points, a first transformation is determined and used to calculate a second transformation experienced by the trailer from the first moment to the second moment. The pose of the trailer at the second moment is estimated based on the second transformation and the dimensions.

[0010] Additional advantages and novel features will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following text and the accompanying drawings, or may be learned by production or operation of the examples. The advantages of the present teachings may be realized and obtained by practicing or using the various aspects of the methods, means, and combinations set forth in the detailed examples discussed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The methods, systems and / or programs described herein are further described according to exemplary embodiments. These exemplary embodiments are described in detail with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, in which the same reference numerals represent similar structures throughout the several views of the drawings, and in which:

[0012] Figure 1A illustrates a view observed by an autonomous vehicle from sensors deployed thereon;

[0013] Figure 1B illustrates information obtained by processing data from sensors deployed on an autonomous vehicle according to an embodiment of the present teachings;

[0014] Figure 1C A truck is shown having sensors deployed thereon to facilitate autonomous driving in accordance with an embodiment of the present teachings;

[0015] Figure 1D illustrates a scenario requiring recalibration of sensors deployed on an autonomous truck according to an embodiment of the present teachings;

[0016] Figure 2A A truck with a trailer having associated key points estimated to facilitate the truck performing autonomous driving is shown according to an embodiment of the present teachings;

[0017] Figure 2B shows different scenarios of a truck with a trailer having associated key points estimated to facilitate the truck to perform autonomous driving according to an embodiment of the present teachings;

[0018] Figure 3A Different types of challenges in autonomous driving according to embodiments of the present teachings are described;

[0019] Figure 3B An example scenario of a convoy of trucks on a road is shown;

[0020] Figure 4A-4B shows an exemplary deployment of sensors around different parts of a truck / trailer configuration to facilitate autonomous driving in accordance with an embodiment of the present teachings;

[0021] Figure 5A Two trucks in a convoy are shown driving side-by-side on a road to enable instant recalibration of sensors to facilitate autonomous driving in accordance with an embodiment of the present teachings;

[0022] Figure 5B-Figure 5C A truck / trailer having special markings thereon for assisting the other party in instantaneous sensor calibration while both are in motion is shown in accordance with an embodiment of the present teachings;

[0023] Figure 6A-6D shows how two fleet members traveling in a forward / backward configuration may coordinate to facilitate instant sensor recalibration in accordance with an embodiment of the present teachings;

[0024] Figure 7A-7B An exemplary framework for on-the-fly collaborative sensor calibration according to an embodiment of the present teachings is depicted;

[0025] Figure 7C is a flow chart of an exemplary process for just-in-time collaborative sensor calibration according to an embodiment of the present teachings;

[0026] Fig. 8A depicts an exemplary high-level system diagram of a fleet management center according to an embodiment of the present teachings;

[0027] Figure 8B is a flow chart of an exemplary process of a fleet management center according to an embodiment of the present teachings;

[0028] Fig. 9A depicts an exemplary high-level system diagram of a calibration assistant selector according to an embodiment of the present teachings;

[0029] Fig. 9B illustrates an exemplary scenario of using landmarks as an aid to calibrating sensors while in motion according to an embodiment of the present teachings;

[0030] Fig. 9C is a flow chart of an exemplary process of a calibration assistant selector according to an embodiment of the present teachings;

[0031] Fig. 10A depicts an exemplary high-level system diagram of a host vehicle calibration controller according to an embodiment of the present teachings;

[0032] Fig. 10B is a flow chart of an exemplary process of calibrating a controller of a host vehicle according to an embodiment of the present teachings;

[0033] Fig.11A depicts an exemplary high-level system diagram of a calibration assistant according to an embodiment of the present teachings;

[0034] Fig. 11B is a flow chart of an exemplary process of a calibration assistant according to an embodiment of the present teachings;

[0035] Figure 12A-12C illustrates an exemplary configuration of a truck and trailer with exemplary key points representing the posture of the trailer relative to the truck according to an exemplary embodiment of the present teachings;

[0036] Figure 13A-13Bshows different arrangements of sensors and fiducial markers for a truck / trailer configuration for trailer pose estimation according to embodiments of the present teachings;

[0037] Figure 13C-Figure 13D shows different arrangements of using sensors to view different parts of a truck / trailer configuration for trailer pose estimation in accordance with embodiments of the present teachings;

[0038] Fig.14A depicts an exemplary high-level system diagram of a framework for instantaneous estimation of trailer pose according to an embodiment of the present teachings;

[0039] Fig. 14B is a flow chart of an exemplary process for a framework for instantaneous estimation of trailer pose according to an embodiment of the present teachings;

[0040] Fig.15 visually illustrates the concept of estimating trailer pose based on a reference target used in a truck / trailer configuration according to an embodiment of the present teachings;

[0041] Fig.16 is an illustrative diagram of an exemplary mobile device architecture that can be used to implement a dedicated system for implementing the present teachings in accordance with various embodiments; and

[0042] Fig.17 is an illustrative diagram of an exemplary computing device architecture that can be used to implement a special purpose system implementing the present teachings, according to various embodiments. DETAILED DESCRIPTION

[0043] In the following detailed description, many specific details are set forth by way of example in order to facilitate a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that the present teachings may be practiced without these details. In other cases, in order to avoid unnecessarily obscuring various aspects of the present teachings, well-known methods, processes, components and / or circuit systems have been described at a relatively high level without details.

[0044] The present teachings are intended to address the deficiencies in the quality and accuracy of sensing in the current state of the art in autonomous vehicles. As discussed herein, in order to ensure safe autonomous driving, it is essential that sensors reliably provide accurate information about the vehicle's surroundings. This is particularly true in Figure 1B An autonomous vehicle typically has sensors deployed on it to provide an observation of the vehicle's surroundings. Figure 1C. Truck 120 has various sensors mounted thereon, including, for example: stereo cameras 125 and 150 on the top and 170 and 190 at the bottom, LiDAR sensor 140, radar 180, and cameras on the sides of the vehicle (such as 130 and 160). To ensure the accuracy of the information, such sensors are calibrated in such a way that the information faithfully reflects the scene around the vehicle. For example, the calibration of the stereo camera can change when the pose of the camera changes. For example, if a pair of stereo cameras (e.g., cameras 125 and 150) are calibrated to estimate the depth of an observed object, then calibration is performed so that the left and right images acquired by each camera can be used to perform stereo processing to estimate the depth of objects in the scene.

[0045] When the pose of one or both of the paired cameras changes for any reason, the estimated depth of objects is no longer reliable, which can cause severe miscalculations in obstacle avoidance for autonomous driving. Figure 1D An example is shown in , where the pose of one of the stereo cameras has now changed to 125-1 (with Figure 1C Given the changed pose of camera 125-1, the correct image acquired by this camera can no longer be used to correctly estimate the depth of objects seen by both 125-1 and 150 using the transformation information obtained via the previous calibration. As such, the estimated depth of the object will not reflect the actual distance between the vehicle and the obstacle, which can cause serious problems. There are different reasons for the change in the pose of the sensor. For example, it can be due to vibrations of the vehicle that make the mounting parts loose or some parts of the sensor simply shift while driving. When this happens, a recalibration is required. Although it is possible to perform a recalibration by parking the car, this is not realistic in most scenarios.

[0046] When the calibrated sensors change while the vehicle is traveling, the accuracy of the information estimated from the sensed information will degrade, thereby affecting safety. Therefore, there is a need to recalibrate the sensors installed on the autonomous vehicle on the fly while the vehicle is moving. This corresponds to the external challenges of recalibrating the sensors on the fly. The present teachings disclose herein a method for recalibrating on the fly via collaboration. In some embodiments, this collaboration can be between two members of a fleet that are traveling together substantially. In some embodiments, this collaboration can be between an autonomous vehicle and another vehicle that needs to be recalibrated. In some embodiments, this collaboration can be via a fixed device or landmark (such as a billboard or a structure with known dimensions and / or a planar image visible to the autonomous vehicle).

[0047] In the case of a truck having a trailer attached thereto, such as Figure 2AAs shown in , there are additional challenges in autonomous driving. For example, a critical safety challenge is to ensure that the entire structure (truck and trailer) is within the driving lane. Figure 2A As seen in , in addition to the truck being in the desired lane, the trailers (in this example, there are two trailers) also need to be in the same lane. Figure 2A In the example, the trailer has corner points, including 240-1–240-9 in the visible area. To ensure that the trailer is in the same lane as the truck, the corner points must be in the lane. When the truck changes lanes, the control challenge is how to control the vehicle to ensure that all corner points of the connected trailer will be safely moved into the new lane. This is in Figure 2B . The corner points on the two trailers include 250-1, 250-2, ..., 250-9. Some of them may already be in the same lane as the truck, but some are still outside the range of the lane where the truck is located. Determining how to control the truck so that all the corner points can be safely and quickly moved into the same lane requires the vehicle (or truck) to know the position of all the corner points relative to the lane at different times. This corresponds to the inherent challenge of constantly knowing the arrangement of different parts of the truck-trailer configuration in order to ensure proper control of the autonomous truck.

[0048] Figure 3A Different challenges in autonomous driving are described, including extrinsic challenges for on-the-fly dynamic sensor recalibration and intrinsic challenges for monitoring the dynamic physical deployment of truck / trailer configurations to design appropriate vehicle control strategies in autonomous driving. In this teaching, in order to address the extrinsic challenge of on-the-fly recalibration, the Figure 3B – Fig. 11B Describes cooperative recalibration and landmark-based recalibration. To address the inherent challenges of estimating the deployment of different parts of a truck / trailer configuration, Figure 12A-Figure 15 Fiducial marker-based estimation and feature-based estimation methods are disclosed.

[0049] Trucks are commonly used to transport goods. To maximize efficiency, each truck can be equipped with one or more trailers. In addition, trucks are often Figure 3B As shown in FIG, the trucks 310, 320, 330, and 340 (and possibly more) are traveling as a convoy. Such a feature can be used to facilitate collaborative calibration. Each convoy member can have sensors deployed around it. Figure 4A-4B An exemplary deployment of sensors around different parts of a truck / trailer configuration to facilitate autonomous driving is shown in accordance with an embodiment of the present teachings. Figure 4A and as shown in FIG4B. Sensors or sensor racks (each of which can have more than one sensor mounted thereon) can be strategically affixed to different parts of each fleet member. For example, Figure 4AAs shown in , a truck / trailer 400 may have a sensor rack 410 mounted on the top of the truck, 420 mounted on the front of the truck, 430 mounted on the side of the truck, and 440 mounted on the side of the trailer. Figure 4B In FIG. 4 , a truck / trailer 400 has a sensor rack 450 mounted on one side of the truck and 460 mounted on the other side of the trailer.

[0050] When multiple convoy members are traveling in the same vicinity, if one convoy member (the host vehicle) needs to recalibrate a sensor installed at a known location, then when the cooperating convoy members have a specific target (sign) that facilitates recalibration, another convoy member may be required to collaborate with the host vehicle in real time to calibrate the sensor while both vehicles are traveling. Figure 5A Two trucks 510 and 520 in a convoy are shown driving side by side on a road to collaborate on instant sensor calibration in autonomous driving according to an embodiment of the present teachings. In this configuration, one convoy member (e.g., 520) may need to calibrate a sensor in a sensor rack 530 deployed on one side of the truck 520. Figure 5B A truck is shown having a target marking such as a chessboard 540 on one side of the truck 510 according to an embodiment of the present teachings can be used to assist the truck 520 to calibrate the sensors in 530 on the fly while the two are traveling side by side on a road. Since the chessboard has known structure and features, it can be used to calibrate a camera that can "see" it. Depending on which side of the truck the sensor that needs to be calibrated is located, the assisting truck can be instructed to maneuver to the appropriate side to allow the sensor that needs to be calibrated to "see" a special marking designed to facilitate calibration. In some embodiments, the target can be a pattern on the body of the truck or trailer with some known visual features. Figure 5C An example of a truck is shown with the word "Walmart" printed on the side of its body, next to the logo. Such target markings can also be used to facilitate collaborative calibration.

[0051] Figure 6A-6C 1 shows how two fleet members traveling in a forward / backward configuration can collaborate to facilitate instant sensor recalibration according to an embodiment of the present teachings. Fig. 6A In the example, there are two trucks 610 and 620 traveling in the same lane, one in front (610) and one in the back (620). If the camera mounted in front of truck 620 needs to be calibrated, then camera 620 can observe an object with certain known visual features on the back surface of truck 610 and use it for calibration. And vice versa. If the camera on the back of truck 610 needs to be calibrated, then the camera on 610 can capture an object with certain known visual features in front of truck 620 and use it to perform the required calibration.

[0052] As discussed herein, a target with known visual features may be part of a vehicle / trailer (e.g. Figure 5C ), or may be an external target placed on the body of the vehicle / trailer. Figure 6B An exemplary target 640 is illustrated for collaborative calibration in the back of a truck 630 , which may be used to assist in on-the-fly calibration. Figure 6B The object in is a part of a truck with known visual features of contrasting strips and feature points corresponding to the intersections of these strips. Figure 6C An external target 660 is shown mounted on the front side of the truck 650, which can be used to calibrate a camera located on the rear of another vehicle. In some embodiments, other native features of the truck 650 can also be used as targets for collaborative calibration. For example, Fig.6D As shown in , the grid 670 and its surroundings can be used as native targets for collaborative calibration. Any native part of the truck's exterior can be used as a target as long as there are some known and unique visual features.

[0053] As discussed herein, when a convoy is in motion, members of the same convoy can collaborate with each other to perform sensor calibration. In some embodiments, collaborative calibration can also be performed between the vehicle and another non-convoy vehicle, which is referred to as assisting the vehicle to perform calibration. This arrangement can be performed by either mutual communication between vehicles or via some central control. In the former case, between different vehicles, there can be a cross-communication platform, on which the vehicle that needs to assist in performing instant calibration can make a request with some details (e.g., its geographic location, the type of sensor to be calibrated, and the location of the sensor on the vehicle, etc.) and any other vehicle connected to the same platform can respond. In the latter case, the vehicle that needs to calibrate the sensor can make a request to the center to designate an auxiliary vehicle for collaborative calibration. Similarly, the request can include the geographic location of the vehicle and some details about the sensor to be calibrated (e.g., the type of sensor, the location of the sensor on the vehicle, etc.). Then, the center identifies the auxiliary vehicle, whether it is a member of the same convoy as the vehicle or a non-convoy member vehicle, and assigns it to assist the vehicle in collaborative calibration.

[0054] Figure 7A-7BAn exemplary framework 700 for instant collaborative sensor calibration according to an embodiment of the present teaching is depicted. In this framework, it includes various vehicles on a road 710, such as vehicles 710-a, ..., 710-b traveling together as a convoy, and individually traveling vehicles 710-c, ..., 710-d. The framework 700 also includes a host vehicle 740 that needs to calibrate at least one sensor installed thereon. Such vehicles including the host vehicle and other vehicles can be connected to a fleet management center 760 via a network 730. The host vehicle can be any vehicle traveling on the road. There may also be multiple vehicles that need assistance to perform collaborative calibration. The use of a host vehicle as a vehicle that needs calibration here is only for illustration and is not a limitation of the present teaching.

[0055] In some embodiments, vehicles in the framework or vehicles traveling in close proximity to the framework can communicate with each other in a peer-to-peer network setting (not shown). In this scenario, when the host vehicle 740 requests assistance from another vehicle in close proximity to perform a collaborative calibration, the other nearby vehicles can respond to the request and then communicate directly with the host vehicle to coordinate the necessary maneuvers to achieve instant calibration.

[0056] In some embodiments, coordination or at least a portion thereof is performed by a fleet management center 760, which manages vehicles traveling as a fleet and individual vehicles on various roads. The vehicle is equipped with a vehicle calibration controller 750. When a need for sensor calibration is detected, the vehicle calibration controller 750 sends a request to the fleet management center 760, which has information about, for example, its geographic location, the sensor(s) that need to be calibrated and their location on the vehicle, urgency, etc. Upon receiving the request, the fleet management center 760 can then search for appropriate candidates based on vehicle information stored in the database 770 about the vehicle's current location, planned route, task completion time, configuration (whether each vehicle is configured to perform collaborative calibration), etc. The auxiliary vehicle can be selected based on acceptance of the selected auxiliary vehicle. Fig. 7A In this illustrative embodiment of FIG, an exemplary selected auxiliary vehicle is 710-a, which includes a collaborative calibration assistant 720. As discussed herein, the selected auxiliary vehicle can be from the same fleet as the host vehicle or a non-fleet member operating independently nearby.

[0057] In some embodiments, when the host vehicle 740 and the selected auxiliary vehicle are connected (via the fleet management center 760), they can either communicate directly with each other to coordinate maneuvers via the network 730 or still via the fleet management center 760. In some cases, assigning a non-fleet member as an auxiliary vehicle to the host vehicle to perform collaborative calibration may require the auxiliary vehicle to detour from its originally planned route. This dynamic information can be recorded in the vehicle information database 770 so that the location of all vehicles managed by the fleet management center 760 is up to date and accurate.

[0058] Figure 7B yes Fig. 7A 700, and a collaborative calibration assistant 720 for selected auxiliary vehicles, which communicate via a network 730 in the framework 700 to coordinate collaborative calibration between different vehicles. Although an illustration is provided with respect to a truck, other types of vehicles may also use various aspects of the present teachings disclosed herein to perform instant collaborative calibration while the vehicle is in motion. The fleet management center 760 may be a centralized location, or may be distributed across different geographic locations to form a cloud. Each such distributed fleet management center in the cloud may be configured to be responsible for managing vehicles traveling within a designated geo-fenced area. When a request is made from a host vehicle traveling close to the boundary of a geo-fenced area, the fleet management center responsible for the adjacent area of ​​the boundary may communicate to select an appropriate auxiliary vehicle, which may be from outside the geographic area from which the request was initiated.

[0059] In some embodiments, the network 730 may correspond to a single network or a combination of different networks. For example, the network 730 may be a local area network ("LAN"), a wide area network ("WAN"), a public network, a proprietary network, a public switched telephone network ("PSTN"), the Internet, an intranet, a Bluetooth network, a wireless network, a virtual network, and / or any combination thereof. In one embodiment, the network 730 may also include various network access points. For example, the framework 700 may include wired or wireless access points, such as but not limited to base stations or Internet exchange points 730-a, ..., 730-b. Base stations 730-a and 730-b may facilitate, for example, communications to / from the vehicle 740, the fleet management center 760, and / or other vehicles 710 having one or more other components in the networking framework 700 across different types of networks.

[0060] Figure 7Cis a flow chart of an exemplary process for instant collaborative sensor calibration according to an embodiment of the present teachings. The host vehicle determines at 705 whether the sensor needs to be recalibrated during its travel. If so, at 715 the host vehicle sends a request for a calibration assistant to help with collaborative calibration and related information (e.g., its geographic location, information about the sensor to be recalibrated, etc.) to the fleet management center 760. When the fleet management center 760 receives the request, it identifies at 725 the appropriate auxiliary vehicle to collaborate with the host vehicle to achieve instant calibration, and notifies the selected auxiliary vehicle at 735. Once the host vehicle and the auxiliary vehicle are connected, they collaborate at 745 to coordinate calibration.

[0061] Fig. 8A An exemplary high-level system diagram of a fleet management center 760 according to an embodiment of the present teachings is depicted. In this illustrated embodiment, the fleet management center 760 can be used to schedule vehicle activities (e.g., driving schedules for fleets and individual trucks) and their calibration assistance needs. To this end, the fleet management center 760 receives different types of information and makes decisions accordingly. For example, it can receive profile information about each vehicle, transportation requirements from operators, and schedule fleets and / or individual trucks to fulfill transportation requirements. Depending on the activities being scheduled, trucks can be dispatched individually or as a fleet to roads with specified destinations and schedules. While driving, the vehicles continue to send relevant information to the fleet management center 760, including geographic location, destination, task schedule, surrounding information (e.g., weather, etc.), and dynamically occurring problems. Maintaining such dynamic information enables the fleet management center 760 to track the current status of the vehicles, so that when a need for recalibration arises from any vehicle, the fleet management center 760 can then select a collaborative calibration assistant based on the most recently updated information.

[0062] The fleet management center 760 includes a dynamic vehicle information processor 800 , a calibration assistant selector 820 , an assistant route scheduling unit 830 , a vehicle communication unit 840 , a central operator interface unit 850 , and a vehicle task / activity scheduler 860 . Figure 8B805 is a flow chart of an exemplary process of a fleet management center 760 according to an embodiment of the present teachings. When the central operator interface unit 850 receives from an operator, it forwards the received request to the fleet task scheduler 860. Based on the vehicle profile stored in 880, the vehicle task / activity scheduler 860 schedules the fleet / vehicle activities based on the request according to the vehicle profile information stored in the storage device 880 at 805. The profile of the vehicle may include information about features related to the transportation capabilities of the vehicle, the sensor configuration on the vehicle (e.g., how many trailers can be connected, which sensors are deployed at which locations on the vehicle, what targets exist on which facets of the vehicle, etc.). Such information can be used to schedule activities based on demand and identify collaborative calibration assistants.

[0063] The vehicle mission / activity scheduler 860 may then generate one or more vehicle mission schedules and store such schedules in a log 870, which specifies, for example, which vehicle is dispatched to perform which mission, with which other vehicles, toward which destination, and the current route planned, etc. While performing the scheduled activities, the dispatched vehicle may continue to send dynamic information to the fleet management center 760 so that it can keep track of the dynamics associated with the vehicle at 815. While managing the vehicles, the fleet management center 760 may periodically check at 825 to see if any vehicle on the road has sent a request for collaborative calibration assistance. When there is no request for collaborative calibration, the fleet management center 760 continues to either dispatch other vehicles for different activities at 805, or update the record of the vehicle's dynamic information stored in the log 810.

[0064] As discussed herein, a vehicle on the road can monitor the performance of its sensors and detect whether any deployed sensors need to be recalibrated. In this illustrated embodiment, when such a need arises, the vehicle sends a request to the fleet management center 760. Afterwards, an appropriate assistant will be selected to coordinate with the vehicle to perform an immediate collaborative calibration. Upon receiving such a request, the calibration assistant selector 820 selects an appropriate collaborative calibration assistant at 835 based on different types of information (e.g., the current geographic location of the vehicle, the current geographic location of other vehicles (including other fleet members or other vehicles moving alone)), whether each candidate vehicle has (one or more) targets on it that can be used to calibrate the problematic sensor on the vehicle, whether each candidate vehicle can be called to assist in the requested collaborative calibration (in view of its own predetermined task), whether the route taken by the candidate vehicle is consistent with the direction of the vehicle, etc. For example, if the vehicle requests calibration of two cameras, one in the front and one on the left, the selected auxiliary vehicle needs to have targets on its right side and its rear to support calibration of the front and left cameras.

[0065] Once a candidate auxiliary vehicle is selected, the calibration assistant selector 820 can call the assistant route scheduling unit 830 at 845 based on the request from the host vehicle to design a collaborative calibration route plan. For example, if the host vehicle specifies that the front sensors and the left sensors need to be recalibrated, then a possible calibration route plan may include having the auxiliary vehicle first approach the host vehicle on its left side so that the sensors on the left side of the host vehicle can be recalibrated based on the target visible on the right side of the auxiliary vehicle. Then, the route can indicate that the auxiliary vehicle then drives to the front of the host vehicle so that the sensors at the front of the host vehicle can be calibrated by relying on the target visible at the rear of the auxiliary vehicle. This designed calibration route plan can then be sent from the auxiliary route scheduling unit 830 to the vehicle communication unit 840, which then sends the collaborative calibration instructions together with, for example, additional information (such as the location of the host vehicle, the means of communicating with the host vehicle, etc.) to the selected auxiliary vehicle at 855. In this way, the host vehicle that issues the request and the selected auxiliary vehicle can be connected to interact directly in order to coordinate to perform the calibration plan.

[0066] Fig. 9A An exemplary high-level system diagram of a calibration assistant selector 820 according to an embodiment of the present teachings is depicted. In this illustrated embodiment, the calibration assistant selector 820 includes a vehicle assistant selector 900, a landmark assistant selector 920, a helper package generator 940, and a helper package sender 950. As discussed herein, in a collaborative calibration operation, the host vehicle requests that another vehicle (whether from the same fleet or from elsewhere) coordinate with it to achieve this goal. In some cases, a vehicle assistant may not be found. In this case, the calibration assistant selector 820 may instead select a landmark to assist the host vehicle in achieving collaborative calibration. Such a landmark may be located along a highway on the route of the host vehicle, for example, and may provide a visible target when the host vehicle passes through it or is near it. Such a landmark may have a target with known visual features thereon that may be communicated to the host vehicle prior to calibration. This is particularly useful in the case of vehicle calibration. Fig. 9B It is shown in Fig. 9BAn exemplary scenario of using landmarks as an assistant for calibrating sensors while in motion according to an embodiment of the present teaching is shown. As shown, the vehicle 960 is traveling on a road 970, and the camera at the front of it may need calibration. The calibration assistant selector 820 may not be able to find a vehicle assistant that cooperates with the vehicle 960 for instant calibration. As an alternative, the calibration assistant selector 820 identifies landmarks 980, i.e., billboards along the road 970, to assist the vehicle in calibration. Such a selected landmark assistant can convey its position, orientation, etc. to the vehicle before the vehicle arrives at the location where the landmark is located. Information about what the vehicle expects to "see" on the billboard and the visual features of the target on the billboard can also be conveyed to the vehicle. The vehicle can also be instructed to travel on a specific lane in order to observe the expected visual features in a precise manner. The vehicle can also be notified of the geographical range in which the vehicle can control its sensor to be calibrated to capture the image of the target on the landmark, so that known features can be extracted from the image and used to calibrate the sensor. Landmarks are not limited to billboards. It can be any structure (e.g., building, structure, etc.) with known pose, orientation, and other target features that may be visual or non-visual to facilitate calibration. For example, to calibrate a camera sensor, landmarks with known visual features can be used to assist the vehicle in performing collaborative calibration on the fly. To calibrate different types of sensors, such as sensors that sense depth, such as LiDAR, landmarks with dimensional information associated with known depth can be used for collaborative calibration. For example, to calibrate a pair of stereo cameras, landmarks with known geographic coordinate information can be used for calibration. In this way, landmarks can be used as assistants to the vehicle for calibrating sensors on the fly.

[0067] Fig. 9Cis a flowchart of an exemplary process of the calibration assistant selector 820 according to an embodiment of the present teaching. In operation, after receiving a request to identify a collaborative calibration assistant at 905, the vehicle assistant selector 900 searches for an appropriate vehicle as a collaborative calibration assistant at 915, and such a search may be based on different types of information. For example, geographic proximity may be a requirement. Whether the vehicle assistant candidate has the required target on it to allow the requested calibration may be another requirement. In some embodiments, if the vehicle is associated with an emergency mission, it may not be used as an assistant even if it otherwise meets all requirements. The vehicle assistant selected in this way may or may not be a member of the fleet. If such a vehicle member is found, then the vehicle assistant can be used for collaborative calibration. If not found, then the vehicle assistant selector 900 can then call the landmark assistant selector 920 to identify landmarks at 935 to assist in collaborative calibration. Each identified landmark assistant will be located along the route that the vehicle is to pass. In some embodiments, different priorities can be used, that is, landmark assistants can be preferred, and the selector 820 identifies the vehicle assistant only when no landmarks are available.

[0068] For each sensor to be recalibrated, the landmark helper selector 920 can identify different landmarks. For example, while calibrating a camera requires visual information with known visual characteristics, when the sensor to be calibrated is a LiDAR sensor, visual features on the helper are not required, but the helper landmarks need to have spatial characteristics (such as depth distribution when perceived from a certain position). Each landmark helper identified in this way can be associated with a specific sensor to be recalibrated, and the landmark helper selector 920 can classify them accordingly at 945. In the case where either a vehicle helper or a landmark helper is selected, a calibration helper package can be generated by the helper package generator 940 at 955 and then sent out by the helper package sender 950 at 965. Such a package can specify how to use each helper, whether it is a vehicle or a landmark, to assist in performing collaborative calibration with respect to which sensor on the vehicle.

[0069] Fig. 10A An exemplary high-level system diagram of a host vehicle calibration controller 750 according to an embodiment of the present teachings is depicted. The host vehicle is a vehicle that needs to be calibrated on the fly via collaborative calibration. Each vehicle can be equipped with different operating software modules, such as a module that controls the calibration needs of the host vehicle (host vehicle calibration controller 750) and a module that can be used to assist the host vehicle in the need to calibrate on the fly (collaborative calibration assistant 720), and these modules can be activated to perform the intended functions when the corresponding needs arise. In view of this, in different situations, each vehicle can be a host vehicle or a collaborative calibration assistant.

[0070] In this illustrated embodiment, the host vehicle calibration controller 750 includes a own geographic (GEO) location determiner 1000, a sensor calibration determiner 1020, a calibration schedule determiner 1030, a fleet center communication unit 1040, a peer (P2P) vehicle communication unit 1050 and a calibration coordination controller 1070. Fig. 10B 1 is a flow chart of an exemplary process of the vehicle calibration controller 750 according to an embodiment of the present teaching. In operation, the own geographic location determiner 1000 in the vehicle calibration controller 750 continuously monitors its own geographic location at 1005 and records its moving geographic location in the current task execution log 1010. In order to promote collaborative calibration, the geographic location tracked in this way is also reported to the fleet management center 760. In addition to tracking its own location, the sensor calibration determiner 1020 of the vehicle calibration controller 760 can also monitor the information obtained by different sensors deployed on the vehicle to determine which sensor may require recalibration at 1015. In some embodiments, this determination can be made based on the quality of the received sensor data. Other means can also be used to detect whether the sensor needs to be recalibrated. If it is determined at 1025 that no sensor needs to be recalibrated, then the vehicle calibration controller 760 continues to monitor its own dynamic location at 1005 and / or determines whether there is an upcoming need to recalibrate the sensor.

[0071] When it is determined at 1025 that a need to calibrate a sensor is detected, a calibration schedule determiner 1030 may be called to determine at 1035, for example, how urgently the calibration should be performed, thereby generating a schedule for the required calibration, and then storing the schedule so generated at 1060. Once the schedule(s) for the required calibration are generated, the fleet center communication unit 1040 sends a request to the fleet management center 760 to perform a collaborative calibration at 1045. As discussed herein, such a request may include additional relevant information necessary to fulfill the request, such as the geographic location of the host vehicle, the destination to which the host vehicle is heading, the limited task at hand for the host vehicle (e.g., when to deliver the goods to the destination), the sensors to be recalibrated, the physical location and characteristics of such sensors, the nature of the identified problem (e.g., the camera is not in its installed location), and optionally a schedule for recalibration (e.g., because the camera is used to estimate the depth of obstacles, the ability of the host vehicle to operate in an autonomous driving mode without it would be adversely affected).

[0072] After receiving the request for collaborative calibration, the fleet management center 760 can then search for a collaborative calibration assistant suitable for the host vehicle, and, for example, inform the fleet center communication unit 1040 of information about the selected collaborative calibration assistant (such as its current geographic location, estimated arrival time, contact information, etc.). As the fleet center communication unit 1040 receives a response from the fleet management center 760 at 1055, which has information about the selected assistant, the P2P vehicle communication unit 1050 can then communicate with the selected auxiliary vehicle at 1065 to exchange relevant information between the host vehicle and the auxiliary vehicle, such as the current location and contact information of each vehicle, the driving speed of each vehicle, the distance between the two vehicles, and the precise coordinates of each vehicle. The calibration coordination controller 1070 can then use this information to coordinate activities between the host vehicle and the auxiliary vehicle at 1075 to collaboratively and instantly achieve calibration.

[0073] The coordination between the host vehicle and the auxiliary vehicle may include when to calibrate which sensor and at which position (e.g., front, rear, left, or right) the auxiliary vehicle needs to be in order to assist the host vehicle in calibrating a specific sensor, etc. The coordination may be performed by the calibration coordination controller 1070 based on the calibration schedule generated by the calibration schedule determiner 1030. As a result of the coordination, the calibration coordination controller 1070 may generate instructions for the auxiliary vehicle at 1085 and send them to the P2P vehicle communication unit 1050 so that such information may be sent to the auxiliary vehicle. In addition, the calibration coordination controller 1070 may also generate control signals for the host vehicle control mechanism (not shown) so that the host vehicle can be driven autonomously in a manner consistent with the coordinated activities with respect to the auxiliary vehicle. When the collaborative calibration is completed according to the calibration schedule in 1060, the P2P vehicle communication unit 1050 notifies the auxiliary vehicle and the fleet management center 760 at 1095 (via the fleet center communication unit 1040).

[0074] As discussed herein, in some cases, when no assist vehicle is found and / or when landmarks are used to assist the host vehicle in performing calibration, the fleet management center may send calibration assistant information to indicate landmarks that are reachable by the host vehicle (e.g., along its route of travel). In such embodiments, the calibration assistant information may provide a description of the nature of the landmark (e.g., billboard, building, etc.), the geographic pose (position / orientation) of the landmark, the expected 2D or 3D characteristics of the landmark (which may be used as ground truth) based on perspectives and instructions (such as which lane the host vehicle may need to be in when approaching the landmark and the geographic range that the host vehicle should observe in order to activate its sensors to capture landmark information). In such embodiments, the functionality of the calibration coordination controller 1070 may be different from the functionality of collaborative calibration via assist vehicles. For example, when the fleet center communication unit 1040 receives information related to a selected landmark (see Fig. 10A ), the received information can be provided to the calibration coordination controller 1070, which can then generate appropriate control signals to control the vehicle to approach the identified landmark (for example, enter a lane on a road), control the sensor to be calibrated to collect the expected information related to the landmark (for example, control the camera to capture an image of the landmark in a certain range relative to the landmark), analyze the captured information (for example, analyze the depth information sensed from the sensor and compare it with the ground truth depth information from the center), and then calibrate the sensor accordingly based on the difference (or lack of difference) between the sensed information and the ground truth information received from the center.

[0075] Fig.11A An exemplary high-level system diagram of a collaborative calibration assistant 720 according to an embodiment of the present teachings is depicted. As discussed herein, a collaborative calibration assistant (i.e., an auxiliary vehicle) may be selected by a fleet management center 760 or in response to a posted request from a host vehicle requesting collaborative calibration assistance. Once an auxiliary vehicle is identified, it is dispatched to travel to the location of the host vehicle and maneuver around the host vehicle to assist the host vehicle in calibrating one or more sensors. To this end, different types of information are communicated to the auxiliary vehicle, such as the geographic location of the host vehicle, the number of sensors that need to be calibrated and their relative positions relative to the host vehicle, and optionally information about a schedule or order in which different sensors on the host vehicle are calibrated via collaboration.

[0076] In this illustrated embodiment, the collaborative calibration assistant 720 includes a native geolocation tracker 1100 , a central communication unit 1120 , an auxiliary route determiner 1130 , a vehicle communication unit 1150 , a calibration coordination controller 1160 , and an auxiliary route controller 1170 . Fig. 11B is a flow chart of an exemplary process of the collaborative calibration assistant 720 according to an embodiment of the present teachings. To participate in the collaborative calibration framework 700, the native geolocation tracker 1100 monitors the dynamic geolocation of the assisting vehicle at 1105 and reports such tracked location to the fleet management center 760 via the central communication unit 1120 at 1115. Such tracking and reporting activities occur continuously so that the fleet management center 760 is aware of the current location of all vehicles it manages.

[0077] In some embodiments, the auxiliary vehicle is selected by the fleet management center 760. In this case, the central communication unit 1120 determines at 1125 that an auxiliary request is received from the fleet management center 760. In some embodiments, when the auxiliary vehicle responds to a request (not shown) issued by the host vehicle for collaborative calibration, the auxiliary vehicle is selected by the host vehicle, for example. In either case, the auxiliary vehicle receives relevant information for performing activities to assist in collaborative calibration along with the request. After receiving the request with relevant information, the auxiliary route determiner 1130 analyzes the request at 1135 to determine the location of the host vehicle, and accordingly determines at 1145 the route to be taken to travel to the host vehicle. The auxiliary route determiner 1130 can also analyze relevant information, such as the sensors scheduled to be calibrated and their physical locations relative to the host vehicle (e.g., on the left, in front, or on the right) and derive a certain associated schedule or plan at 1155 (e.g., first approach the rear of the host vehicle so that the rear sensor can be calibrated, then move to the left side of the host vehicle so that the sensor on the left side of the host vehicle can be calibrated, etc.). This auxiliary schedule can be stored in 1140.

[0078] Using the auxiliary route and schedule thus generated, the auxiliary vehicle communicates with the host vehicle via the vehicle communication unit 1150 at 1165 to coordinate the collaboration. For example, the auxiliary vehicle can inform the host vehicle of the route it will take to get there, the estimated arrival time, the resulting auxiliary schedule (e.g., the order of locations the auxiliary vehicle will be in and the corresponding sensors to be calibrated, etc.). Then, at 1175, the calibration coordination controller 1160 coordinates with the host vehicle throughout the process according to the auxiliary schedule along the planned route, one calibration at a time, until it is determined at 1185 that all sensors that need to be recalibrated have been calibrated. During the calibration process, the auxiliary route controller 1170 generates the control signals required to manipulate the auxiliary vehicle in the planned manner. When the calibration is completed, the auxiliary vehicle can then return to the task scheduled before the collaborative calibration at 1195.

[0079] Using the collaborative calibration framework discussed herein, when a calibration assistant can be identified, whether it is another moving vehicle or a landmark, the moving vehicle can calibrate its sensors instantly. This framework can be particularly effective when the vehicle is a member of a convoy, so that another nearby convoy member can be called to assist in calibration. In different situations, alternatives to using landmarks to assist in instant calibration can be called. In the above discussion, when an auxiliary vehicle cannot be found, a landmark assistant can be used. In some embodiments, for various considerations, a landmark assistant can be used instead of an auxiliary vehicle, for example, when road conditions make it more risky to use an auxiliary vehicle, a landmark assistant can be used instead. Other conditions may also have a deterministic effect on the use of a road sign assistant or an auxiliary vehicle for collaborative calibration. For example, the time of day (day or night), the level of congestion on the road where the vehicle is observed, the clarity of the landmarks encountered within the specified time range to complete the calibration, etc. The different details and aspects to be considered when implementing this teaching are within the scope of this teaching.

[0080] As discussed herein, in addition to the extrinsic need to recalibrate sensors via collaboration, there are also situations where intrinsic needs related to autonomous driving arise. Figure 2A-2B An example is illustrated in , where a truck with (one or more) trailers attached needs to monitor the pose of each trailer in order to ensure that the entire truck / trailer configuration is in place (e.g., within the same lane). Figure 12A-12C It further illustrates the different configurations between a truck and its trailer, and the need to understand trailer posture in autonomous driving to ensure proper control of the trailer in relation to the truck for safety. Fig. 12A , a single trailer 1220 is attached to a truck 1210 via a rigid connector 1215. In this exemplary configuration, the truck 1210 can have different feature points that characterize its posture, for example, vertex points (vertices) 1210-1, 1210-2, 1210-3, and 1210-4. The posture of the trailer 1220 can also be characterized using its corresponding feature points. For example, in order to control autonomous driving to ensure that the trailer is in the same lane as the truck 1210, it is sufficient to know the position of the center point on the rear surface (CRS) of the trailer 1220, which can be determined based on the center point on the front surface (CFS) of the trailer 1220, provided that the dimensions of the trailer (e.g., length L) are known. In some embodiments, other feature points can also be used to characterize the posture of the trailer 1220, for example, four center points on each vertical surface of the trailer 1220 can be used to characterize its posture.

[0081] When the truck 1210 and trailer 1220 are in a known configuration, a camera mounted (either on the rear surface of the truck 1210 or on the front surface of the trailer 1220) can be calibrated. Fig. 12A1210-3 and 1210-2 and 1210-4 can form two parallel lines substantially parallel to the connector 1215 and the line formed by the corner points on the trailer 1220. That is, the line 1220-1-1220-3 and the line 1220-2-1220-4 are substantially parallel lines 1210-1 and 1210-3 and 1210-2 and 1210-4, respectively. The object observed in this known configuration and the features extracted therefrom can form a reference object.

[0082] Fig. 12B A perspective view of trailer 1220 is provided. In addition to top corner points 1220-1, 1220-2, 1220-3, and 1220-4, bottom corner points 1220-5, 1220-6, 1220-7, and 1220-8 can also be used to characterize the pose of trailer 1220. Assuming trailer 1220 is a rigid rectangular prism, the top corner point (e.g., 1220-4) and the corresponding bottom corner point (e.g., 1220-8) are projected to the same point 1230 on the ground. Similarly, other pairs of top and bottom corner points can also be projected to the same point on the ground (1240, 1250, and 1260). To ensure that trailer 1220 is within the lane (formed by two lane markings 1270 and 1280), the autonomous truck can determine the position of the trailer by estimating the coordinates of designated feature points on trailer 1220. For example, CFS and CRS can be estimated. As another example, corner points (at least some on the top or bottom) may also be determined based on the known dimensions of the trailer 1220. Thus, the autonomous truck needs to continuously estimate such feature points in real time to provide a pose representation for the truck.

[0083] As discussed herein, when the truck 1210 and trailer 1220 are aligned, as shown in FIG. Fig. 12A As shown in , connector 1215 is parallel to the sides of truck 1210 and trailer 1220. The camera can be calibrated in this configuration and a reference feature point set is established by observing targets on the opposite surfaces and identifying multiple feature points in the target. This reference feature point set detected during calibration can be used to estimate the transformation between reference target feature points and future target feature points based on planar homography. This transformation (rotation and translation) can then be used to estimate the pose of trailer 1220.

[0084] Denote I_0 as the initially acquired target image (on the front surface of the trailer or the rear surface of the truck), which can be during calibration or at the beginning of estimating the trailer pose or during initialization or when the system is just started when the truck-trailer is in a known safe state. Denote image I_t as the subsequent image of the target at time t, t>0. The task is to estimate the pose difference between I_0 and I_n. That is, the trailer pose is estimated based on the difference between the target at time t and time 0. Therefore, image I_0 is used as a reference. In order to perform homography, a set of 2D feature points is identified from I_0. Similarly, these 2D feature points are also identified from the target image I_t at time t. In some embodiments, a minimum number of feature points is required, such as at least four feature points. In general, in order to overcome the effects of noise, more feature points can be used in the homography.

[0085] use Represents the coordinates of the feature points. Assuming there are N feature points from I_0 and I_t respectively, the transformation matrix can be calculated so that

[0086]

[0087] To determine H, any known method may be used to obtain H. In some embodiments, a direct linear transform (DLT) based solution may be used. Rearranging the above equation, with Ah=0, where A is a 2N x 9 matrix and h is a 9x1 vector, is shown below.

[0088]

[0089]

[0090] in

[0091]

[0092]

[0093] The vector h can be solved using different techniques, such as Singular Value Decomposition (SVD) or Sum of Squared Errors (SSE). The 1x9 vector h can be reshaped into H, where H is a 3x3 matrix:

[0094]

[0095] In order to maintain 8 degrees of freedom, H 22 Set to zero. The H matrix can be normalized by dividing each element in H by the L2 norm of the H matrix.

[0096] The H thus obtained can be decomposed into different degrees of freedom, such as rotation R and translation t(R, t). The R and t thus obtained represent the motion of the camera relative to the trailer surface. Such motion parameters of the camera can then be used to convert them into rotation (R') and translation (t') of the trailer via the following equations:

[0097] R'=R T (5)

[0098] t'=-R T .t (6)

[0099] The pose of the trailer can then be calculated using this determined motion parameters for the trailer according to the following equation. Assuming the trailer is a rigid body, the rotation of the front face relative to the camera (R T ) is the same as the rotation of the rear of the trailer. The translation of the rear of the trailer can then be calculated accordingly. For example, about a chosen point, for example, the center of the front of the trailer is Tf = [T X ,T Y , T Z ], if the length of the trailer is L, then the center point behind is Tr = [T X ,T Y ,T Z +L]. Then, the posture of this selected center point after the movement is as follows:

[0100] Tf'=[R'|t']*Tf (7)

[0101] Tr'=[R'|t']*Tr (8)

[0102] In some embodiments, the front and / or rear center points may be used to represent the pose of the trailer. In various situations, this may be sufficient for vehicle control purposes to ensure that the trailer and truck are in the same lane. In some situations, more points may be needed for greater accuracy. For example, if Figure 12A-12C The top corner points shown in can be estimated to represent the pose of trailer 1220. To do this, for example, the top corner points on the front of trailer 1220 can be first determined based on, for example, the known dimensions of the trailer. Each such top corner point from the front can be represented as T f Then, we can derive the transformation based on L before and after and with T f The corresponding top corner point is denoted as T r All of these rear top corner points are then used to apply a rotation R' and a translation t' to arrive at the transformed top corner point to obtain T representing the trailer pose. r '.

[0103] The estimation of the transformation can be applied when the truck and trailer are in motion. This is particularly important when, for example, the truck 1210 changes from one lane (within 1270 and 1280) to an adjacent lane (within 1280 and 1290), as shown in FIG. Fig. 12C . In this case, the truck 1210 and trailer 1220 are no longer aligned or transformed. This transformation causes the target observed by the camera to be visually different, i.e., due to the transformation, the feature points now observable from the target image are moved or deformed from their initial positions in image I_0 to positions in I_n. As discussed herein, this change in the detected positions of corresponding feature points is used to calculate the transformation matrix H, which can then be decomposed to obtain R and t used to calculate R' and t' to determine the pose of the trailer.

[0104] To estimate the transformation, a camera may be placed on one of the opposing surfaces using the opposing surfaces of the truck 1210 and trailer 1220 to observe visual features (or changes thereto) of an object or fiducial marker present on the other surface. The observed visual features may then be used to compare with stored reference visual features of the fiducial marker, and the difference may be used to calculate the transformation. Figure 13A-13B Different arrangements of sensors and fiducial markers for a truck / trailer configuration for trailer pose estimation according to embodiments of the present teachings are shown. In an exemplary configuration as shown in FIG. 13A , a camera 1310 may be mounted on the rear of the truck 1300, which has a field of view relative to the front of the trailer 1320. On the front surface of the trailer, a fiducial marker 1330 may be provided with certain known visual features. For example, on the fiducial marker 1330, there are visible high-contrast feature points, such as corner points, 90-degree concave points, 90-degree convex feature points, symmetric features, etc. Fig. 13B The reverse configuration between the camera and the fiducial marker is shown, i.e., the camera 1310 is now mounted on the front surface of the trailer, looking at the fiducial marker 1330 on the rear surface of the truck.

[0105] In some embodiments, visual features characterizing the surface of either the truck (the rear of the truck) or the trailer (the front of the trailer) may be used directly, rather than relying on visual features of objects placed on opposing surfaces. Figure 13C-Figure 13D Different configurations for trailer pose estimation using a portion of the surface of a truck or trailer are shown in accordance with embodiments of the present teachings. Figure 6B As shown in , the back of the truck can provide various visual cues that can be used to determine the transformation. Figure 6B The visual cues in include high contrast distinct stripes and the junction of these stripes. Other parts of the rear of the truck can also be used as visual cues to be detected and used to estimate the transformation. Such visual targets can also exist in the front surface of the trailer, such as Fig. 13CIn some embodiments, such a target may preferably have some structural features with different feature points having some known spatial relationships that can be used for pose estimation purposes. Similarly, in some alternative embodiments, when the rear of the truck 1300 has some built-in visual targets with known visual features, the camera 1310 can be mounted on the front surface of the trailer so that the observed actual features from the visual targets on the rear of the truck 1300 can be used to estimate the pose. This is Fig.13D When the camera is mounted on the front surface of the trailer, the orientation of the camera can be determined in different ways. For example, an inertial measurement unit can be used in conjunction with the camera to report the pose of the camera relative to the surface or edge points of the trailer. In some embodiments, if the camera's mounting bracket is rigid, the mounting pose (position and orientation) can be used.

[0106] As discussed herein, in order to estimate the transformation H using the visual features of the target, a reference feature may be required. Using this reference or ground truth feature, the observed corresponding features can then be compared to determine, for example, the transformation to be applied so that the actual observed visual features match the known reference or ground truth features, and vice versa. As discussed herein, once the transformation is determined, the key points (e.g., center points or corner points 1220-1, ..., 1220-8) representing the posture of the trailer can be calculated based on the known dimensions of the trailer. In this way, while both the truck and the trailer are moving, the truck 1210 can continuously monitor the posture of the trailer connected thereto. When the truck and the trailer are aligned in a straight line, the ground truth or reference feature can come from the initial setting. In this case, the posture of the trailer is determined based on the transformation from the initial position when the trailer is aligned with the truck. In some cases, the reference feature can come from the previous position of the truck / trailer, so that compared with its previous posture, the posture estimated for the trailer can correspond to the change of its posture via the transformation as discussed herein. In such a case, the reference features used to estimate the next pose of the trailer may be periodically updated to provide a reference at a previous point in time.

[0107] Fig.14AAn exemplary high-level system diagram of a framework 1400 for instant estimation of trailer pose according to an embodiment of the present teaching is depicted. In this illustrated embodiment, the framework 1400 includes a configuration unit 1402, an intrinsic calibration unit 1410, a transformation generation unit 1420, a target image acquisition unit 1430, a target image feature extractor 1440, and a trailer pose estimator 1450. The operation of the framework 1400 may include two stages. One stage corresponds to an offline stage, in which a camera mounted on a surface (either the rear surface of a truck or the front surface of a trailer) is used to capture visual features of an object on an opposing surface (or an object posted on an opposing surface or a portion of an opposing surface). While the camera is calibrated in this stage, target features are generated and used to calculate calibration parameters. Target features may include feature points observable in the target and / or spatial relationships between feature points. The configuration unit 1402 and the intrinsic calibration unit 1410 are provided to facilitate calibration in this first stage.

[0108] Another stage corresponds to an online process for estimating the pose of the trailer when both the truck and trailer are in motion. This online process of estimating the pose of the trailer at each specific moment is intended to facilitate autonomous driving. To this end, a calibrated camera acquires real-time images of objects on the opposing surface, and such images are used to detect feature points. In some cases, for example, when the truck is turning, such as Fig. 12C As shown in , the feature points detected from the real-time acquired image may have been deformed to have different image coordinates, and the spatial relationship between these feature points will also be different from the spatial relationship between the feature points detected during calibration. As discussed herein, using the feature points of the acquired image and the reference feature points, the transformation matrix H or h can be in online processing. A target image acquisition unit 1430, a target image feature extractor 1440, a transformation generation unit 1420, and a trailer pose estimator 1450 are provided for processing during online operation.

[0109] Fig. 14B 1400 is a flow chart of an exemplary process for a framework 1400 for instant estimation of trailer pose according to an embodiment of the present teachings. In this illustrated embodiment, to enable offline operation, an operator specifies various types of information required for calibration, such as parameters related to the truck / trailer configuration, dimensional information, etc., via an interface of a configuration unit 1402, for example, at 1405. This may include the type of truck and trailer involved, the width / length / height of the trailer, the location and length of the connector 1215, etc. The information related to the truck / trailer configuration may be stored in a truck / trailer dimensional configuration log 1412 and may be used by an intrinsic calibration unit 1410 to perform calibration of the intrinsic camera at 1415.

[0110] Calibration can be performed under certain specified conditions, for example, the truck and trailer can be aligned. During calibration, various visual feature points can be extracted and the spatial relationships between these feature points can be identified. These feature points and the spatial relationships between them represent what the intrinsic camera should see at the target at the reference point when the truck / trailer is configured in a certain way. In some embodiments, the calibration parameters and reference features can be stored at 1425 in the calibration parameter database 1422 and the reference feature configuration log 1432, respectively.

[0111] In the case where the camera is calibrated and the calibration results (calibration parameters) and reference features are stored, when the truck and the attached trailer are moving, the information stored in this way can be used to estimate the posture of the trailer in real time, so as to facilitate vehicle control to ensure the normal behavior of the truck and the trailer, for example, driving in the same lane. To this end, a schedule for the frequency of monitoring the posture of the trailer can be set at 1435 by the operator via the configuration unit 1400, for example. The schedule set in this way can be stored in the estimation schedule log 1407 so that it can be accessed by the target image acquisition unit 1430 to determine when to acquire an image of a target on the opposite surface to facilitate the online estimation of the trailer posture. In some embodiments, the schedule can correspond to a fixed interval. In some embodiments, the schedule can be set to be dynamic, for example, when there is no change in direction (e.g., no lane change, no turn), the trailer posture is estimated at a first frequency (lower), and when the truck / trailer is changing lanes, the trailer posture is estimated at a second (e.g., much higher) frequency. In some embodiments, the schedule can be set to be adaptive with respect to a certain qualitative assessment of, for example, speed change, which can be used as a trigger to adjust the trailer posture estimation operation. For example, if the truck is slowing down, the frequency of estimating the trailer's posture may be increased.

[0112] By setting the estimation schedule, when the truck / trailer is on the road (in motion), an online estimation of the trailer pose is performed based on the estimation schedule, for example, determined in the estimation schedule log 1407. At 1445, an image of the target is acquired by the target image acquisition unit 1430 according to the estimation schedule, and at 1455, features are extracted from the target image corresponding to the reference target features specified by the information stored in the log 1432 by the target image feature extractor 1440. Such extracted features are sent to the transformation generation unit 1420 to estimate the transformation between the reference features and the extracted features at 1475. In some embodiments, the reference features can come from a calibration phase. In some embodiments, the reference features can come from a previous estimation step or a previous moment. Such reference features and dimensional information can be retrieved at 1465 to estimate the transformation matrix at 1475. Then, the estimated transformation matrix can be sent to the trailer pose estimator 1450 to estimate the pose of the trailer at 1485 based on the features extracted from the acquired image and the reference features stored in 1432. Once estimated, the estimated trailer pose is output at 1495 to facilitate autonomous driving of the truck and trailer.

[0113] Fig.15 The concept of instant trailer pose estimation according to an embodiment of the present teachings is illustrated. As shown herein, the trailer 1220 has two poses, one pose having a center point at the CFS at an initial pose and the other pose having a center point at the CFS at a second pose. As discussed herein, based on the camera transformation [R, t] estimated from two feature point sets, any (one or more) key points of the trailer at the initial pose can be mapped to (one or more) transformed positions using [R', t'], which [R', t'] represents the trailer transformation calculated from [R, t] to derive the current pose of the trailer. For example, if the center point of the rear of the trailer (CRS) is a key point, then as described herein, given the estimated trailer transformation, the center point of the CRS at a previous time point can be mapped to the center point of CRS' at the current time point via equations (7) and (8) (see Fig.15 ). Similarly, any other keypoints used to represent the posture of the trailer can also be calculated in a similar manner by mapping from their initial position or posture (e.g., 1220-3, 1220-4) to their current position or posture at a later point in time (e.g., 1220-3', 1220-4') using [R', t']. In this way, the truck can continuously obtain the updated posture of the trailer and use this dynamic posture information to control autonomous driving so that the truck and trailer are located in the spatial region in a coherent and consistent manner.

[0114] Fig.1616 is an illustrative diagram of an exemplary mobile device architecture that can be used to implement a dedicated system for implementing methods of protecting sensors and sensor assemblies as disclosed in the present teachings according to various embodiments. In this example, the device on which the present teachings are implemented corresponds to a mobile device 1600, including but not limited to a smart phone, a tablet computer, a music player, a handheld game console, a global positioning system (GPS) receiver, a wearable computing device (e.g., glasses, a watch, etc.), or in any other form factor. The mobile device 1600 may include one or more central processing units ("CPU") 1660, one or more graphics processing units ("GPU") 1630, a display 1620, a memory 1660, a communication platform 1610 (such as a wireless communication module), a storage device 1690, and one or more input / output (I / O) devices 1640. Any other suitable components including but not limited to a system bus or controller (not shown) may also be included in the mobile device 1600. As shown in FIG. Fig.16 As shown in FIG. 1 , a mobile operating system 1670 (e.g., iOS, Android, Windows Phone, etc.) and one or more applications 1680 may be loaded from a storage device 1690 into a memory 1660 for execution by a CPU 1640. The applications 1680 may include suitable mobile applications for managing tasks related to the present teachings on the mobile device 1600. User interaction may be achieved via the I / O device 1640.

[0115] In order to realize the various modules, units and functions thereof described in the present disclosure, a computer hardware platform can be used as (one or more) hardware platforms of one or more elements described herein. The hardware elements, operating systems and programming languages ​​of such computers are conventional in nature, and it is assumed that those skilled in the art are sufficiently familiar with this, so that these technologies are adapted to the appropriate settings as described herein. A computer with a user interface element can be used to realize a personal computer (PC) or other types of workstations or terminal devices, but if properly programmed, the computer can also serve as a server. It is believed that those skilled in the art are familiar with the structure, programming and general operation of this computer equipment, and therefore, the accompanying drawings should be self-explanatory.

[0116] Fig.171700 is an illustrative diagram of an exemplary computing device architecture that can be used to implement a dedicated system that implements various functions associated with the present teachings according to various embodiments. This dedicated system incorporating the present teachings has a functional block diagram illustration of a hardware platform that includes a user interface element. The computer can be a general-purpose computer or a special-purpose computer. Both can be used to implement the dedicated system of the present teachings. As described herein, this computer 1700 can be used to implement any component of a conversation or dialogue management system. For example, various functions associated with the present teachings can be implemented on a computer such as computer 1900 via its hardware, software program, firmware, or a combination thereof. Although only one such computer is shown for convenience, the computer functions associated with the conversation management system described herein can be implemented in a distributed manner on multiple similar platforms to distribute the processing load.

[0117] Computer 1700, for example, includes a COM port 1750 connected to a network and connected thereto from a network to facilitate data communication. Computer 1700 also includes a central processing unit (CPU) 1720 in the form of one or more processors for executing program instructions. This exemplary computer platform includes an internal communication bus 1710, program storage devices and data storage devices (e.g., disk 1770, read-only memory (ROM) 1730 or random access memory (RAM) 1740) in different forms, for various data files to be processed and / or transmitted by computer 1700, and possible program instructions to be executed by CPU 1720. Computer 1700 also includes an I / O assembly 1760, which supports input / output streams between the computer and other components therein (such as user interface elements 1780). Computer 1700 can also receive programming and data via network communications.

[0118] Thus, aspects of the methods of conversation management and / or other processes as described above may be implemented in programming. The programmatic aspects of the technology may be viewed as a "product" or "article of manufacture" typically in the form of executable code and / or associated data executed on or implemented in some type of machine-readable medium. Tangible, non-transitory "storage" type media include any or all memory or other storage devices (such as various semiconductor memories, tape drives, disk drives, etc.) for a computer, processor, etc., or its associated modules that may provide storage for software programming at any time.

[0119] Sometimes all or part of the software can be transmitted over a network such as the Internet or various other telecommunication networks. For example, such communications can enable software to be loaded from one computer or processor to another computer or processor, for example in conjunction with session management. Therefore, another type of medium that can carry software elements includes light waves, electric waves, and electromagnetic waves, such as through wired and optical landline networks and through various air links across the physical interface between local devices. The physical elements that carry such waves (such as wired or wireless links, optical links, etc.) can also be regarded as the medium that carries the software. As used herein, unless limited to tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to the processor for execution.

[0120] Thus, machine-readable media can take many forms, including but not limited to tangible storage media, carrier media, or physical transmission media. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any (one or more) computers, etc., which can be used to implement the system shown in the figure or any component thereof. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that form a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals or sound or light waves (such as those generated during radio frequency (RF) and infrared (IR) data communications). Therefore, common forms of computer-readable media include, for example: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers for transporting data or instructions, cables or links for transporting such carriers, or any other media from which a computer can read programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution.

[0121] Those skilled in the art will recognize that the present teachings may be subject to a variety of modifications and / or enhancements. For example, while the various component embodiments described above may be implemented in a hardware device, it may also be implemented as a pure software solution - e.g., installation on an existing server. Furthermore, the fraudulent network detection techniques as disclosed herein may be implemented as firmware, a firmware / software combination, a firmware / hardware combination, or a hardware / firmware / software combination.

[0122] Although the above has described what is considered to constitute the present teaching and / or other examples, it should be understood that various modifications may be made thereto, and the subject matter disclosed herein may be implemented in various forms and examples, and that these teachings may be applied to many applications, only some of which are described herein. The following claims are intended to claim any and all applications, modifications, and variations that fall within the true scope of the present teaching.

Claims

1. A method for posture estimation implemented on at least one machine, the at least one machine comprising at least one processor, a memory and a communication platform capable of connecting to a network, the method comprising: obtaining at least a dimension of a trailer connected to a vehicle, wherein the vehicle and the trailer have respective opposing surfaces facing each other; identifying a first set of feature points from a first image of an object present on a first surface of the opposing surfaces acquired at a first moment by a sensor disposed on a second surface of the opposing surfaces, wherein the object includes a fiducial marker placed on one of the opposing surfaces; Identifying a second set of feature points from a second image of the target acquired by the sensor at a second time, wherein each feature point in the second set of feature points corresponds to a feature point in the first set of feature points; determining a first transformation based on the first set of feature points and the second set of feature points, the first transformation being based on a transformation matrix that aligns feature points in the first set of feature points and the second set of feature points, the first transformation being associated with a change in the position and orientation of the trailer; Calculating a second transformation experienced by the trailer from the first moment to the second moment based on the first transformation, wherein the second transformation is associated with decomposing the transformation matrix into rotational components and translational components; as well as For autonomous driving of the vehicle, a pose of the trailer at a second moment is estimated based on the second transformation and the dimension.

2. The method of claim 1, wherein A first surface of the opposing surfaces is a rear surface of the vehicle; and A second of the opposing surfaces is a front surface of the trailer.

3. The method of claim 1, wherein A first of the opposing surfaces is a front surface of the trailer; and The second surface of the opposing surfaces is a rear surface of the vehicle. 4 . The method of claim 1 , wherein the pose of the trailer is represented by one or more coordinates of corresponding one or more points located on the trailer.

5. The method of claim 1, wherein the target further comprises an area having a recognizable visual feature in one of the opposing surfaces.

6. The method of claim 1, wherein the first time instant corresponds to one of: The initial moment when the vehicle and trailer are aligned in a straight line; and any time before the second time, so that the estimated pose at the second time represents a change in the pose of the trailer compared to the pose of the trailer at the first time. The method of claim 1 , wherein the pose of the trailer is estimated based on a planar homography.

8. A machine-readable and non-transitory medium for pose estimation having information recorded thereon, wherein when the information is read by a machine, the information causes the machine to execute: obtaining at least a dimension of a trailer connected to a vehicle, wherein the vehicle and the trailer have respective opposing surfaces facing each other; identifying a first set of feature points from a first image of an object present on a first surface of the opposing surfaces acquired at a first moment by a sensor disposed on a second surface of the opposing surfaces, wherein the object includes a fiducial marker placed on one of the opposing surfaces; Identifying a second set of feature points from a second image of the target acquired by the sensor at a second time, wherein each feature point in the second set of feature points corresponds to a feature point in the first set of feature points; determining a first transformation based on the first set of feature points and the second set of feature points, the first transformation being based on a transformation matrix that aligns feature points in the first set of feature points and the second set of feature points, the first transformation being associated with a change in the position and orientation of the trailer; Calculating a second transformation experienced by the trailer from the first moment to the second moment based on the first transformation, wherein the second transformation is associated with decomposing the transformation matrix into rotational components and translational components; as well as For autonomous driving of the vehicle, a pose of the trailer at a second moment is estimated based on the second transformation and the dimension.

9. The medium of claim 8, wherein A first surface of the opposing surfaces is a rear surface of the vehicle; and A second of the opposing surfaces is a front surface of the trailer.

10. The medium of claim 8, wherein A first of the opposing surfaces is a front surface of the trailer; and The second surface of the opposing surfaces is a rear surface of the vehicle.

11. The medium of claim 8, wherein the pose of the trailer is represented by one or more coordinates of corresponding one or more points located on the trailer.

12. The medium of claim 8, wherein the target further comprises an area of ​​one of the opposing surfaces having a recognizable visual feature.

13. The medium of claim 8, wherein the first time instant corresponds to one of: The initial moment when the vehicle and trailer are aligned in a straight line; and any time before the second time, so that the estimated pose at the second time represents a change in the pose of the trailer compared to the pose of the trailer at the first time.

14. The medium of claim 8, wherein the pose of the trailer is estimated based on a planar homography.

15. A system for posture estimation, comprising: a configuration unit configured to obtain at least a dimension of a trailer connected to a vehicle, wherein the vehicle and the trailer have respective opposing surfaces facing each other; A target image feature extractor, wherein the target image feature extractor is configured to: identifying a first set of feature points from a first image of an object present on a first surface of the opposing surfaces acquired at a first moment by a sensor disposed on a second surface of the opposing surfaces, wherein the object includes a fiducial marker placed on one of the opposing surfaces, and Identifying a second set of feature points from a second image of the target acquired by the sensor at a second time, wherein each feature point in the second set of feature points corresponds to a feature point in the first set of feature points; a transformation generation unit configured to determine a first transformation based on the first set of feature points and the second set of feature points, the first transformation being based on a transformation matrix that aligns feature points in the first set of feature points and the second set of feature points, the first transformation being associated with a change in the position and orientation of the trailer; as well as A trailer pose estimator configured to: Calculating a second transformation experienced by the trailer from the first moment to the second moment based on the first transformation, wherein the second transformation is associated with decomposing the transformation matrix into a rotation component and a translation component, and For autonomous driving of the vehicle, a pose of the trailer at a second moment is estimated based on the second transformation and the dimension.

16. The system of claim 15, wherein A first surface of the opposing surfaces is a rear surface of the vehicle; and A second of the opposing surfaces is a front surface of the trailer.

17. The system of claim 15, wherein A first of the opposing surfaces is a front surface of the trailer; and The second surface of the opposing surfaces is a rear surface of the vehicle.

18. The system of claim 15, wherein the pose of the trailer is represented by one or more coordinates of corresponding one or more points located on the trailer.

19. The system of claim 15, wherein the target further comprises an area of ​​one of the opposing surfaces having a recognizable visual feature.

20. The system of claim 15, wherein the first time instant corresponds to one of: The initial moment when the vehicle and trailer are aligned in a straight line; and any time before the second time, so that the estimated pose at the second time represents a change in the pose of the trailer compared to the pose of the trailer at the first time.

21. The system of claim 15, wherein the pose of the trailer is estimated based on a planar homography.

Images