System and method for estimating levelness of a trailer and providing leveling guidance

Abstract

Systems and methods for measuring how level a trailer is and providing trailer leveling guidance. The systems and methods include a vehicle including a vehicle levelness estimator, a trailer connectable to the vehicle, and a rear-facing camera mounted on the vehicle. The methods and systems include capturing an image of the trailer using the rear-facing camera, receiving a vehicle levelness parameter from the vehicle levelness estimator of the vehicle, determining a rotation angle of the trailer relative to the vehicle based on the image, determining an estimated trailer levelness parameter based on the rotation angle and the vehicle levelness parameter, and outputting the estimated trailer levelness parameter.

Description

Technical Field

[0001] This disclosure generally relates to estimating the levelness of trailers and leveling trailers. Background Technology

[0002] Typically, certain vehicles (such as motor vehicles) can be used to tow different types of objects that can be mounted on trailers. For example, a motor vehicle can tow a boat, a recreational vehicle, a trailer filled with one or more objects, etc. Trailers such as touring trailers or recreational campervan trailers are moved from one location to another, and some serve as living areas at each new location. The terrain at each location may not be perfectly flat and level, and if possible, it is desirable to keep the trailer in a horizontally level condition at each location so that doors swing normally, gutters drain, the horizontal surface in the trailer does not tilt out of level, appliances function correctly, and generally life in the trailer is not adversely affected.

[0003] Therefore, it is desirable to provide systems and methods for estimating trailer levels without necessarily requiring additional hardware to be installed in the tractor or trailer. Furthermore, other desirable features and characteristics of the invention will become apparent from the following detailed description and appended claims, taken in conjunction with the accompanying drawings and the foregoing technical and background information. Summary of the Invention

[0004] In one aspect, a method for measuring the level of a trailer is provided. The trailer is attached to a vehicle. The method includes capturing an image of the trailer using a rear-facing camera of the vehicle, and receiving vehicle level parameters from a vehicle level estimator of the vehicle via at least one processor. The vehicle level parameters indicate the level of the vehicle. The method includes determining a rotation angle of the trailer relative to the vehicle based on the image via at least one processor, and determining an estimated trailer level parameter based on the rotation angle and the vehicle level parameters via at least one processor. The estimated trailer level parameter indicates the level of the trailer. The method includes outputting the estimated trailer level parameter via at least one processor.

[0005] In one embodiment, outputting the estimated trailer level parameters includes generating a user interface that indicates the estimated trailer level parameters. The user interface may include at least one of the vehicle's native display device and a display device of a mobile computer.

[0006] In one embodiment, outputting the estimated trailer levelness parameters includes integrating levelness information based on the levelness parameters into a digital elevation map.

[0007] In an embodiment, outputting the estimated trailer level parameters includes providing the estimated trailer level parameters to an automatic trailer leveling system.

[0008] In one embodiment, determining the trailer's rotation angle relative to the vehicle based on the image includes aligning the optical features of the image with the optical features of a reference image.

[0009] In one embodiment, the reference image is acquired by the camera during the initialization process when the trailer is known to be leveled and the reference image is stored in memory.

[0010] In one embodiment, the method includes: determining sensor-based trailer level parameters based on level measurements from a trailer level estimator disposed on the trailer, via at least one processor; and determining fused trailer level parameters based on the vision-based trailer level parameters and the sensor-based trailer level parameters, via the at least one processor. In one embodiment, determining the sensor-based trailer level parameters includes regression from a model. In one embodiment, determining the sensor-based trailer level parameters includes determining the height of the trailer at each of the opposite sides. In one embodiment, determining the sensor-based trailer level parameters includes converting the height of the trailer at each of the opposite sides into a roll angle.

[0011] In an embodiment, determining the fused trailer level parameters includes fusing sensor-based trailer level parameters and roll angles via at least one processor using a variance-based weighting scheme or other fusion techniques including at least one of Bayesian fusion or Dempster combination rules.

[0012] In one embodiment, the trailer level estimator is included in a personal mobile computing device that includes an inertial measurement unit (IMU). In another embodiment, the vehicle level estimator includes a vehicle inertial measurement unit (IMU).

[0013] In another aspect, a system for measuring the level of a trailer is provided. The system includes a vehicle comprising a vehicle level estimator, a trailer connectable to the vehicle, a rear-facing camera mounted on the vehicle, and at least one processor operatively communicating with the rear-facing camera. The at least one processor is configured to execute program instructions. The program instructions are configured to cause the at least one processor to: capture an image of the trailer using the rear-facing camera, and receive vehicle level parameters from the vehicle level estimator. The vehicle level parameters indicate the level of the vehicle. The program instructions are also configured to cause the at least one processor to determine a rotation angle of the trailer relative to the vehicle based on the image, and to determine an estimated trailer level parameter based on the rotation angle and the vehicle level parameters. The estimated trailer level parameter indicates the level of the trailer. The program instructions are further configured to cause the at least one processor to output the estimated trailer level parameter.

[0014] In an embodiment, program instructions are configured to cause at least one processor to: determine sensor-based trailer level parameters based on level measurements from a trailer level estimator mounted on the trailer, and determine fused trailer level parameters based on the roll angle and the sensor-based trailer level parameters. In an embodiment, determining the sensor-based trailer level parameters includes regression from a model. In an embodiment, determining the sensor-based trailer level parameters includes determining the height of the trailer at each of the opposite sides and converting the height of the trailer at each of the opposite sides into a roll angle.

[0015] In one embodiment, the trailer level estimator is included in a personal mobile computing device that includes an inertial measurement unit (IMU).

[0016] In one embodiment, the vehicle level estimator includes a vehicle inertial measurement unit (IMU). Attached Figure Description

[0017] Exemplary embodiments will be described below in conjunction with the accompanying drawings, wherein the same reference numerals denote the same elements, and wherein:

[0018] Figure 1 This is a schematic diagram of a tractor vehicle according to various embodiments, which is coupled to and tows a trailer and includes components of a trailer leveling estimation system.

[0019] Figure 2 This illustrates various embodiments. Figure 1 Functional block diagram of the components including the tractor and the trailer level estimation system;

[0020] Figure 3 This is a functional block diagram illustrating a trailer level estimation system according to various embodiments;

[0021] Figure 4 This is a data flow diagram illustrating vision-based level estimation according to various embodiments;

[0022] Figure 5 This is a data flow diagram illustrating sensor-based level estimation according to various embodiments; and

[0023] Figure 6 This is a flowchart illustrating the steps of a method for estimating the levelness of a trailer according to various embodiments. Detailed Implementation

[0024] The detailed description below is merely exemplary in nature and is not intended to limit application or use. Furthermore, it is not intended to be bound by any express or implied theory presented in the foregoing technical field, background art, summary of the invention, or the following detailed description. As used herein, the term "module" refers individually or in any combination of any hardware, software, firmware, electronic control components, processing logic, and / or processor devices, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped) and memories executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functionality.

[0025] This document describes embodiments of the present disclosure in terms of functional and / or logical block components and various processing steps. It should be understood that such block components can be implemented by any number of hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of the present disclosure may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices. Furthermore, those skilled in the art will understand that embodiments of the present disclosure can be practiced in conjunction with any number of systems, and the systems described herein are merely exemplary embodiments of the present disclosure.

[0026] For the sake of brevity, this document may not describe in detail conventional techniques related to signal processing, data transmission, signaling, control, and other functional aspects of the system (and its various operating components). Furthermore, the connecting lines shown in the various figures included herein are intended to illustrate exemplary functional relationships and / or physical couplings between various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in the embodiments of this disclosure.

[0027] This disclosure utilizes an onboard camera and one or more trailer IMU sensors (or other trailer level estimators) to generate trailer leveling features to help users properly level their trailers. High-definition (HD) elevation data can be used to determine the level status for initializing / calibrating the system.

[0028] The system and method described in this paper can initialize a leveling algorithm using high-definition elevation data or buffered in-vehicle level estimation. A vision-based algorithm is described that predicts the angle or height of trailer unevenness relative to the vehicle. A sensor-based technique (e.g., inertial measurement unit (IMU)) is disclosed that estimates the angle or height of trailer unevenness relative to the horizontal plane. The statistical trailer unevenness angle or height prediction technique is formulated using information fusion to leverage complementary aspects of single-modal level estimation (sensor-based and vision-based) as a way to enhance overall accuracy and robustness. Location-based leveling scores and user experience scores are provided through crowd perception for recommending the best nearby known campsites.

[0029] The trailer leveling estimate obtained by the system and method described in this paper can be used as an enabler for manual leveling guidance, such as informing the user to place leveling blocks (made of wood, plastic, or other suitable materials) of a certain thickness and width under the wheels to level the trailer. Other manual leveling techniques are also possible. Furthermore, the trailer leveling estimate can be used as an enabler for automatic leveling mechanisms. The trailer leveling estimate can be integrated into off-road high-definition 3D (three-dimensional) maps.

[0030] The systems and methods described herein can help users level trailers during parking by indicating trailer levelness using only a single camera on the towing vehicle (e.g., a truck) and a low-cost in-vehicle image analysis algorithm, without requiring users to install additional equipment or sensors on / inside the trailer. To provide users with the ability to receive more accurate levelness indications, the algorithm can fuse predictions generated from (a) onboard vision and (b) common levelness (e.g., IMU) sensors placed inside the trailer (such as a mobile phone with an IMU) or placed on the trailer's external surface (such as a drone with an IMU), supported by a companion application and data communication with the vehicle. The systems and methods described herein save users time and the cost of additional equipment for leveling their RV trailers while parking. The systems and methods allow users to receive recommendations about possible RV parking locations based on inference from previously leveled maps and user experience. As used herein, the terms "levelness" and "levelness" are used to describe whether the trailer is level and are used in conjunction with a quantitative measurement of the trailer's angle relative to the horizontal plane.

[0031] refer to Figure 1According to various embodiments, a trailer leveling estimation system, generally shown as 100, is associated with a tractor unit 10 and a trailer 8. Typically, the trailer leveling estimation system 100 uses at least a vision-based technique to estimate the level of the trailer 8, thereby capturing an image of the trailer with a rear-facing camera and processing the image to determine the relative rotation (tilt) angle between the tractor unit 10 and the trailer 8. In some embodiments, the vision-based leveling estimation is combined with a leveling sensor-based leveling estimation, through which one or more mobile computing devices, including a leveling sensor (e.g., an IMU), are placed on the trailer 8. The fusion of the vision-based and sensor-based leveling estimations provides enhanced robustness and accuracy of the leveling estimation results. The trailer leveling estimation parameter output can be used as input to an automated trailer leveling system, as input to generate a graphical user interface indicating the leveling status, and as input for integration into a digital elevation map.

[0032] like Figure 1 and Figure 2 As shown, the tractor unit 10 is connected to the trailer unit 8. The tractor unit 10 typically includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is mounted on the chassis 12 and substantially surrounds the components of the vehicle 10. The body 14 and the chassis 12 may together form a frame. The wheels 16-18 are each rotatably connected to the chassis 12 near a corresponding corner of the body 14.

[0033] In various embodiments, the trailer level estimation system 100 is at least partially integrated into the tractor unit 10 (hereinafter referred to as vehicle 10). In other embodiments, the trailer level estimation system 100 includes computer program software implemented in a computer of vehicle 10, in a personal mobile computing device, in a computer of trailer 8, in a cloud computing service, or distributed between them. In some embodiments, vehicle 10 may be an autonomous vehicle capable of sensing its environment and navigating with little or no user input. Vehicle 10 is depicted as a truck in the illustrated embodiments, but it should be understood that any other vehicle may be used, including motorcycles, passenger cars, sport utility vehicles (SUVs), recreational vehicles (RVs), boats, aircraft, etc. Furthermore, the teachings herein are not limited to use only with automobiles, but may also be used with other types of vehicles. For example, the teachings herein may be compatible with vehicles including, but not limited to, aircraft, railcars, commercial vehicles, including tractors, aircraft tractors, aircraft push-pull tractors, forklifts, etc.

[0034] As shown in the figure, refer to Figure 1 and Figure 2Vehicle 10 typically includes a hitch receiver 20, a hitch ball joint 22, a driver communication system 24, a propulsion system 26, a transmission system 28, a steering system 30, a braking system 32, a sensor system 34, an actuator system 36, at least one controller 40, and a communication system 42. As will be discussed, the tractor 10 also includes one or more cameras 44. (See reference...) Figure 1 The hook-up receiver 20 is attached to the rear 46 of the vehicle 10 and is typically attached to the frame of the vehicle 10. The hook-up receiver 20 defines a cavity or slot for receiving the hook-up ball head 22. The hook-up ball head 22 is typically locked into the hook-up receiver 20 such that the hook-up ball head 22 is secured within the slot. The hook-up ball head 22 receives the tongue 48 of the trailer 8. It should be noted that the hook-up ball head 22 does not necessarily include a ball head, but may include a pivot hook, gooseneck hook, tow hitch hook, etc. The tongue 48 typically defines a concave receiving portion that mates with the hook-up ball head 22. In some cases, the tongue 48 of the trailer 8 is attached to the body 50 of the trailer 8 via one or more tie rods 52.

[0035] In various embodiments, propulsion system 26 may include an internal combustion engine, an electric motor such as a traction motor, and / or a fuel cell propulsion system. Transmission system 28 is configured to transmit power from propulsion system 26 to wheels 16-18 according to a selectable speed ratio. According to various embodiments, transmission system 28 may include a step-ratio automatic transmission, a continuously variable transmission (CVT), or other suitable transmission. Braking system 32 is configured to provide braking torque to wheels 16-18. In various embodiments, braking system 32 may include friction brakes, line brakes, regenerative braking systems such as electric motors, and / or other suitable braking systems. Steering system 30 affects the position of wheels 16-18. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of the invention, steering system 30 may not include a steering wheel.

[0036] The driver communication system 24 displays data to the driver and occupants of vehicle 10 and also receives input from the driver and occupants of vehicle 10. In one example, the driver communication system 24 includes an infotainment system and includes a display 54 and an input device 56. The display 54 is implemented as a flat panel display in the dashboard or console of vehicle 10. In this example, the display 54 is an electronic display capable of graphically displaying one or more user interfaces under the control of controller 40. Those skilled in the art will recognize other techniques for implementing the display 54 in vehicle 10. The display 54 includes any suitable technology for displaying information, including but not limited to liquid crystal displays (LCDs), organic light-emitting diodes (OLEDs), plasma displays, or cathode ray tubes (CRTs). As will be described, a graphical user interface depicting the levelness state of trailer 8 estimated by trailer levelness estimation system 100 can be displayed on the display 54 in vehicle 10.

[0037] Input device 56 receives input from the driver and / or occupants of vehicle 10. Input device 56 may be implemented as a keyboard (not shown separately), a microphone associated with a voice recognition system (not shown separately), a touchscreen layer associated with display 54, a switch or lever, one or more buttons, a driver monitoring system (DMS), or other suitable devices for receiving data and / or commands from the user. Multiple input devices may also be used. For example, at least one input device may be coupled to a handwheel. Display 54 and input device 56 communicate with controller 40 through a suitable communication architecture or arrangement that facilitates the transmission of data, commands, power, etc.

[0038] Sensor system 34 includes one or more sensing devices 58a-58n, each sensing device sensing observable conditions of the external and / or internal environment of vehicle 10. Sensing devices 58a-58n include, but are not limited to, radar, lidar, optical cameras, thermal cameras, ultrasonic sensors, and / or other sensors. Additionally, one or more of the sensing devices 58a-58n may include at least a vehicle speed sensor. The one or more sensing devices 58a-58n communicate with controller 40 via a suitable communication structure or arrangement that facilitates the transmission of data, commands, power, etc. The vehicle speed sensor measures and observes the speed of the tractor 10, and in one example, the vehicle speed sensor measures and observes the speed of the axle of the tractor 10 and generates a sensor signal based on this. It should be noted that although the vehicle speed sensor is described herein as being associated with measuring and observing the speed of the axle, alternatively, the speed of the tractor 10 may be inferred from the engine speed or determined through modeling, computer vision methods (such as visual odometers, etc.). The one or more sensing devices 58a-58n also include one or more steering angle sensors that observe the angles of steering system components, such as the steering knuckle angle, handwheel angle, etc. This indicates whether vehicle 10 is turning. Typically, one or more steering angle sensors observe the movement of steering system 30, which is processed by controller 40 to determine the steering angle of vehicle 10. Sensing devices 58a-58n may also include range sensors that observe the current range of transmission system 28 and generate a sensor signal based on this, which is transmitted to controller 40. One or more sensing devices 58a-58n include a vehicle level estimator 58a, which provides a measurement of the current level of vehicle 10. The vehicle level estimator 58a may be implemented by a local IMU of vehicle 10 or by some other vehicle level estimator 58a, such as digital inclinometers of the following types: liquid capacitive, electrolytic, bubble-in-liquid, and pendulum.

[0039] The actuator system 36 includes one or more actuator devices 60a-60n that control one or more vehicle features, such as, but not limited to, the propulsion system 26, the transmission system 28, the steering system 30, and the braking system 32. In various embodiments, the vehicle features may also include interior and / or exterior vehicle features, such as, but not limited to, doors, trunks, and cabin features, such as air, music, lighting, etc. (not numbered in the figures).

[0040] Communication system 42 is configured to wirelessly transmit information to and from other entities 62, such as, but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems, one or more cameras 44a, 44b coupled to trailer 8, and / or personal devices, such as one or more smartphones, tablets, etc. In an exemplary embodiment, communication system 42 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communication. However, additional or alternative communication methods, such as Dedicated Short Range Communication (DSRC) channels, are also considered within the scope of this disclosure. A DSRC channel refers to a one-way or two-way short-to-medium range wireless communication channel specifically designed for automotive applications, along with a corresponding set of protocols and standards.

[0041] The controller 40 includes at least one processor 64 and a computer-readable storage device or medium 66. The processor 64 may be any custom or commercially available processor, central processing unit (CPU), graphics processing unit (GPU), auxiliary processor among several processors associated with the controller 40, semiconductor-based microprocessor (in the form of a microchip or chipset), microprocessor, any combination thereof, or any device generally used for executing instructions. The computer-readable storage device or medium 66 may include volatile and non-volatile storage, such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operational variables when the processor 64 is powered off. The computer-readable storage device or medium 66 may be implemented using any of a variety of known memory devices, such as PROMs (programmable read-only memory), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or any other electrical, magnetic, optical, or combined memory device capable of storing data, some of which represents executable instructions used by the controller 40 in controlling the vehicle 10.

[0042] The instructions may include one or more separate programs, each forming an ordered list of executable instructions for implementing logical functions. When executed by processor 64, the instructions receive and process signals from sensor system 34, execute logic, calculations, methods, and / or algorithms for automatically controlling components of vehicle 10, and generate control signals to actuator system 36 to automatically control components of vehicle 10 based on logic, calculations, methods, and / or algorithms. Although Figure 1Only one controller 40 is shown, but embodiments of vehicle 10 may include any number of controllers 40 that communicate via any suitable communication medium or combination of communication media and cooperate to process sensor signals, perform logic, calculations, methods and / or algorithms, and generate control signals to automatically control the features of vehicle 10.

[0043] In various embodiments, one or more instructions of controller 40 are embodied in trailer level estimation system 100, and when executed by processor 64, cause processor 64 to receive and process signals from one or more cameras 44a, 44b, vehicle level estimator 58a, trailer level estimator (discussed further below), and estimate the level (or tilt angle or roll angle) of trailer 8. When executed by processor 64, one or more instructions of controller 40 may fuse vision-based level estimation and sensor-based level estimation, and may generate a display indicating the degree of deviation of trailer 8 from the horizontal plane or level conditions.

[0044] According to various embodiments, the tractor unit 10 includes one or more cameras 44a, 44b. In one embodiment, one of the cameras 44b is coupled to the rear 46 of the vehicle 10. In this example, one of the cameras 44b is mounted to the tailgate of the vehicle 10; however, the cameras 44a, 44b can be mounted at any desired location along the rear 46 to face the trailer 8. An additional or alternative camera 44a is mounted to the top of the cab of the tractor unit 10 between a center-high mounted parking light and the roof of the cab, and has a field of view surrounding the flat plate of the tractor unit 10 (referred to as a center-high mounted parking light (CHMSL) camera). The one or more cameras 44a, 44b face rearward and have a field of view that includes at least a portion of the front of the trailer 8, making it possible to process any captured images to determine the relative roll angle between the trailer 8 and the vehicle 10. Each of the cameras 44a, 44b on the vehicle 10 communicates wirelessly or via a wired connection to a controller 40 to a communication architecture that facilitates the transmission of data, power, commands, etc.

[0045] Cameras 44a and 44b each include any camera capable of capturing image data or image data streams, as known to those skilled in the art. In some embodiments, one or more of cameras 44A and 44B may include a color camera capable of capturing color images. In other embodiments, one or more of cameras 44A and 44B may include an infrared camera to capture infrared images. In some embodiments, one or more of cameras 44A and 44B may include a grayscale camera for capturing grayscale images. In some embodiments, one or more of cameras 44A and 44B may include a stereo camera assembly capable of capturing stereo images with depth information. For example, one or more of cameras 44A and 44B may include a stereo camera having two or more lenses and image sensors arranged to capture stereo images with depth information about vehicle 10 and / or trailer 8. One or more of cameras 44A and 44B may be remote cameras for capturing images over extended distances. Cameras 44A and 44B may capture images according to various timing or other considerations. In some embodiments, for example, the respective cameras 44a, 44b may capture images continuously while the vehicle 10 is moving (based on sensor signals from one or more sensing devices 58a-58n) or based on the gear range of the transmission. In some embodiments, when the vehicle 10 is moving, the controller 40 may cause the respective cameras 44a, 44b to capture images of the area near the rear 46 of the vehicle 10 and the rear 70 of the trailer 8 at regular time intervals. For example, when the vehicle 10 is stopped or moving very slowly, the controller 40 may process the images captured by the cameras 44a, 44b to estimate the levelness of the trailer 8. In other embodiments, the cameras 44A, 44B capture images upon request from a user via input device 56 or via an application on a personal mobile computing device to initiate trailer levelness estimation.

[0046] As will be discussed further below, the processor 64 receives and processes image data or image streams from one or more cameras 44a, 44b, and compares their optical characteristics with the optical characteristics of a reference image taken with the trailer 8 rotated relative to the vehicle 10 at a known angle and with the vehicle 10 at a known angle relative to the horizontal plane. The known angle can be zero degrees. This comparison allows the processor 64 to derive the current angular rotation of the trailer 8 relative to the vehicle 10.

[0047] For example, such as regarding Figure 3 For more details, please refer to [link / reference]. Figures 1 to 2The data flow diagram illustrates various embodiments of the trailer level estimation system 100, which can be embedded within the controller 40. Various embodiments of the trailer level estimation system 100 according to this disclosure may include any number of modules embedded within the controller 40. It is understood that... Figure 3 The modules shown can be combined and / or further subdivided to similarly generate estimates of trailer levelness (or tilt relative to the horizontal plane). In various embodiments, the trailer levelness estimation system 100 includes a digital elevation map 106, one or more cameras 44a, 44b, a vision-based levelness estimation module 104, a vehicle levelness estimator 58a, an initialization module 110, a levelness sensor-based levelness estimation module 124, a trailer levelness estimator 126, a personal mobile computing device 128, a fusion module 120, an output module 130, a display 54, and an automatic trailer leveling system 134.

[0048] Continue to refer to Figure 3 In the exemplary embodiment described herein, one or more cameras 44A, 44B provide image data 102 to a vision-based level estimation module 104. Image data 102 includes one or more digital images surrounding the front of trailer 8. The vision-based level estimation module 104 compares the optical features of the images in image data 102 with corresponding optical features of a reference image contained in reference image data 141 to determine the relative rotation angle between them. Since the reference image was captured with a known roll angle of trailer 8, the relative rotation angle between trailer 8 and vehicle 10 can be determined. The vision-based level estimation module 104 combines the relative rotation angle between trailer 8 and vehicle 10 with a vehicle level parameter 108 measured from a vehicle level estimator 58a. In other embodiments, the vehicle level parameter 108 can be derived from a digital elevation map 106 and the current position of vehicle 10 (which can be obtained from the GPS of the sensor system 34 of vehicle 10). The vision-based level estimation module 104 generates a vision-based trailer level parameter 136 indicating the roll angle of the trailer 8 relative to the horizontal plane. The vision-based trailer level parameter 136 is a quantitative estimate of the trailer level based on the vision processing technique described herein, and can be expressed as the roll angle of the trailer 8 relative to the vehicle 10.

[0049] In some embodiments, the quality of image data 102 is evaluated by a vision-based level estimation module 104 to determine whether the image data 102 has sufficient quality to allow for accurate level estimation using vision techniques. The vision-based level estimation module 104 can determine an Image Quality Index (IQI) on the image data, which is an algorithm used to estimate the level of degradation present in the image. If the IQI is below a threshold, the trailer level estimation system 100 can revert to a level sensor-based technique, which will be discussed further below. When more than one camera providing image data is present (e.g., CHMSL camera 44a and rear-view camera 44b), the IQI can be used to determine how much weight should be given to each vision-based trailer level parameter 136 determined based on images from the respective cameras 44a, 44b. Lower IQIs result in a smaller weight being given to the corresponding vision-based trailer level parameter 136 when the vision-based trailer level parameters 136 are combined in the fusion module 120.

[0050] exist Figure 3 In an exemplary embodiment, the trailer level estimation system 100 includes a level sensor-based level estimation module 124. The level sensor-based level estimation module 124 receives trailer level parameters 142 from a trailer level estimator 126. The trailer level estimator 126 can be any type of digital inclinometer, including an IMU, a liquid capacitor, an electrolyzer, a bubble in a liquid, and a pendulum. In the illustrated embodiment, the trailer level estimator 126 is a multi-axis IMU included in a personal mobile computing device 128. The personal mobile computing device 128 can be a laptop computer, tablet computer, smartphone, etc. The trailer level parameters 142 can be included in other types of electronic devices with local IMUs, such as unmanned aerial vehicles (UAVs). In some embodiments, the personal mobile computing device 128 is removably mounted (e.g., placed) on the surface of the trailer 8, so that fixed installation is not required. The level sensor-based level estimation module 124 receives the trailer level parameters 142 and determines the trailer level based on them. The level estimation module 124 based on the level sensor can use a regression model or artificial intelligence to determine the sensor-based trailer level parameter, which indicates the level of the trailer. The sensor-based trailer level parameter 138 is a quantitative estimate of the trailer level using the trailer level parameter 142 provided by the trailer level estimator 126 and the regression model.

[0051] The fusion module 120 receives vision-based trailer level parameters 136 and sensor-based trailer level parameters 138, and determines fused trailer level parameters 140 based on these. Information fusion can be performed in various ways. In one exemplary embodiment, the following equation is used:

[0052]

[0053] in:

[0054] Using variance σ v The trailer level is estimated at time t using the vision (i.e., using the vision-based level estimation module 104);

[0055] Using variance σ i The trailer level is estimated at time t by the IMU (i.e., the level estimation module 124 using the level sensor-based level estimation module);

[0056] w v and w i : These are the weights used to aggregate multiple estimates; the weights are inversely proportional to the variance; as mentioned earlier, for vision-based trailer level estimation, the weights can also be inversely proportional to the IQI; and

[0057] Trailer level estimated based on information fusion (i.e., fused trailer level parameter 140).

[0058] While the fusion module 120 allows for enhanced accuracy in the determination of the estimated trailer level, it can derive a vision-based level estimate that is not fused with the level sensor-based estimate. Figure 1 Other fusion techniques are possible, including Bayesian fusion or Dempster combination rules, which take into account the confidence level of each modality.

[0059] Output module 130 receives the fused trailer leveling parameters 140 and orchestrates one or more outputs, including generating user interface 132, providing trailer leveling information to automated trailer leveling system 134, and integrating the trailer leveling information into digital elevation map 106. User interface 132 may be a graphical user interface provided to display 54 of vehicle 10. In other embodiments, user interface 132 is provided to display of personal mobile computing device 128. User interface 132 depicts an interpretation of the fused trailer leveling parameters 140, allowing the user to determine the leveling state of trailer 8 (e.g., roll angle) and take corrective actions. Corrective actions may include placing material under one of the wheels or jacks of trailer 8 to bring the trailer level (zero tilt relative to the horizontal plane along the wheel axis). Corrective actions may additionally or alternatively include changing the height setting of the mast jacks. The fused trailer leveling parameters 140 may be in the form of a roll angle (the angle of inclination of the wheel axis of trailer 8), and this can be converted by trigonometric methods to the height (or height difference) of each of the opposite sides of trailer 8. Height provides a more intuitive value for the user because the amount of adjustment required to balance the height is more directly understandable than the roll angle. The user interface 132 can depict the tilt angle—the difference in level or height between opposite sides of the trailer 8—numerically and / or graphically. Figure 3 In the example, the user interface displays the horizontal axis of trailer 8 based on the fused trailer level parameter 140, and may also display a horizontal axis for reference.

[0060] The automatic trailer leveling system 134 may have motorized or hydraulic devices for adjusting the height of the trailer 8 on each of the opposite sides of the trailer 8. Based on the integrated trailer leveling parameter 140, the automatic trailer leveling system 134 receives which side of the trailer needs height adjustment and by how much in order to automatically level the trailer 8.

[0061] Digital elevation map 106 may include high-density georeferenced elevation information for roads, but may exclude such data for off-road locations or other common trailer parking areas. Output module 130 receives GPS data from personal mobile computing device 128 and / or from sensor system 34 of vehicle 10 and integrates levelness information along with the current location into digital elevation map 106. Digital elevation map 106 may be cloud-based, allowing levelness information to be accessed by other users to determine flat areas suitable for trailer parking 8.

[0062] Figure 4A data flow diagram illustrating the steps performed in the vision-based level estimation module 104 according to an exemplary embodiment is provided. The vision-based level estimation module 104 receives a reference image frame 202 and a test image frame 206. The reference image frame 202 is retrieved from a computer-readable storage device or medium 66 or other memory. The reference image frame 206 is included in reference image data 140 and is established as part of an initialization process performed by the initialization module 110. The initialization process can be initiated based on a request from a user via input device 56 or based on the detection of level conditions. The level condition can be detected by referring to the digital elevation map 106 and the current positions of the vehicle 10 and trailer 8, or by referring to the current vehicle level parameter 108 from the vehicle level estimator 58a. Alternatively, the level condition of the trailer 8 can be confirmed by a user via input device 56. In other embodiments, the reference image frame 202 may be stored as predetermined information, for example, during manufacturing. When the trailer level is known, image data 102 received from one or more cameras 44a, 44b can be stored as a reference image frame 202 in a computer-readable storage device or medium 66. Therefore, when the roll angle between the trailer 8 and the vehicle 10 is zero degrees, the reference image frame 202 provides a baseline. However, assuming the roll angle is known, the reference image frame 202 can be established at roll angles other than zero degrees.

[0063] The vision-based level estimation module 104 further receives a test image frame 206 as part of the image data 102. The test image frame 206 is an input image captured by one or more cameras 44a, 44b when the roll angle of the trailer 8 relative to the vehicle 10 is unknown. The vision-based level estimation module 104 may include an image processing submodule 204, which performs image preprocessing steps including image correction and region of interest (ROI) cropping. Image correction of the reference image frame 202 and the test image frame 206 projects the received images onto a common plane. Figure 4 Reference image frame 202 and test image frame 218 include some fisheye distortion, which in one embodiment is corrected by image processing submodule 204 using a spherical distortion model and a digital distortion model to provide a corrected reference image 214 and a corrected test image 220. Furthermore, a region of interest (ROI) selection step can be performed by image processing submodule 204, thereby selecting a portion of the corrected reference image 214 and the corrected test image 220, and cropping the surrounding area to provide a cropped reference image 216 and a cropped test image 222. ROI selection can be based on a predetermined portion of the input image, or it can be performed by identifying optical features of interest and drawing the associated region of interest.

[0064] The vision-based level estimation module 104 includes a rotation angle estimation submodule 210. The rotation angle estimation submodule 210 compares a cropped reference image 216 and a cropped test image 222, and aligns optical features to estimate the relative rotation required to align the optical features of the cropped test image 222 and the cropped reference image 216. This relative rotation reveals a rotation angle indicating the roll angle between the trailer 8 and the vehicle 10. The vision-based level estimation module 104 can combine the rotation angle 224 with the vehicle level parameter 108 to determine a vision-based trailer level parameter 136. The vision-based trailer level parameter 136 can be expressed as a roll angle in degrees, or as the relative height of opposite sides of the trailer 8 via trigonometric transformation.

[0065] Figure 5 A data flow diagram illustrating algorithmic steps performed in a level estimation module 124 based on a level sensor, according to an exemplary embodiment, is shown. The level estimation module 124 based on a level sensor is shown having a training phase 306 and a testing phase 304. During the training phase 306, the trailer 8 has height measuring devices 302a, 302b, 302c, 302d mounted to at least two and optionally four opposite corners of the trailer 8. In one embodiment, height measuring devices 302a-d include laser altimeters that can provide digital output or may include a display recorded by a camera. Other digital altimeters, such as ultrasonic-based digital altimeters, may be used. The height measuring devices 302a-d measure the height of the trailer 8 relative to the ground, thereby providing an indication of the trailer 8's roll angle relative to the truck 10. During the training phase 306, one or more personal mobile computing devices 128a, 128b each include a trailer level estimator 126 and provide a signal representing a trailer level parameter 142. In the case of an IMU, the signal output from the trailer level estimator 126 can be a magnetic field measurement. Training phase 306 may include an optical character recognition (OCR) submodule 316 for converting height measurements recorded by the camera onto the displays of the height measurement devices 302a-d. The OCR provides digital height values ​​for further processing. Training phase 306 includes a signal alignment and tagging submodule 308, which times-aligns the trailer level parameter 142 with the height measurements from the height measurement devices 302a-d (using signal timestamps) and associates it with the height measurements from the height measurement devices 302a-d. The time-aligned association of the trailer level parameter with the height (or height difference) measurement is essentially an automatic tagging of the trailer level parameter with the height measurement value (or relative height measurement value).

[0066] Training phase 306 also includes a feature extraction submodule, which extracts relevant features from the signal representing trailer levelness parameter 142. During training phase 306, a regression model 312 is created and refined, reflecting the relationship between the heights (or relative heights or roll angles) of trailer 8 at opposite sides of two or four corners of the trailer. Once the regression model 312 is established, it can then be used during testing phase 304, through which the trailer levelness parameter 142 is input from personal mobile computing devices 128a, 128b, features are extracted by the feature extraction submodule 310, and the regression model 312 is used to convert the trailer levelness parameter into an estimated height of each of the opposite sides of trailer 8 relative to the ground. Alternatively, the regression model 312 can produce roll angle or relative height measurements. Testing phase 304 may also include a transformation submodule 314 to convert the estimated height output from regression model 312 into roll angles, allowing similar parameters to be fused by fusion module 120. The roll angle is output from the level estimation module 124 based on the level sensor as the sensor-based trailer level parameter 138.

[0067] Continue to refer to Figure 5 In one embodiment, during training phase 306, digital altimeter readings from one or more locations of interest on each side of trailer 8 are used from height measurement devices 302a-d. A weighted average of the height readings from the same side is used to summarize a meter reading on the left and right sides of trailer 8 at a given timestamp. IMU signals generated from one or more sensors placed inside or on trailer 8 (e.g., included in personal mobile computing devices 128a, 128b) are aligned with the input meter readings using timestamps. A median filter may be used to smooth the IMU signals at the time of interest. A regression model 312 is constructed using a common support vector machine with a radial basis kernel, or other nonlinear regression fitting may be used as a quadratic regression fitting. During testing phase 304, the IMU signals (trailer level parameter 142) are used to predict the trailer height on each side, and this height can be converted into a rotation angle using trigonometric calculations via transformation submodule 314.

[0068] Now for reference Figure 6 And continue to refer to Figures 1 to 5 The flowchart illustrates the process that can be performed according to this disclosure by Figures 1 to 5 The trailer level estimation system 100 performs a method 400 for estimating the level of trailer 8. In various embodiments, method 400 is performed by processor 64 of controller 40. In other embodiments, method 400 is performed by processor of personal mobile computing device 128. As will be understood from this disclosure, the order of operations within method 400 is not limited to... Figure 6The method is not executed in the order shown, but may be executed in one or more different orders as applicable and in accordance with this disclosure. In various embodiments, method 400 may be arranged to run based on one or more predetermined events, such as detecting the parking state of trailer 8 (e.g., by determining a predetermined minimum amount of time that vehicle 10 is stationary), and / or may run in response to a user request via input device 56.

[0069] In step 410, a vehicle level parameter 108 is received from vehicle 10. The vehicle level parameter 108 may correspond to an IMU reading obtained from the vehicle level estimator 85a via the vehicle 10's controller area network (CAN). The vehicle level parameter 108 indicates the tilt angle of vehicle 10 relative to the horizontal plane.

[0070] In step 420, one or more cameras 44 capture images of the front of the trailer 8. The captured images are as follows: Figure 4 The test image frame 218 is shown. The image may include easily identifiable optical features, such as writing or other graphics included in the front of the trailer 8. In step 430, a vision-based level estimation is performed. Step 430 may depend on the IQI measurement of the test image frame 218 exceeding a predetermined threshold. In cases where more than one camera 44A, 44B is providing the corresponding test image frame, the IQI measurement may be used to determine the weights to be used when combining the vision-based trailer level parameter 136 determined based on the corresponding test image frame. In step 430, the test image frame 218 is corrected and the ROI is cropped to provide a cropped test image 222. The cropped test image 222 is compared with a cropped reference image 216 to determine how much relative rotation is needed to align the optical features of the cropped test image 222 and the cropped reference image 216, which was taken at a known rotation angle of the vehicle 10 relative to the horizontal plane and relative to the trailer 8. The step of estimating the required rotation angle may include detecting features in the cropped reference image 216 and the cropped test image 222, extracting feature descriptors, determining the position of corresponding points in each image, matching points, and estimating a transform to transform between the matching point pairs. An exemplary algorithm for optical feature matching to determine the rotation angle is Scale Invariant Feature Transform (SIFT). Since the angle relative to the horizontal plane of the vehicle 10 is known from the vehicle level parameter 108, and the relative roll angle between the vehicle 10 and the trailer 8 has been estimated in step 430, the roll angle relative to the horizontal plane can be estimated. Using trigonometric methods, this angle can be converted into the absolute height of the two sides of the trailer 8 relative to the ground, or into the relative height difference between opposite sides of the trailer 8.

[0071] In step 440, trailer leveling parameter 142 is received. Trailer leveling parameter 142 can be received from a trailer leveling estimator 126 (e.g., an IMU) disposed on or within trailer 8. In one embodiment, a personal mobile computing device 128 (such as a mobile phone) is placed on a surface (e.g., a desktop) of trailer 8 designed for horizontal orientation. In step 450, sensor-based leveling estimation is performed, which converts the trailer leveling parameter into a roll angle of trailer 8 or a height measurement of trailer 8 relative to its lateral side. In an exemplary embodiment, sensor-based leveling estimation utilizes a regression model 312, which has been trained to define the relationship between features extracted from trailer leveling parameter 142 and the height of trailer 8 relative to its lateral side above the ground or the roll angle of trailer 8.

[0072] In step 460, the sensor-based trailer leveling parameter 138 and the vision-based trailer leveling parameter 136 from step 450 are fused. The fusion of these leveling parameters may be weighted based on the inverse of the variance (or confidence score) tracked over time during use or during testing of the trailer leveling estimation system 100.

[0073] In step 470, an output is generated based on the fused trailer leveling parameter 140 from step 460. The output may include a user interface 132 on the display 54 of vehicle 10 or the display of personal mobile computing device 128. The user interface 132 may provide a graphical depiction of the tilt angle or height adjustment required to level the trailer 8 relative to the horizontal plane. The output may additionally or alternatively include integrating the leveling measurement results along with the location where the leveling measurement was taken into a digital elevation map 106. The output may additionally or alternatively include providing the fused trailer leveling parameter 140 to an automated trailer leveling system 134 for automatically adjusting the relative height of opposite sides of the trailer 8. This leveling process may be repeated cyclically until the automated trailer leveling system 134 has achieved the level conditions according to the fused trailer leveling parameter 140.

[0074] While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be understood that numerous variations exist. It should also be understood that the one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of this disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient roadmap for implementing one or more exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. A method for measuring the level of a trailer, wherein, The trailer is connected to the vehicle, and the method includes: The vehicle's rear-facing camera captures an image of the trailer, which includes the front of the trailer; Vehicle level parameters are received from the vehicle level estimator via at least one processor, the vehicle level parameters indicating the level of the vehicle based on a digital elevation map and the vehicle's current position; The at least one processor determines the rotation angle of the trailer relative to the vehicle based on the image and a comparison of the image with a stored reference image captured at a known roll angle of the trailer. The at least one processor determines a vision-based estimated trailer level parameter based on the rotation angle and the vehicle level parameter, the vision-based estimated trailer level parameter using an image from the vehicle's rear camera, a vehicle level parameter from a digital elevation map, the vehicle's current position, and a stored reference image to indicate the trailer's level. Via the at least one processor, sensor-based trailer level parameters are received based on level measurements from a trailer level estimator mounted on the trailer, wherein the level estimator includes an inertial measurement unit (IMU) mounted on the surface of the trailer; By means of the at least one processor, based on the fusion of (A) vision-based estimated trailer level parameters and (B) sensor-based trailer level parameters, a fused trailer level parameter is determined by utilizing the different estimates of the vision-based estimated trailer level parameters and the sensor-based trailer level parameters at different corresponding times, wherein the fusion of the estimated trailer level parameters and the sensor-based trailer level parameters is weighted based on the inverse of the variance tracked over time; The estimated trailer levelness parameter is output via the at least one processor; and The height under one or more wheels of the trailer is automatically adjusted according to instructions provided by at least one processor.

2. The method according to claim 1, wherein, Outputting the estimated trailer level parameters includes generating a user interface that indicates the estimated trailer level parameters.

3. The method according to claim 2, wherein, The user interface is included on at least one of the vehicle's native display device and the mobile computer's display device.

4. The method according to claim 1, wherein, Outputting the estimated trailer level parameters includes providing the estimated trailer level parameters to the automatic trailer leveling system.

5. The method according to claim 1, wherein, Determining the rotation angle of the trailer relative to the vehicle based on the image includes aligning the optical features of the image with the optical features of a reference image.

6. The method according to claim 5, wherein, When the trailer level is known and the reference image is stored in memory, the reference image is acquired by the camera during the initialization process.

7. The method according to claim 1, wherein, Determining the sensor-based trailer level parameters involves regression from a model.

8. A system for measuring the level of a trailer, the system comprising: A vehicle, the vehicle including a vehicle levelness estimator; A trailer that can be attached to the vehicle; A rear-facing camera, the rear-facing camera being mounted on the vehicle; At least one processor, operably in communication with the rear-facing camera, wherein the at least one processor is configured to execute program instructions, wherein the program instructions are configured to cause the at least one processor to: The rear-facing camera is used to capture an image of the trailer, which includes the front of the trailer; The vehicle level parameters are received from the vehicle level estimator, which indicate the vehicle's level based on a digital elevation map and the vehicle's current position. The rotation angle of the trailer relative to the vehicle is determined based on the image and a comparison of the image with a stored reference image captured at a known roll angle of the trailer; A vision-based estimated trailer level parameter is determined based on the rotation angle and the vehicle level parameter. The vision-based estimated trailer level parameter uses an image from the vehicle's rear camera, vehicle level parameters from a digital elevation map, the vehicle's current position, and a stored reference image to indicate the trailer's level. Via the at least one processor, sensor-based trailer level parameters are received based on level measurements from a trailer level estimator mounted on the trailer, wherein the level estimator includes an inertial measurement unit (IMU) mounted on the surface of the trailer; By means of the at least one processor, based on the fusion of (A) vision-based estimated trailer level parameters and (B) sensor-based trailer level parameters, a fused trailer level parameter is determined by utilizing the different estimates of the vision-based estimated trailer level parameters and the sensor-based trailer level parameters at different corresponding times, wherein the fusion of the estimated trailer level parameters and the sensor-based trailer level parameters is weighted based on the inverse of the variance tracked over time; Output the estimated trailer levelness parameters; as well as The height under one or more wheels of the trailer is automatically adjusted according to instructions provided by at least one processor.

Images